diff --git a/CHANGELOG.MD b/CHANGELOG.MD index ab2e41b6..86116707 100644 --- a/CHANGELOG.MD +++ b/CHANGELOG.MD @@ -2,6 +2,8 @@ ## [Unreleased] +## [4.12.0] + #### Security - Preserve native structured PII detection in smart, spaCy, GLiNER, legacy and @@ -23,7 +25,12 @@ - Reuse tokens for repeated values, avoid existing token collisions, reject invalid explicit ranges and unsupported entity selections, and report backend entity capabilities. Async decorators await results before filtering. -- Keep Core within the reviewed minor series, raise the Pydantic floor to 2.1, +- Require the reviewed Core 1.5 minor series (`datafog-core~=1.5.0`) in the base + package and `rust` extra; review native differences in `rust-1.5.0.json`. + Core's complete mapped-IPv6 spans, email boundaries, Unicode mailboxes and + phone-plus-ZIP classification are available through the wrapper. Published + Core artifacts passed dependent installed-wheel qualification. +- Raise the Pydantic floor to 2.1, include spaCy in `nlp-advanced`, and allow Torch 2.x versions compatible with `datafog-core[gliner2]`. Add an isolated build declaration and runnable sdist tests; remove unused duplicate implementation modules. diff --git a/README.md b/README.md index 73749bed..8c647a1e 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ and `ADDRESS` from an embedded model, so name and address detection needs no spaCy and no model download. Findings keep `engine="regex"` provenance and `confidence=1.0` as compatibility values, not native confidence estimates. Reviewed differences from the Python engine are listed in -[tests/contracts/rust-1.4.0.json](tests/contracts/rust-1.4.0.json). +[tests/contracts/rust-1.5.0.json](tests/contracts/rust-1.5.0.json). `datafog.native` exposes the native schema directly. Python 3.13 support is certified for the core SDK, CLI, `nlp`, @@ -214,7 +214,7 @@ scan/redact helpers, or guardrail helpers. DataFog deprecates OCR and Spark with visible use-time warnings; their APIs and extras will be removed in 5.0. The last supported -version in this release train is 4.12.0 — after publication, pin it with +version in this release train is 4.12.0 — pin it with `pip install "datafog==4.12.0"` — and every retired surface is tracked in the [retirement schedule](docs/retirement.rst). See the [4.9 migration guide](docs/migration-4.9.md) for the transition plan, @@ -288,9 +288,11 @@ Telemetry does not include input text or detected PII values. The native engine is the default, with a warned Python fallback. The 4.12 review fixes preserve structured detection across every engine and legacy API. -The dependency stays within the reviewed Core 1.4 minor series (`~=1.4.0`); -Core 1.5 is a separate release candidate that must be published and qualified -before this SDK raises its floor. See [release preparation](docs/review-remediation.md). +DataFog 4.12 requires the reviewed Core 1.5 minor series +(`datafog-core~=1.5.0`) in the base package and the `rust` extra. The published +Core 1.5.0 wheel passed installed integration and compatibility qualification +with this release. Python and Core have independent release versions. +See [release preparation](docs/review-remediation.md). Entity labels, locales and activation settings come from the installed Core. `get_supported_entities()` reports that backend's supported inventory, including diff --git a/RELEASE_NOTES_4.12.0.md b/RELEASE_NOTES_4.12.0.md new file mode 100644 index 00000000..ad598fd8 --- /dev/null +++ b/RELEASE_NOTES_4.12.0.md @@ -0,0 +1,56 @@ +# DataFog 4.12.0 + +DataFog 4.12.0 carries structured PII detection through every engine, legacy +API and CLI anonymization path. Smart, spaCy and GLiNER modes retain the same +native card, IP and credential findings as the regex mode, then add NER spans. +TextService scans complete documents, and spaCy/GLiNER character offsets prevent +an earlier substring or repeated name from receiving the wrong redaction. + +This release requires `datafog-core~=1.5.0` in the base package and the `rust` +extra. Core 1.5 supplies complete IPv4-mapped IPv6 redaction, corrected email +field boundaries and Unicode mailboxes, and phone-plus-ZIP classification that +does not suppress the phone behind a false card finding. Native differences +are recorded in `tests/contracts/rust-1.5.0.json`. Capability contract 1 remains +unchanged; Python and Core have independent version numbers. + +Lone surrogate characters no longer disable native scans. Dense overlap +suppression, Python fallback patterns and allowlist matching have bounded work; +typoed entity labels, invalid ranges and invalid locales raise explicit errors. +Anonymizing pipeline operations imply scanning. Async guardrail decorators +await their result, and non-string results raise TypeError. + +LiteLLM scans nested tool/function arguments and prompt/input requests with a +shared token assignment for the request. Repeated values reuse tokens, and +existing markers do not collide. Streaming buffers the complete upstream +response before emitting its first redacted chunk, including text and tool +arguments split across deltas. This adds the full upstream completion latency +to first output. Streams exceeding eight million text characters or 100,000 +chunks are rejected. Engine errors retain the configured `open` or `closed` +failure policy. Image and binary parts remain opaque and receive no OCR. + +The Claude Code hook asks for review, or denies according to configuration, +when input exceeds its scan budget; it no longer silently permits an unscanned +tail. Unexpected hook failures retain the documented fail-open policy. + +Hash APIs honor the selected algorithm while preserving existing plain-hash +formats. Engine APIs, the DataFog class and CLI retain SHA-256 with 12 hex +characters by default. Deprecated top-level `datafog.process(method="hash")` +retains MD5 with eight hex characters. The legacy Anonymizer model retains its +existing full plain digest. Explicit `hash_key` adds full-length HMAC with at +least 32 key bytes; omitting the algorithm in keyed mode selects SHA-256. +Unkeyed hashes remain guessable deterministic pseudonyms, not protection +against enumerating SSNs, phone numbers or other low-entropy values. + +The minimum Pydantic version is 2.1. The `nlp-advanced` extra includes spaCy and +permits Torch 2.x versions compatible with Core's GLiNER2 extra. Base installs +show a CLI-extra installation hint instead of a missing-Typer traceback. +Unused duplicate implementations are removed, source distributions include +runnable tests and corpora, and CLI configuration output hides API keys. + +Scheduled release jobs qualify candidates without publishing. Manual jobs +default to dry runs, reject stale or occupied versions, and record immutable +release refs before uploading artifacts. + +The published Core 1.5.0 dependency passed installed-wheel qualification with +this Python release, including the native bridge and compatibility contracts. +The Python and Core versions are published independently. diff --git a/docs/review-remediation.md b/docs/review-remediation.md index 7618b363..bd7a67dd 100644 --- a/docs/review-remediation.md +++ b/docs/review-remediation.md @@ -1,6 +1,6 @@ # November 2026 review remediation -This candidate addresses the review of DataFog 4.11.0 and Core 1.4.0. The +DataFog 4.12.0 addresses the review of DataFog 4.11.0 and Core 1.4.0. The review's suggested fixes were assessed against the local implementations; regression tests cover the implemented behavior. @@ -50,15 +50,26 @@ result = datafog.redact("mail jane@example.com", strategy="hash", hash_key=key) ## Release sequence and qualification -The Python candidate is 4.12.0. Its standalone CI uses the published Core 1.4 -minor series, pinned with `~=1.4.0`. Core 1.5.0 is prepared separately as a minor -release correcting detection and transformation behavior through the existing -APIs. Public APIs remain available and capability contract 1 is unchanged. Before -shipping the combined fixes, publish the qualified Core candidate, then raise -both Python dependency declarations to `~=1.5.0`, update the reviewed native -contract/corpus, and run Python's installed-wheel and release qualification -against the published artifacts. Source-candidate integration is additional -evidence; it does not substitute for that post-publication gate. +Python 4.12.0 declares `datafog-core~=1.5.0` in its base +requirements and `rust` extra. Its reviewed native contract is +[`tests/contracts/rust-1.5.0.json`](../tests/contracts/rust-1.5.0.json). +Core 1.5.0 is published separately as a minor release with the detector and +transformation safety fixes from the review, including bounded mask reveal +and complete scanner coverage for automatic overlap handling. Public APIs +remain available and both packages continue to use capability contract 1. + +The actual PyPI Core 1.5.0 macOS ARM wheel was downloaded and verified against +its registry SHA-256, +`b3ebaca3c7751ec0288ffebec154ffbb6ba73e54b340f3c4e287e1946bbe5e87`. +All five runtime package files, including the native library, exactly match +the final qualified Core build from source `b183bede189ac35413120b995efcb9a29256b4b0`; +only the build SBOM and RECORD differ. Installing that published wheel into +the isolated Python 4.12 environment passes `pip check`, the installed native +bridge smoke test, and the complete non-slow suite: 1,143 passed, 21 skipped, +19 expected failures. The 726-pass real-model/native corpus qualification +continues to apply through the verified byte-identical runtime payload. +Hosted release gates independently qualify published Core on all required +Python and platform combinations before publishing Python 4.12. Scheduled Python jobs only qualify candidates. Manual `release.yml` runs default to `dry_run: true`; explicitly selecting false is the publication boundary. @@ -82,7 +93,7 @@ build requirements are documented in Core's platform policy. All 35 Python findings have a concrete resolution below. Core-owned fixes are separately tracked in Core's `docs/reviews/2026-11-core-findings.md`; their combined -release still requires the Core publication and Python dependency update above. +release includes the published-artifact qualification above. | Finding | Resolution and proof | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | diff --git a/setup.py b/setup.py index bffb7ff5..76049bfd 100644 --- a/setup.py +++ b/setup.py @@ -19,7 +19,7 @@ # Core lightweight dependencies only core_deps = [ # Core minor releases may change findings; stay on this reviewed minor. - "datafog-core~=1.4.0", + "datafog-core~=1.5.0", "pydantic>=2.1,<3.0", "pydantic-settings>=2.0.0", "typing-extensions>=4.0", @@ -89,7 +89,7 @@ extras_require = { # Retained for explicitness; the base install already pulls Core. - "rust": ["datafog-core~=1.4.0"], + "rust": ["datafog-core~=1.5.0"], "nlp": nlp_deps, "nlp-advanced": nlp_deps + nlp_advanced_deps, "ocr": ocr_deps,