diff --git a/CHANGELOG.MD b/CHANGELOG.MD index ab2e41b6..62acaddb 100644 --- a/CHANGELOG.MD +++ b/CHANGELOG.MD @@ -23,7 +23,12 @@ - Reuse tokens for repeated values, avoid existing token collisions, reject invalid explicit ranges and unsupported entity selections, and report backend entity capabilities. Async decorators await results before filtering. -- Keep Core within the reviewed minor series, raise the Pydantic floor to 2.1, +- Require the reviewed Core 2.0 minor series (`datafog-core~=2.0.0`) in the base + package and `rust` extra; review native differences in `rust-2.0.0.json`. + Core's complete mapped-IPv6 spans, email boundaries, Unicode mailboxes and + phone-plus-ZIP classification are available through the wrapper. Published + Core artifacts must pass dependent qualification before this candidate ships. +- Raise the Pydantic floor to 2.1, include spaCy in `nlp-advanced`, and allow Torch 2.x versions compatible with `datafog-core[gliner2]`. Add an isolated build declaration and runnable sdist tests; remove unused duplicate implementation modules. diff --git a/README.md b/README.md index 375cbc53..7ee0f892 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ and `ADDRESS` from an embedded model, so name and address detection needs no spaCy and no model download. Findings keep `engine="regex"` provenance and `confidence=1.0` as compatibility values, not native confidence estimates. Reviewed differences from the Python engine are listed in -[tests/contracts/rust-1.4.0.json](tests/contracts/rust-1.4.0.json). +[tests/contracts/rust-2.0.0.json](tests/contracts/rust-2.0.0.json). `datafog.native` exposes the native schema directly. Python 3.13 support is certified for the core SDK, CLI, `nlp`, @@ -288,9 +288,12 @@ Telemetry does not include input text or detected PII values. The native engine is the default, with a warned Python fallback. The 4.12 review fixes preserve structured detection across every engine and legacy API. -The dependency stays within the reviewed Core 1.4 minor series (`~=1.4.0`); -Core 2.0 is a separate release candidate that must be published and qualified -before this SDK raises its floor. See [release preparation](docs/review-remediation.md). +This release candidate declares the reviewed Core 2.0 minor series +(`datafog-core~=2.0.0`) in the base package and the `rust` extra. Core 2.0 is a +separate release candidate; this branch does not publish it. Merge and release +require Core 2.0 publication followed by qualification against its actual +published artifacts. Local source-wheel results are additional evidence. +See [release preparation](docs/review-remediation.md). Entity labels, locales and activation settings come from the installed Core. `get_supported_entities()` reports that backend's supported inventory, including diff --git a/RELEASE_NOTES_4.12.0.md b/RELEASE_NOTES_4.12.0.md new file mode 100644 index 00000000..e392b15c --- /dev/null +++ b/RELEASE_NOTES_4.12.0.md @@ -0,0 +1,57 @@ +# DataFog 4.12.0 + +DataFog 4.12.0 carries structured PII detection through every engine, legacy +API and CLI anonymization path. Smart, spaCy and GLiNER modes retain the same +native card, IP and credential findings as the regex mode, then add NER spans. +TextService scans complete documents, and spaCy/GLiNER character offsets prevent +an earlier substring or repeated name from receiving the wrong redaction. + +This candidate requires `datafog-core~=2.0.0` in the base package and the `rust` +extra. Core 2.0 supplies complete IPv4-mapped IPv6 redaction, corrected email +field boundaries and Unicode mailboxes, and phone-plus-ZIP classification that +does not suppress the phone behind a false card finding. Native differences +are recorded in `tests/contracts/rust-2.0.0.json`. Capability contract 1 remains +unchanged; Python and Core have independent version numbers. + +Lone surrogate characters no longer disable native scans. Dense overlap +suppression, Python fallback patterns and allowlist matching have bounded work; +typoed entity labels, invalid ranges and invalid locales raise explicit errors. +Anonymizing pipeline operations imply scanning. Async guardrail decorators +await their result, and non-string results raise TypeError. + +LiteLLM scans nested tool/function arguments and prompt/input requests with a +shared token assignment for the request. Repeated values reuse tokens, and +existing markers do not collide. Streaming buffers the complete upstream +response before emitting its first redacted chunk, including text and tool +arguments split across deltas. This adds the full upstream completion latency +to first output. Streams exceeding eight million text characters or 100,000 +chunks are rejected. Engine errors retain the configured `open` or `closed` +failure policy. Image and binary parts remain opaque and receive no OCR. + +The Claude Code hook asks for review, or denies according to configuration, +when input exceeds its scan budget; it no longer silently permits an unscanned +tail. Unexpected hook failures retain the documented fail-open policy. + +Hash APIs honor the selected algorithm while preserving existing plain-hash +formats. Engine APIs, the DataFog class and CLI retain SHA-256 with 12 hex +characters by default. Deprecated top-level `datafog.process(method="hash")` +retains MD5 with eight hex characters. The legacy Anonymizer model retains its +existing full plain digest. Explicit `hash_key` adds full-length HMAC with at +least 32 key bytes; omitting the algorithm in keyed mode selects SHA-256. +Unkeyed hashes remain guessable deterministic pseudonyms, not protection +against enumerating SSNs, phone numbers or other low-entropy values. + +The minimum Pydantic version is 2.1. The `nlp-advanced` extra includes spaCy and +permits Torch 2.x versions compatible with Core's GLiNER2 extra. Base installs +show a CLI-extra installation hint instead of a missing-Typer traceback. +Unused duplicate implementations are removed, source distributions include +runnable tests and corpora, and CLI configuration output hides API keys. + +Scheduled release jobs qualify candidates without publishing. Manual jobs +default to dry runs, reject stale or occupied versions, and record immutable +release refs before uploading artifacts. + +This is a release candidate prepared for review. Core 2.0 publication and +qualification of this Python candidate against the actual published Core +artifacts remain mandatory before merge and release. Local source-wheel +qualification does not claim either package has been published. diff --git a/docs/review-remediation.md b/docs/review-remediation.md index 04e010ea..adfd9933 100644 --- a/docs/review-remediation.md +++ b/docs/review-remediation.md @@ -50,14 +50,18 @@ result = datafog.redact("mail jane@example.com", strategy="hash", hash_key=key) ## Release sequence and qualification -The Python candidate is 4.12.0. Its standalone CI uses the published Core 1.4 -minor series, pinned with `~=1.4.0`. Core 2.0.0 is prepared separately because -safe mask-reveal and automatic-overlap transformation semantics are incompatible -changes under Core's version policy. Both use capability contract 1. Before -shipping the combined fixes, publish the qualified Core candidate, then raise -both Python dependency declarations to `~=2.0.0`, update the reviewed native -contract/corpus, and run Python's installed-wheel and release qualification -against the published artifacts. Source-candidate integration is additional +The Python 4.12.0 candidate declares `datafog-core~=2.0.0` in its base +requirements and `rust` extra. Its reviewed native contract is +[`tests/contracts/rust-2.0.0.json`](../tests/contracts/rust-2.0.0.json). +Core 2.0.0 is prepared separately because safe mask-reveal and automatic-overlap +transformation semantics are incompatible changes under Core's version policy. +Both packages continue to use capability contract 1. + +This branch prepares the dependent package without publishing either project. +Its local source wheel is qualified with the source-built Core 2.0.0 wheel. +Merge and publication remain blocked until Core 2.0.0 is published and this +exact Python candidate passes installed-wheel and release qualification against +the actual published Core artifacts. Source-candidate integration is additional evidence; it does not substitute for that post-publication gate. Scheduled Python jobs only qualify candidates. Manual `release.yml` runs default @@ -82,7 +86,8 @@ build requirements are documented in Core's platform policy. All 35 Python findings have a concrete resolution below. Core-owned fixes are separately tracked in Core's `docs/reviews/2026-11-core-findings.md`; their combined -release still requires the Core publication and Python dependency update above. +release still requires Core publication and the published-artifact qualification +above. | Finding | Resolution and proof | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | diff --git a/setup.py b/setup.py index bffb7ff5..62154455 100644 --- a/setup.py +++ b/setup.py @@ -19,7 +19,7 @@ # Core lightweight dependencies only core_deps = [ # Core minor releases may change findings; stay on this reviewed minor. - "datafog-core~=1.4.0", + "datafog-core~=2.0.0", "pydantic>=2.1,<3.0", "pydantic-settings>=2.0.0", "typing-extensions>=4.0", @@ -89,7 +89,7 @@ extras_require = { # Retained for explicitness; the base install already pulls Core. - "rust": ["datafog-core~=1.4.0"], + "rust": ["datafog-core~=2.0.0"], "nlp": nlp_deps, "nlp-advanced": nlp_deps + nlp_advanced_deps, "ocr": ocr_deps,