Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.MD
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## [Unreleased]

## [4.12.0]

#### Security

- Preserve native structured PII detection in smart, spaCy, GLiNER, legacy and
Expand All @@ -23,7 +25,12 @@
- Reuse tokens for repeated values, avoid existing token collisions, reject
invalid explicit ranges and unsupported entity selections, and report backend
entity capabilities. Async decorators await results before filtering.
- Keep Core within the reviewed minor series, raise the Pydantic floor to 2.1,
- Require the reviewed Core 1.5 minor series (`datafog-core~=1.5.0`) in the base
package and `rust` extra; review native differences in `rust-1.5.0.json`.
Core's complete mapped-IPv6 spans, email boundaries, Unicode mailboxes and
phone-plus-ZIP classification are available through the wrapper. Published
Core artifacts passed dependent installed-wheel qualification.
- Raise the Pydantic floor to 2.1,
include spaCy in `nlp-advanced`, and allow Torch 2.x versions compatible with
`datafog-core[gliner2]`. Add an isolated build declaration and runnable sdist
tests; remove unused duplicate implementation modules.
Expand Down
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ and `ADDRESS` from an embedded model, so name and address detection needs no
spaCy and no model download. Findings keep `engine="regex"` provenance and
`confidence=1.0` as compatibility values, not native confidence estimates.
Reviewed differences from the Python engine are listed in
[tests/contracts/rust-1.4.0.json](tests/contracts/rust-1.4.0.json).
[tests/contracts/rust-1.5.0.json](tests/contracts/rust-1.5.0.json).
`datafog.native` exposes the native schema directly.

Python 3.13 support is certified for the core SDK, CLI, `nlp`,
Expand Down Expand Up @@ -214,7 +214,7 @@ scan/redact helpers, or guardrail helpers.

DataFog deprecates OCR and Spark with visible use-time
warnings; their APIs and extras will be removed in 5.0. The last supported
version in this release train is 4.12.0 — after publication, pin it with
version in this release train is 4.12.0 — pin it with
`pip install "datafog==4.12.0"` — and every
retired surface is tracked in the [retirement schedule](docs/retirement.rst).
See the [4.9 migration guide](docs/migration-4.9.md) for the transition plan,
Expand Down Expand Up @@ -288,9 +288,11 @@ Telemetry does not include input text or detected PII values.

The native engine is the default, with a warned Python fallback. The 4.12
review fixes preserve structured detection across every engine and legacy API.
The dependency stays within the reviewed Core 1.4 minor series (`~=1.4.0`);
Core 1.5 is a separate release candidate that must be published and qualified
before this SDK raises its floor. See [release preparation](docs/review-remediation.md).
DataFog 4.12 requires the reviewed Core 1.5 minor series
(`datafog-core~=1.5.0`) in the base package and the `rust` extra. The published
Core 1.5.0 wheel passed installed integration and compatibility qualification
with this release. Python and Core have independent release versions.
See [release preparation](docs/review-remediation.md).

Entity labels, locales and activation settings come from the installed Core.
`get_supported_entities()` reports that backend's supported inventory, including
Expand Down
56 changes: 56 additions & 0 deletions RELEASE_NOTES_4.12.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# DataFog 4.12.0

DataFog 4.12.0 carries structured PII detection through every engine, legacy
API and CLI anonymization path. Smart, spaCy and GLiNER modes retain the same
native card, IP and credential findings as the regex mode, then add NER spans.
TextService scans complete documents, and spaCy/GLiNER character offsets prevent
an earlier substring or repeated name from receiving the wrong redaction.

This release requires `datafog-core~=1.5.0` in the base package and the `rust`
extra. Core 1.5 supplies complete IPv4-mapped IPv6 redaction, corrected email
field boundaries and Unicode mailboxes, and phone-plus-ZIP classification that
does not suppress the phone behind a false card finding. Native differences
are recorded in `tests/contracts/rust-1.5.0.json`. Capability contract 1 remains
unchanged; Python and Core have independent version numbers.

Lone surrogate characters no longer disable native scans. Dense overlap
suppression, Python fallback patterns and allowlist matching have bounded work;
typoed entity labels, invalid ranges and invalid locales raise explicit errors.
Anonymizing pipeline operations imply scanning. Async guardrail decorators
await their result, and non-string results raise TypeError.

LiteLLM scans nested tool/function arguments and prompt/input requests with a
shared token assignment for the request. Repeated values reuse tokens, and
existing markers do not collide. Streaming buffers the complete upstream
response before emitting its first redacted chunk, including text and tool
arguments split across deltas. This adds the full upstream completion latency
to first output. Streams exceeding eight million text characters or 100,000
chunks are rejected. Engine errors retain the configured `open` or `closed`
failure policy. Image and binary parts remain opaque and receive no OCR.

The Claude Code hook asks for review, or denies according to configuration,
when input exceeds its scan budget; it no longer silently permits an unscanned
tail. Unexpected hook failures retain the documented fail-open policy.

Hash APIs honor the selected algorithm while preserving existing plain-hash
formats. Engine APIs, the DataFog class and CLI retain SHA-256 with 12 hex
characters by default. Deprecated top-level `datafog.process(method="hash")`
retains MD5 with eight hex characters. The legacy Anonymizer model retains its
existing full plain digest. Explicit `hash_key` adds full-length HMAC with at
least 32 key bytes; omitting the algorithm in keyed mode selects SHA-256.
Unkeyed hashes remain guessable deterministic pseudonyms, not protection
against enumerating SSNs, phone numbers or other low-entropy values.

The minimum Pydantic version is 2.1. The `nlp-advanced` extra includes spaCy and
permits Torch 2.x versions compatible with Core's GLiNER2 extra. Base installs
show a CLI-extra installation hint instead of a missing-Typer traceback.
Unused duplicate implementations are removed, source distributions include
runnable tests and corpora, and CLI configuration output hides API keys.

Scheduled release jobs qualify candidates without publishing. Manual jobs
default to dry runs, reject stale or occupied versions, and record immutable
release refs before uploading artifacts.

The published Core 1.5.0 dependency passed installed-wheel qualification with
this Python release, including the native bridge and compatibility contracts.
The Python and Core versions are published independently.
33 changes: 22 additions & 11 deletions docs/review-remediation.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# November 2026 review remediation

This candidate addresses the review of DataFog 4.11.0 and Core 1.4.0. The
DataFog 4.12.0 addresses the review of DataFog 4.11.0 and Core 1.4.0. The
review's suggested fixes were assessed against the local implementations;
regression tests cover the implemented behavior.

Expand Down Expand Up @@ -50,15 +50,26 @@ result = datafog.redact("mail jane@example.com", strategy="hash", hash_key=key)

## Release sequence and qualification

The Python candidate is 4.12.0. Its standalone CI uses the published Core 1.4
minor series, pinned with `~=1.4.0`. Core 1.5.0 is prepared separately as a minor
release correcting detection and transformation behavior through the existing
APIs. Public APIs remain available and capability contract 1 is unchanged. Before
shipping the combined fixes, publish the qualified Core candidate, then raise
both Python dependency declarations to `~=1.5.0`, update the reviewed native
contract/corpus, and run Python's installed-wheel and release qualification
against the published artifacts. Source-candidate integration is additional
evidence; it does not substitute for that post-publication gate.
Python 4.12.0 declares `datafog-core~=1.5.0` in its base
requirements and `rust` extra. Its reviewed native contract is
[`tests/contracts/rust-1.5.0.json`](../tests/contracts/rust-1.5.0.json).
Core 1.5.0 is published separately as a minor release with the detector and
transformation safety fixes from the review, including bounded mask reveal
and complete scanner coverage for automatic overlap handling. Public APIs
remain available and both packages continue to use capability contract 1.

The actual PyPI Core 1.5.0 macOS ARM wheel was downloaded and verified against
its registry SHA-256,
`b3ebaca3c7751ec0288ffebec154ffbb6ba73e54b340f3c4e287e1946bbe5e87`.
All five runtime package files, including the native library, exactly match
the final qualified Core build from source `b183bede189ac35413120b995efcb9a29256b4b0`;
only the build SBOM and RECORD differ. Installing that published wheel into
the isolated Python 4.12 environment passes `pip check`, the installed native
bridge smoke test, and the complete non-slow suite: 1,143 passed, 21 skipped,
19 expected failures. The 726-pass real-model/native corpus qualification
continues to apply through the verified byte-identical runtime payload.
Hosted release gates independently qualify published Core on all required
Python and platform combinations before publishing Python 4.12.

Scheduled Python jobs only qualify candidates. Manual `release.yml` runs default
to `dry_run: true`; explicitly selecting false is the publication boundary.
Expand All @@ -82,7 +93,7 @@ build requirements are documented in Core's platform policy.

All 35 Python findings have a concrete resolution below. Core-owned fixes are
separately tracked in Core's `docs/reviews/2026-11-core-findings.md`; their combined
release still requires the Core publication and Python dependency update above.
release includes the published-artifact qualification above.

| Finding | Resolution and proof |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand Down
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# Core lightweight dependencies only
core_deps = [
# Core minor releases may change findings; stay on this reviewed minor.
"datafog-core~=1.4.0",
"datafog-core~=1.5.0",
"pydantic>=2.1,<3.0",
"pydantic-settings>=2.0.0",
"typing-extensions>=4.0",
Expand Down Expand Up @@ -89,7 +89,7 @@

extras_require = {
# Retained for explicitness; the base install already pulls Core.
"rust": ["datafog-core~=1.4.0"],
"rust": ["datafog-core~=1.5.0"],
"nlp": nlp_deps,
"nlp-advanced": nlp_deps + nlp_advanced_deps,
"ocr": ocr_deps,
Expand Down
Loading