Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGELOG.MD
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,12 @@
- Reuse tokens for repeated values, avoid existing token collisions, reject
invalid explicit ranges and unsupported entity selections, and report backend
entity capabilities. Async decorators await results before filtering.
- Keep Core within the reviewed minor series, raise the Pydantic floor to 2.1,
- Require the reviewed Core 2.0 minor series (`datafog-core~=2.0.0`) in the base
package and `rust` extra; review native differences in `rust-2.0.0.json`.
Core's complete mapped-IPv6 spans, email boundaries, Unicode mailboxes and
phone-plus-ZIP classification are available through the wrapper. Published
Core artifacts must pass dependent qualification before this candidate ships.
- Raise the Pydantic floor to 2.1,
include spaCy in `nlp-advanced`, and allow Torch 2.x versions compatible with
`datafog-core[gliner2]`. Add an isolated build declaration and runnable sdist
tests; remove unused duplicate implementation modules.
Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ and `ADDRESS` from an embedded model, so name and address detection needs no
spaCy and no model download. Findings keep `engine="regex"` provenance and
`confidence=1.0` as compatibility values, not native confidence estimates.
Reviewed differences from the Python engine are listed in
[tests/contracts/rust-1.4.0.json](tests/contracts/rust-1.4.0.json).
[tests/contracts/rust-2.0.0.json](tests/contracts/rust-2.0.0.json).
`datafog.native` exposes the native schema directly.

Python 3.13 support is certified for the core SDK, CLI, `nlp`,
Expand Down Expand Up @@ -288,9 +288,12 @@ Telemetry does not include input text or detected PII values.

The native engine is the default, with a warned Python fallback. The 4.12
review fixes preserve structured detection across every engine and legacy API.
The dependency stays within the reviewed Core 1.4 minor series (`~=1.4.0`);
Core 2.0 is a separate release candidate that must be published and qualified
before this SDK raises its floor. See [release preparation](docs/review-remediation.md).
This release candidate declares the reviewed Core 2.0 minor series
(`datafog-core~=2.0.0`) in the base package and the `rust` extra. Core 2.0 is a
separate release candidate; this branch does not publish it. Merge and release
require Core 2.0 publication followed by qualification against its actual
published artifacts. Local source-wheel results are additional evidence.
See [release preparation](docs/review-remediation.md).

Entity labels, locales and activation settings come from the installed Core.
`get_supported_entities()` reports that backend's supported inventory, including
Expand Down
57 changes: 57 additions & 0 deletions RELEASE_NOTES_4.12.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# DataFog 4.12.0

DataFog 4.12.0 carries structured PII detection through every engine, legacy
API and CLI anonymization path. Smart, spaCy and GLiNER modes retain the same
native card, IP and credential findings as the regex mode, then add NER spans.
TextService scans complete documents, and spaCy/GLiNER character offsets prevent
an earlier substring or repeated name from receiving the wrong redaction.

This candidate requires `datafog-core~=2.0.0` in the base package and the `rust`
extra. Core 2.0 supplies complete IPv4-mapped IPv6 redaction, corrected email
field boundaries and Unicode mailboxes, and phone-plus-ZIP classification that
does not suppress the phone behind a false card finding. Native differences
are recorded in `tests/contracts/rust-2.0.0.json`. Capability contract 1 remains
unchanged; Python and Core have independent version numbers.

Lone surrogate characters no longer disable native scans. Dense overlap
suppression, Python fallback patterns and allowlist matching have bounded work;
typoed entity labels, invalid ranges and invalid locales raise explicit errors.
Anonymizing pipeline operations imply scanning. Async guardrail decorators
await their result, and non-string results raise TypeError.

LiteLLM scans nested tool/function arguments and prompt/input requests with a
shared token assignment for the request. Repeated values reuse tokens, and
existing markers do not collide. Streaming buffers the complete upstream
response before emitting its first redacted chunk, including text and tool
arguments split across deltas. This adds the full upstream completion latency
to first output. Streams exceeding eight million text characters or 100,000
chunks are rejected. Engine errors retain the configured `open` or `closed`
failure policy. Image and binary parts remain opaque and receive no OCR.

The Claude Code hook asks for review, or denies according to configuration,
when input exceeds its scan budget; it no longer silently permits an unscanned
tail. Unexpected hook failures retain the documented fail-open policy.

Hash APIs honor the selected algorithm while preserving existing plain-hash
formats. Engine APIs, the DataFog class and CLI retain SHA-256 with 12 hex
characters by default. Deprecated top-level `datafog.process(method="hash")`
retains MD5 with eight hex characters. The legacy Anonymizer model retains its
existing full plain digest. Explicit `hash_key` adds full-length HMAC with at
least 32 key bytes; omitting the algorithm in keyed mode selects SHA-256.
Unkeyed hashes remain guessable deterministic pseudonyms, not protection
against enumerating SSNs, phone numbers or other low-entropy values.

The minimum Pydantic version is 2.1. The `nlp-advanced` extra includes spaCy and
permits Torch 2.x versions compatible with Core's GLiNER2 extra. Base installs
show a CLI-extra installation hint instead of a missing-Typer traceback.
Unused duplicate implementations are removed, source distributions include
runnable tests and corpora, and CLI configuration output hides API keys.

Scheduled release jobs qualify candidates without publishing. Manual jobs
default to dry runs, reject stale or occupied versions, and record immutable
release refs before uploading artifacts.

This is a release candidate prepared for review. Core 2.0 publication and
qualification of this Python candidate against the actual published Core
artifacts remain mandatory before merge and release. Local source-wheel
qualification does not claim either package has been published.
23 changes: 14 additions & 9 deletions docs/review-remediation.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,18 @@ result = datafog.redact("mail jane@example.com", strategy="hash", hash_key=key)

## Release sequence and qualification

The Python candidate is 4.12.0. Its standalone CI uses the published Core 1.4
minor series, pinned with `~=1.4.0`. Core 2.0.0 is prepared separately because
safe mask-reveal and automatic-overlap transformation semantics are incompatible
changes under Core's version policy. Both use capability contract 1. Before
shipping the combined fixes, publish the qualified Core candidate, then raise
both Python dependency declarations to `~=2.0.0`, update the reviewed native
contract/corpus, and run Python's installed-wheel and release qualification
against the published artifacts. Source-candidate integration is additional
The Python 4.12.0 candidate declares `datafog-core~=2.0.0` in its base
requirements and `rust` extra. Its reviewed native contract is
[`tests/contracts/rust-2.0.0.json`](../tests/contracts/rust-2.0.0.json).
Core 2.0.0 is prepared separately because safe mask-reveal and automatic-overlap
transformation semantics are incompatible changes under Core's version policy.
Both packages continue to use capability contract 1.

This branch prepares the dependent package without publishing either project.
Its local source wheel is qualified with the source-built Core 2.0.0 wheel.
Merge and publication remain blocked until Core 2.0.0 is published and this
exact Python candidate passes installed-wheel and release qualification against
the actual published Core artifacts. Source-candidate integration is additional
evidence; it does not substitute for that post-publication gate.

Scheduled Python jobs only qualify candidates. Manual `release.yml` runs default
Expand All @@ -82,7 +86,8 @@ build requirements are documented in Core's platform policy.

All 35 Python findings have a concrete resolution below. Core-owned fixes are
separately tracked in Core's `docs/reviews/2026-11-core-findings.md`; their combined
release still requires the Core publication and Python dependency update above.
release still requires Core publication and the published-artifact qualification
above.

| Finding | Resolution and proof |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand Down
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# Core lightweight dependencies only
core_deps = [
# Core minor releases may change findings; stay on this reviewed minor.
"datafog-core~=1.4.0",
"datafog-core~=2.0.0",
"pydantic>=2.1,<3.0",
"pydantic-settings>=2.0.0",
"typing-extensions>=4.0",
Expand Down Expand Up @@ -89,7 +89,7 @@

extras_require = {
# Retained for explicitness; the base install already pulls Core.
"rust": ["datafog-core~=1.4.0"],
"rust": ["datafog-core~=2.0.0"],
"nlp": nlp_deps,
"nlp-advanced": nlp_deps + nlp_advanced_deps,
"ocr": ocr_deps,
Expand Down