Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@ ignore = E203, E266, E501, W503, B006, B007, B008, F401, C416, B950, B904
max-line-length = 88
max-complexity = 18
select = B,C,E,F,W,T4,B9
exclude = venv, .venv, tests/.datafog_env, examples/venv
exclude = venv, .venv, tests/.datafog_env, examples/venv# datafog.native.py re-exports the installed datafog-core surface through a
# dynamic __getattr__, so its __all__ names resolve at runtime.
per-file-ignores =
datafog/native.py: F822
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,16 @@ jobs:
--cov-report=xml \
--cov-report=term-missing

- name: Run detection accuracy corpus on the native engine
if: matrix.python-version == '3.11' && matrix.install-profile == 'nlp-advanced'
env:
DATAFOG_ACCURACY_BACKEND: rust
run: |
pytest tests/test_detection_accuracy.py \
--tb=short \
--ignore=tests/test_spark_integration.py \
--ignore=tests/test_text_service_integration.py

- name: Run tests (nlp)
if: matrix.install-profile == 'nlp'
run: |
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,11 @@ jobs:
- name: Run detection accuracy corpus
run: python -m pytest tests/test_detection_accuracy.py -v --tb=short

- name: Run detection accuracy corpus on the native engine
env:
DATAFOG_ACCURACY_BACKEND: rust
run: python -m pytest tests/test_detection_accuracy.py --tb=short

- name: Run performance validation
run: |
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 python tests/simple_performance_test.py
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.MD
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
# ChangeLog

## [4.10.0]

#### Added

- Native engine is the default: `backend` resolves through `auto`, which uses
`datafog-core` when it imports and the pure-Python engine otherwise. One
warning on fallback, `DATAFOG_BACKEND` and `backend="python"` to opt out.
- Prose `PERSON` and `ADDRESS` from the model embedded in `datafog-core`; no
spaCy, GLiNER, download or interpreter.
- `API_KEY`, `BEARER_TOKEN`, `JWT`, `PRIVATE_KEY`, `CREDENTIAL_URI`,
`ACCOUNT_NUMBER`, `US_ROUTING_NUMBER`, `NPI` and `UUID` entity types, plus the
German set behind the `de` locale.
- `datafog.native` native schema access and `VERSIONING.md`.

#### Changed

- `datafog-core>=1.3.0,<2` is a normal dependency; `[rust]` remains as an
explicit spelling.
- `Development Status: 5 - Production/Stable`.
- Retirement warnings name `4.10.0` as the last supported version with an exact
pin, replacing "the final 4.x release".

#### Fixed

- `redact(..., entities=[...])` validates `backend` instead of ignoring it.

## [4.9.0]

#### Added
Expand Down
53 changes: 39 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@ values never echoed into logs or transcripts:

- **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell
commands, web requests, file writes, MCP tools) and warns the model when
prompts or tool results carry PII. ~70–90ms per invocation including
process startup. Easiest install is the
prompts or tool results carry PII. ~105–110ms per invocation including
process startup and the native engine load (`DATAFOG_BACKEND=python` trims
~20ms at the cost of detection coverage). Easiest install is the
[Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin):

```
Expand All @@ -32,8 +33,8 @@ values never echoed into logs or transcripts:

- **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in
requests and responses at the gateway, for any LiteLLM-proxied provider.
In-process (~40µs per message scanned; a request clears the guardrail in
well under a millisecond), no sidecar service. Setup:
In-process (tens of microseconds per message scanned on the native engine),
no sidecar service. Setup:
[examples/litellm_guardrail/](examples/litellm_guardrail/).

Both default to the high-precision entity set (`EMAIL`, `PHONE`,
Expand All @@ -51,7 +52,7 @@ NER live in [benchmarks/](benchmarks/).
## Installation

```bash
# Core install (regex engine)
# Core install: native engine, no extra packages needed
pip install datafog

# Add spaCy support
Expand All @@ -70,9 +71,33 @@ pip install datafog[distributed]
pip install datafog[all]
```

`pip install datafog` pulls `datafog-core`, which is the native engine. Wheels
exist for Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
Windows (amd64/arm64); anywhere else the source build needs a Rust toolchain.
If Core cannot be imported, detection falls back to the pure-Python engine and
warns once — `DATAFOG_BACKEND=python` selects that path deliberately.

The `nlp-advanced` extra includes SentencePiece and protobuf for GLiNER's
multilingual tokenizer; installing the OCR extra is not required for GLiNER.

## Backends

`backend` defaults to `auto`: the native engine when `datafog-core` is
importable, the pure-Python regex engine otherwise. `DATAFOG_BACKEND` sets the
default and `backend="python"` / `backend="rust"` override it per call.
`auto` never selects the native engine for `engine="spacy"`, `"gliner"` or
`"smart"`, which stay on the Python ML path.

The native engine adds detectors the Python engine never had — `API_KEY`,
`BEARER_TOKEN`, `JWT`, `PRIVATE_KEY`, `CREDENTIAL_URI`, `ACCOUNT_NUMBER`,
`US_ROUTING_NUMBER`, `NPI`, `UUID` and a German set — and serves prose `PERSON`
and `ADDRESS` from an embedded model, so name and address detection needs no
spaCy and no model download. Findings keep `engine="regex"` provenance and
`confidence=1.0` as compatibility values, not native confidence estimates.
Reviewed differences from the Python engine are listed in
[tests/contracts/rust-1.3.0.json](tests/contracts/rust-1.3.0.json).
`datafog.native` exposes the native schema directly.

Python 3.13 support is certified for the core SDK, CLI, `nlp`,
`nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model
that is available locally before runtime use. `distributed` and `all` remain
Expand Down Expand Up @@ -188,10 +213,12 @@ scan/redact helpers, or guardrail helpers.
model.
- A Java runtime is required by PySpark.

DataFog 4.9.0 deprecates OCR and Spark with visible use-time
warnings; their APIs and extras will be removed in 5.0. They remain functional
in 4.9. Users who need these features can stay on the final 4.x release. See
the [4.9 migration guide](docs/migration-4.9.md) for the transition plan.
DataFog deprecates OCR and Spark with visible use-time
warnings; their APIs and extras will be removed in 5.0. The last supported
version is 4.10.0 — pin it with `pip install "datafog==4.10.0"` — and every
retired surface is tracked in the [retirement schedule](docs/retirement.rst).
See the [4.9 migration guide](docs/migration-4.9.md) for the transition plan,
and [VERSIONING.md](VERSIONING.md) for what a version number promises.

## Backward-Compatible APIs

Expand Down Expand Up @@ -261,11 +288,9 @@ The [4.9 migration guide](docs/migration-4.9.md) explains opt-in Rust detection,
the native `datafog.v5` preview, and the revised 5.0 retirement schedule for
`detect`/`process`, OCR, and Spark. The Python detector remains the default.

The experimental Rust adapter in 4.9.0 requires Core `>=0.4.0,<0.5` and capability
contract 1. Upgrade with `python -m pip install --upgrade "datafog[rust]==4.9.0"`
to evaluate it. Base-only users can install `datafog==4.9.0` without Core;
installing the Rust extra does not change the default backend. See the
[4.9.0 release notes](RELEASE_NOTES_4.9.0.md).
4.10.0 requires Core `>=1.3.0,<2` and capability contract 1, and installs it by
default. See the [4.10.0 release notes](RELEASE_NOTES_4.10.0.md) and the
[4.10 migration guide](docs/migration-4.10.md).
Entity labels, locales, and activation settings come from the installed
Core, allowing compatible releases to add detectors without a Python update.
German detection is opt-in through locale or entity selection; UUID is opt-in
Expand Down
71 changes: 71 additions & 0 deletions RELEASE_NOTES_4.10.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# DataFog Python 4.10.0

4.10.0 turns on the native engine. `pip install datafog` installs `datafog-core`
and detection uses it. Nothing is removed in this release and no call changes
shape; `backend="python"` restores the previous behaviour exactly.

Read the [4.10 migration guide](docs/migration-4.10.md) before upgrading a
pipeline whose output is under test.

## Upgrade

```bash
python -m pip install --upgrade "datafog==4.10.0"
```

Wheels cover Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
Windows (amd64/arm64). Elsewhere the source build needs Rust 1.88+. If Core
cannot be imported, detection falls back to the pure-Python engine and warns
once.

## What changes

- `backend` defaults to `auto`: native when `datafog-core` imports, Python
otherwise. `DATAFOG_BACKEND` sets the default; `backend="python"` forces the
fallback. Non-regex engines stay on the Python path.
- `PERSON` and `ADDRESS` for prose now come from an embedded model compiled into
`datafog-core` — no spaCy, no GLiNER, no model download, no interpreter.
- The native engine adds `API_KEY`, `BEARER_TOKEN`, `JWT`, `PRIVATE_KEY`,
`CREDENTIAL_URI`, `ACCOUNT_NUMBER`, `US_ROUTING_NUMBER`, `NPI`, `UUID` and the
German identifiers. Credential detectors are on by default; German and `UUID`
stay opt-in.
- `datafog.native` exposes the native schema, `capabilities()`, the embedded
detector and version reporting. Experimental until 5.0.
- `datafog-core` is pinned `>=1.3.0,<2`. Detection output is contract-versioned.

## Detection differences

Four reviewed differences from the Python engine are recorded in
[tests/contracts/rust-1.3.0.json](tests/contracts/rust-1.3.0.json) and asserted
by the suite: Core validates card checksums, requires non-alphanumeric SSN
boundaries, requires a North American area code starting 2-9 (which is what
keeps Unix timestamps out of `PHONE`), and rejects unadvertised locales instead
of ignoring them.

`(555) 123-4567` and similar form data now match: Core keeps the area-code check
and accepts exchange codes starting with 1.

## Accuracy corpus

The frozen corpus runs twice in CI now: once against the Python engine, whose
expectations it encodes, and once against the native engine with its twelve
recorded divergences in
[tests/contracts/native-corpus.json](tests/contracts/native-corpus.json). The
native divergences are strictness differences (Core validates more) plus email
boundary handling that Core currently exposes through `format: "env"` /
`"sql"` settings — flagged for a default decision in Core 1.4.

## Deprecations unchanged

`detect()`, `process()`, OCR and Spark still work and still warn; all are removed
in 5.0. Their messages now name `4.10.0` as the last supported version and give
the exact pin. See the [retirement schedule](docs/retirement.rst).

## Notes

- `VERSIONING.md` is new and defines what a version number promises here, with a
deprecation window, a detection-output rule and a support window.
- Hook invocations cost ~105–110ms including process startup, of which ~20ms is
the native engine load. `DATAFOG_BACKEND=python` trades that back for coverage.
- `datafog[rust]` remains as an explicit spelling of a dependency the base
install already has.
123 changes: 123 additions & 0 deletions VERSIONING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Versioning and compatibility policy

DataFog Python follows [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html).
This document defines what that means here, so a version number is a promise
rather than a guess.

DataFog versions independently of `datafog-core`. See
[Independent versioning](#independent-versioning).

## What the numbers mean

- **MAJOR** (`5.0.0`) — a public API is removed or renamed, an import path moves,
or a default changes such that working caller code stops working.
- **MINOR** (`4.10.0`) — new capability. New entities, new backends, new extras,
new commands, new options, and **changes to detection output**.
- **PATCH** (`4.10.1`) — fixes that do not change findings for a given input,
configuration, and locked dependency set.

### Detection output is a minor change, never a patch

The finding set is part of the contract. A patch release must not start finding
more email addresses, or fewer, for input that already worked. If a fix changes
what is found, it ships as a MINOR and the changelog says which entities moved
and in which direction. People write golden files against this output.

### Dependency-default changes are minor

Raising a floor, dropping support for a Python version, or changing which
backend is used by default is a MINOR. It is never a PATCH.

## Deprecation window

A public API is deprecated in a MINOR, warning at the call boundary, and removed
no earlier than the second following MINOR or 90 days, whichever is longer.

A deprecation warning must name:

1. the exact last version that supports the surface, as an installable pin
(`pip install "datafog==4.10.0"`), and
2. the MAJOR that removes it.

Warnings fire at the public call boundary, before any optional import, so a plain
`import datafog` never pays for a deprecated surface.

At the removal MAJOR, the removed name stays present as a stub raising
`ImportError` with the same pin, rather than disappearing into an
`AttributeError`.

## Experimental surface

Experimental code is marked as such in its documentation and docstrings. It may
break in a MINOR without a deprecation cycle. Anything unmarked is covered by
this policy.

Currently experimental:

- `datafog.v5` (native Core preview)
- `datafog.core` (from 4.10.0)
- GLiNER2 and other model sessions
- Browser/WASM entry points

## Scope

Covered by this policy:

- top-level `datafog` callables listed in `datafog.__all__`
- `datafog.compat.v4` and its result classes
- the `datafog` CLI and the `datafog-hook` entry point
- `DataFogGuardrail` and the Claude Code hook integration
- documented configuration keys and environment variables
- the finding set for a given engine, entity selection, and locale

Not covered:

- private modules and names beginning with `_`
- `main_lean.py`, `main_original.py`, `__init___original.py` and other
transitional duplicates
- logging text, warning text, and internal error strings
- model accuracy targets for optional ML engines

## Support window

The current MAJOR receives fixes for its latest two MINOR releases. Older MINORs
receive security fixes only. Every published version stays installable on PyPI
permanently; a retired surface has a final supported version documented in
[`docs/retirement.rst`](docs/retirement.rst), and that version is the answer.

## Independent versioning

`datafog` and `datafog-core` are separate distributions on separate tracks.
`datafog-core` carries the engine; `datafog` carries the application surface.
The transport between them is `capabilities()` and its `contract_version`.
Bumping either `contract_version` or `workspace_contract_version` is MAJOR work
for `datafog-core`, and requires a matching floor in `datafog`.

Both are exposed separately so a bug report is attributable:

```python
import datafog, datafog.core
datafog.__version__ # application version
datafog.core.__version__ # engine version
```

## Yanking

A broken PATCH may be yanked on PyPI and replaced by the next PATCH. A broken
MINOR or MAJOR is superseded, not yanked, because dependents are often already
pinned to it; the changelog opens with the replacement version.

## Enforcement

- `RELEASE_NOTES_<version>.md` is required for stable releases.
- The changelog classifies every entry as Added / Changed / Deprecated / Removed
/ Fixed and names the version at which each deprecation ends.
- `scripts/check_rust_install.py` and the parity suite run against the installed
wheel, not the checkout.

## History

Before 4.10.0 this policy was not written down, and the versioning record shows
it: the 5.0 retirement plan was revised mid-stream (commit `5f5bdbb3`), and the
4.9.0 final publish preceded its own `a1` and `a2` tags. 4.10.0 is the first
release prepared under this policy.
2 changes: 1 addition & 1 deletion datafog/__about__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "4.9.0"
__version__ = "4.10.0"
Loading
Loading