Repository navigation
feat: native engine by default with Python fallback (4.10.0) - #185
Merged
Merged
Conversation
Resolve backend through 'auto': the native engine when datafog-core imports, the pure-Python engine otherwise with a single warning. DATAFOG_BACKEND and backend="python" force the fallback; non-regex engines stay on the Python ML path. Serve prose PERSON and ADDRESS from the embedded model in datafog-core, so names and addresses need no spaCy, GLiNER, download or interpreter. Expose the new credential, account and UUID entity types through the alias API and add datafog.native for the native schema. Deprecation warnings now name 4.10.0 as the last supported version with an exact pin, and VERSIONING.md plus docs/retirement.rst make the window and the removal schedule auditable. Nothing is removed in this release.
|
|
1.2.0 shipped without the exchange-code fix, so 4.10.0 depends on the release that has it. Contract file follows the engine version.
The corpus encodes Python-engine expectations, so it pins backend=python by default and scores the native engine under DATAFOG_ACCURACY_BACKEND=rust against tests/contracts/native-corpus.json, which records the twelve known divergences with reasons. CI runs both variants; pre-commit gains a ruff per-file-ignores entry for the dynamic re-exports in datafog/native.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Core is the engine;
pip install datafogis the front door. This release connects them. It is a MINOR because nothing is removed —backend="python"restores 4.9.0 behaviour exactly.Depends on DataFog/datafog-core#233 (Core 1.2.0): the embedded model and the musl/arm64/s390x/ppc64le wheels. Core 1.2.0 must publish first.
What
backenddefaults toauto: native whendatafog-coreimports, Python otherwise with oneCoreUnavailableWarning.DATAFOG_BACKENDsets the default,backend="python"|"rust"overrides per call. Non-regex engines stay on the Python ML path.PERSON/ADDRESSserved by the embedded model over the native path — no spaCy, GLiNER, download or interpreter.PERSONonbackend="python"still has no answer, which is now documented instead of implied.API_KEY,BEARER_TOKEN,JWT,PRIVATE_KEY,CREDENTIAL_URI,ACCOUNT_NUMBER,US_ROUTING_NUMBER,NPI,UUID) reachable through the existing alias API.datafog.nativefor the native schema (experimental),datafog.native.version()/.contract_version().VERSIONING.md,docs/retirement.rst,docs/migration-4.10.md,RELEASE_NOTES_4.10.0.md;Development Status: 5 - Production/Stable.4.10.0as the last supported version with an exact pin, replacing "the final 4.x release".datafog-core>=1.2.0,<2as a normal dependency.Reviewed detection differences
Recorded in
tests/contracts/rust-1.2.0.jsonand asserted: Luhn card validation, non-alphanumeric SSN boundaries, NANP area-code 2-9 (the Unix-timestamp guard), strict locale validation. Parity run: 76 match, 3 detector differences, 1 validation difference, 31 outside backend scope.Verification
pytest(non-slow, minus network OCR download): 761 passed, 3 skippedengine="smart"/"spacy"paths re-run withen_core_web_lginstalled: passpython -I scripts/check_rust_install.pygreen against the installed wheelpython -m tests.rust_contractmatches the reviewed inventoryDATAFOG_BACKEND=pythonrecovers it