Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4937653
Merge pull request #181 from DataFog/dev
sidmohan0 Sep 28, 2026
2b2d114
feat: native engine by default with Python fallback (4.10.0)
sidmohan0 Nov 14, 2026
9a8adb1
Merge remote-tracking branch 'origin/main' into feature/4.10-convergence
sidmohan0 Nov 14, 2026
d0b3892
chore: require datafog-core 1.3.0
sidmohan0 Nov 14, 2026
288e2c2
ci: re-run after datafog-core 1.3.0 publication
sidmohan0 Nov 14, 2026
f8d737f
test: score the accuracy corpus against both backends
sidmohan0 Nov 14, 2026
e3d4300
style: prettier formatting for the native corpus delta
sidmohan0 Nov 14, 2026
8567bd1
Merge pull request #185 from DataFog/feature/4.10-convergence
sidmohan0 Nov 14, 2026
4f9b126
feat: track datafog-core 1.4.0 email-boundary defaults (#186)
sidmohan0 Nov 15, 2026
352cafe
fix(engine): preserve detection across backends, spans and large inputs
sidmohan0 Nov 15, 2026
f8700e3
fix(regex): keep email context boundaries linear and complete
sidmohan0 Nov 15, 2026
67730c9
fix(guardrails): scan complete requests and buffered streaming output
sidmohan0 Nov 15, 2026
a9c19fc
test: review wrapper contract changes and qualify shared detection
sidmohan0 Nov 15, 2026
c486a52
fix(engine): validate entity support for each optional NER engine
sidmohan0 Nov 15, 2026
0102e13
fix: complete review entrypoint, packaging and release safeguards
sidmohan0 Nov 15, 2026
dd01fa9
docs: align contributor release guidance with safe qualification
sidmohan0 Nov 15, 2026
bd42280
fix(guardrails): enforce matching stream group counts
sidmohan0 Nov 15, 2026
2cd5127
fix(release): account for untagged publications and preserve release …
sidmohan0 Nov 15, 2026
c682409
style: format updated release guidance
sidmohan0 Nov 15, 2026
0eaa6e7
fix(ci): provision benchmark models and validate current annotation c…
sidmohan0 Nov 15, 2026
ad36d6a
docs: record every review finding and its regression proof
sidmohan0 Nov 15, 2026
fb262c8
fix(hash): preserve existing pseudonyms while honoring algorithms and…
sidmohan0 Nov 15, 2026
e4e751e
fix(engine): retain structured findings when NER spans overlap
sidmohan0 Nov 15, 2026
260b04f
fix(anonymizer): preserve protection with legacy empty entity filters
sidmohan0 Nov 15, 2026
d6b8a2d
test: recognize exact restored GLiNER URL label coverage
sidmohan0 Nov 15, 2026
5476c74
chore: prepare Core 1.5 compatibility qualification
sidmohan0 Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 4.9.0
current_version = 4.12.0
commit = True
tag = True
tag_name = v{new_version}
Expand Down
5 changes: 4 additions & 1 deletion .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@ ignore = E203, E266, E501, W503, B006, B007, B008, F401, C416, B950, B904
max-line-length = 88
max-complexity = 18
select = B,C,E,F,W,T4,B9
exclude = venv, .venv, tests/.datafog_env, examples/venv
exclude = venv, .venv, tests/.datafog_env, examples/venv# datafog.native.py re-exports the installed datafog-core surface through a
# dynamic __getattr__, so its __all__ names resolve at runtime.
per-file-ignores =
datafog/native.py: F822
9 changes: 2 additions & 7 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install -e ".[nlp]"
python -m spacy download en_core_web_lg
pip install -r requirements-benchmark.txt
# Verify pytest-benchmark is installed and working
python -c "import pytest_benchmark; print('pytest-benchmark version:', pytest_benchmark.__version__)"
Expand Down Expand Up @@ -54,13 +55,7 @@ jobs:
# Run benchmarks with optimal performance settings (no memory debugging)
echo "Running benchmarks with performance-optimized settings..."

# Try pytest-benchmark first
if python -m pytest tests/benchmark_text_service.py -v --benchmark-autosave --benchmark-json=benchmark-results.json --tb=short -p no:cacheprovider; then
echo "✅ pytest-benchmark tests completed successfully"
else
echo "⚠️ pytest-benchmark failed, running simple performance test as fallback"
python tests/simple_performance_test.py
fi
python -m pytest tests/benchmark_text_service.py -v --benchmark-autosave --benchmark-json=benchmark-results.json --tb=short -p no:cacheprovider

- name: Check for performance regression
run: |
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,16 @@ jobs:
--cov-report=xml \
--cov-report=term-missing

- name: Run detection accuracy corpus on the native engine
if: matrix.python-version == '3.11' && matrix.install-profile == 'nlp-advanced'
env:
DATAFOG_ACCURACY_BACKEND: rust
run: |
pytest tests/test_detection_accuracy.py \
--tb=short \
--ignore=tests/test_spark_integration.py \
--ignore=tests/test_text_service_integration.py

- name: Run tests (nlp)
if: matrix.install-profile == 'nlp'
run: |
Expand Down
139 changes: 69 additions & 70 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: Release

on:
schedule:
# Monday-Wednesday at 2 AM UTC: alpha builds from dev
# Scheduled runs validate candidates without publishing.
- cron: "0 2 * * 1-3"
# Thursday at 2 AM UTC: beta builds from dev
# Thursday beta qualification also stops before publication.
- cron: "0 2 * * 4"
workflow_dispatch:
inputs:
Expand All @@ -19,7 +19,7 @@ on:
dry_run:
description: "Dry run (skip PyPI publish)"
required: false
default: false
default: true
type: boolean
dry_run_ref:
description: "Candidate ref to validate (allowed only with dry_run)"
Expand Down Expand Up @@ -86,6 +86,15 @@ jobs:
fetch-depth: 0
ref: ${{ steps.resolve.outputs.checkout_ref }}

- name: Reject a development branch missing stable changes
if: steps.resolve.outputs.release_type != 'stable'
run: |
git fetch origin main
git merge-base --is-ancestor origin/main HEAD || {
echo "Development source is missing stable changes; merge main into dev first"
exit 1
}

- name: Freeze source commit
id: source
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -125,7 +134,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v6
with:
Expand Down Expand Up @@ -167,6 +176,11 @@ jobs:
- name: Run detection accuracy corpus
run: python -m pytest tests/test_detection_accuracy.py -v --tb=short

- name: Run detection accuracy corpus on the native engine
env:
DATAFOG_ACCURACY_BACKEND: rust
run: python -m pytest tests/test_detection_accuracy.py --tb=short

- name: Run performance validation
run: |
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 python tests/simple_performance_test.py
Expand Down Expand Up @@ -267,7 +281,7 @@ jobs:
pip install build twine bump2version

- name: Configure git
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
run: |
git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --local user.name "github-actions[bot]"
Expand All @@ -277,57 +291,20 @@ jobs:
env:
VERSION_OVERRIDE: ${{ inputs.version_override }}
run: |
set -e
set -euo pipefail
git fetch --tags

TYPE="${{ needs.determine-release.outputs.release_type }}"
CURRENT=$(sed -n 's/^__version__ = "\([^"]*\)"/\1/p' datafog/__about__.py)
if [ -z "$CURRENT" ]; then
echo "Failed to parse current version from datafog/__about__.py"
exit 1
fi
echo "Current version: $CURRENT"

# Strip any pre-release suffix to get base version
BASE=$(echo "$CURRENT" | sed -E 's/(a|b)[0-9]+([.][0-9A-Za-z]+)?$//')
if [ -n "$VERSION_OVERRIDE" ]; then
BASE="$VERSION_OVERRIDE"
if echo "$BASE" | grep -Eq '(a|b)[0-9]+([.][0-9A-Za-z]+)?$'; then
echo "version_override must be a stable base version like 4.4.0, not a prerelease"
exit 1
fi
fi
if ! echo "$BASE" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then
echo "Release base must have numeric major.minor.patch format"
exit 1
fi
echo "Base version: $BASE"

if [ "$TYPE" = "alpha" ]; then
ALPHA_NUM=1
while git tag -l "v${BASE}a${ALPHA_NUM}" | grep -q .; do
ALPHA_NUM=$((ALPHA_NUM + 1))
done
VERSION="${BASE}a${ALPHA_NUM}"

elif [ "$TYPE" = "beta" ]; then
BETA_NUM=1
while git tag -l "v${BASE}b${BETA_NUM}" | grep -q .; do
BETA_NUM=$((BETA_NUM + 1))
done
VERSION="${BASE}b${BETA_NUM}"

else
VERSION="$BASE"
fi

VERSION=$(python scripts/release_version.py --type "$TYPE" --override "$VERSION_OVERRIDE")
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Publishing version: $VERSION"

sed -i "s/__version__ = \".*\"/__version__ = \"$VERSION\"/" datafog/__about__.py
if grep -q 'version="' setup.py 2>/dev/null; then
sed -i "s/version=\".*\"/version=\"$VERSION\"/" setup.py
fi
RELEASE_VERSION="$VERSION" python - <<'PYCODE'
import os, re
from pathlib import Path
version = os.environ["RELEASE_VERSION"]
about = Path("datafog/__about__.py")
about.write_text(re.sub(r'^__version__ = ".*"', f'__version__ = "{version}"', about.read_text(), flags=re.M))
config = Path(".bumpversion.cfg")
config.write_text(re.sub(r'^current_version = .*', f'current_version = {version}', config.read_text(), flags=re.M))
PYCODE

- name: Generate changelog
env:
Expand Down Expand Up @@ -366,7 +343,7 @@ jobs:
if-no-files-found: error

- name: Verify release branch still matches tested source
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
RELEASE_BRANCH: ${{ needs.determine-release.outputs.target_branch }}
TESTED_SHA: ${{ needs.determine-release.outputs.source_sha }}
Expand All @@ -377,30 +354,36 @@ jobs:
exit 1
fi

- name: Commit and tag the validated release before publication
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
VERSION: ${{ steps.version.outputs.version }}
BRANCH: ${{ needs.determine-release.outputs.target_branch }}
run: |
set -euo pipefail
git add datafog/__about__.py .bumpversion.cfg
if ! git diff --cached --quiet; then
git commit -m "chore: bump version to $VERSION [skip ci]"
fi
git tag -a "v$VERSION" -m "Release $VERSION"
git push --atomic origin "HEAD:$BRANCH" "v$VERSION"

- name: Publish to PyPI
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
python -m twine upload dist/* --verbose

- name: Commit version bump & create release
if: inputs.dry_run != true
- name: Create GitHub release
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
GITHUB_TOKEN: ${{ secrets.GH_PAT }}
VERSION: ${{ steps.version.outputs.version }}
TYPE: ${{ needs.determine-release.outputs.release_type }}
BRANCH: ${{ needs.determine-release.outputs.target_branch }}
run: |
VERSION="${{ steps.version.outputs.version }}"
TYPE="${{ needs.determine-release.outputs.release_type }}"
BRANCH="${{ needs.determine-release.outputs.target_branch }}"

git add datafog/__about__.py setup.py
git commit -m "chore: bump version to $VERSION [skip ci]" || echo "No version changes to commit"
git push origin "HEAD:$BRANCH"

git tag -a "v$VERSION" -m "Release $VERSION"
git push origin "v$VERSION"

PRERELEASE_FLAG=""
TITLE=""
if [ "$TYPE" = "alpha" ]; then
Expand All @@ -421,17 +404,33 @@ jobs:
dist/*

- name: Dry run summary
if: inputs.dry_run == true
if: github.event_name != 'workflow_dispatch' || inputs.dry_run == true
run: |
echo "DRY RUN COMPLETE"
echo "Would have published: ${{ steps.version.outputs.version }}"
echo "Package contents:"
ls -la dist/

# ── 4. Cleanup old pre-releases ───────────────────────────────────────
sync-development:
needs: [determine-release, publish]
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true && needs.determine-release.outputs.release_type == 'stable'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: dev
fetch-depth: 0
token: ${{ secrets.GH_PAT }}
- name: Bring stable release metadata back to development
run: |
git fetch origin main
git merge --ff-only origin/main
git push origin HEAD:dev

cleanup:
needs: [determine-release, publish]
if: needs.determine-release.outputs.release_type != 'stable' && inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true && needs.determine-release.outputs.release_type != 'stable'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ docs/*
!docs/migration-4.8.1-contract.md
!docs/migration-4.9.md
!docs/core-0.4-integration.md
!docs/review-remediation.md
!docs/agents/
!docs/agents/**
!docs/audit/
Expand Down
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
repos:
- repo: https://github.com/PyCQA/isort
rev: 5.12.0
rev: 5.13.2
hooks:
- id: isort
args: [--profile=black]
exclude: .venv

- repo: https://github.com/psf/black
rev: 24.2.0
rev: 26.3.1
hooks:
- id: black
language_version: python3
Expand Down
22 changes: 11 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@

## Current Project Status

**Stable version: 4.9.0**
**Stable version: 4.11.0**

**Development version: 4.9.0**
**Development candidate: 4.12.0**

**Next major target: 5.0.0**

Expand Down Expand Up @@ -239,17 +239,17 @@ except ImportError:

### Workflow Architecture (3 workflows)

| Workflow | Purpose | Trigger |
| --------------- | ----------------------------------- | -------------------------- |
| `ci.yml` | Lint + Test + Coverage + Wheel size | Push/PR to main/dev |
| `release.yml` | Alpha/Beta/Stable publishing | Schedule + manual dispatch |
| `benchmark.yml` | Performance benchmarks | Push/PR/weekly |
| Workflow | Purpose | Trigger |
| --------------- | -------------------------------------- | --------------------------------------------- |
| `ci.yml` | Lint + Test + Coverage + Wheel size | Push/PR to main/dev |
| `release.yml` | Candidate qualification and publishing | Schedule validates; manual dispatch publishes |
| `benchmark.yml` | Performance benchmarks | Push/PR/weekly |

### Release Cadence

- **Alpha** (Mon-Wed 2AM UTC): Automatic from `dev`, date+commit versioning
- **Beta** (Thursday 2AM UTC): Automatic from `dev`, incremental beta numbers
- **Stable** (manual dispatch): From `main`, base version or override
- **Alpha** (Mon-Wed 2AM UTC): Candidate qualification from `dev`, without publication
- **Beta** (Thursday 2AM UTC): Candidate qualification from `dev`, without publication
- **Publication** (manual dispatch): Dry run by default; explicit non-dry runs publish alpha/beta from `dev` or stable from `main`

### Release Pipeline

Expand Down Expand Up @@ -320,7 +320,7 @@ Before beginning any task please checkout a branch from `dev` and create a pull

### Release Preparation

- Alpha/beta releases are automated via `release.yml` schedule
- Alpha/beta candidate qualification is automated via `release.yml`; publication requires an explicit manual dispatch
- Stable releases: merge `dev` → `main`, then trigger `release.yml` with `stable` type
- Use `dry_run: true` to validate before actual publish
- Performance validation on realistic data sets
Expand Down
Loading
Loading