Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
352cafe
fix(engine): preserve detection across backends, spans and large inputs
sidmohan0 Nov 15, 2026
f8700e3
fix(regex): keep email context boundaries linear and complete
sidmohan0 Nov 15, 2026
67730c9
fix(guardrails): scan complete requests and buffered streaming output
sidmohan0 Nov 15, 2026
a9c19fc
test: review wrapper contract changes and qualify shared detection
sidmohan0 Nov 15, 2026
c486a52
fix(engine): validate entity support for each optional NER engine
sidmohan0 Nov 15, 2026
0102e13
fix: complete review entrypoint, packaging and release safeguards
sidmohan0 Nov 15, 2026
dd01fa9
docs: align contributor release guidance with safe qualification
sidmohan0 Nov 15, 2026
bd42280
fix(guardrails): enforce matching stream group counts
sidmohan0 Nov 15, 2026
2cd5127
fix(release): account for untagged publications and preserve release …
sidmohan0 Nov 15, 2026
c682409
style: format updated release guidance
sidmohan0 Nov 15, 2026
0eaa6e7
fix(ci): provision benchmark models and validate current annotation c…
sidmohan0 Nov 15, 2026
ad36d6a
docs: record every review finding and its regression proof
sidmohan0 Nov 15, 2026
fb262c8
fix(hash): preserve existing pseudonyms while honoring algorithms and…
sidmohan0 Nov 15, 2026
0d7519d
chore(release): prepare DataFog 4.12 with Core 2
sidmohan0 Nov 15, 2026
e4e751e
fix(engine): retain structured findings when NER spans overlap
sidmohan0 Nov 15, 2026
43363a6
Merge commit 'e4e751ecd28feb4b6c5e8469664936754b08050e' into release/…
sidmohan0 Nov 15, 2026
260b04f
fix(anonymizer): preserve protection with legacy empty entity filters
sidmohan0 Nov 15, 2026
d6b8a2d
test: recognize exact restored GLiNER URL label coverage
sidmohan0 Nov 15, 2026
5476c74
chore: prepare Core 1.5 compatibility qualification
sidmohan0 Nov 15, 2026
e55df68
chore(release): target Core 1.5 for DataFog 4.12
sidmohan0 Nov 15, 2026
6811135
Merge commit '5476c74716ac27356d7029f3b4007058e41b3859' into release/…
sidmohan0 Nov 15, 2026
822b79b
Fix Python 4.11 review findings and prepare 4.12 (#187)
sidmohan0 Nov 16, 2026
caa865a
Merge remote-tracking branch 'origin/dev' into release/4.12-core1.5
sidmohan0 Nov 16, 2026
3e48c34
docs(release): finalize 4.12 notes after published Core qualification
sidmohan0 Nov 16, 2026
bffff50
Release DataFog 4.12 with qualified published Core 1.5 (#189)
sidmohan0 Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 4.9.0
current_version = 4.12.0
commit = True
tag = True
tag_name = v{new_version}
Expand Down
9 changes: 2 additions & 7 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install -e ".[nlp]"
python -m spacy download en_core_web_lg
pip install -r requirements-benchmark.txt
# Verify pytest-benchmark is installed and working
python -c "import pytest_benchmark; print('pytest-benchmark version:', pytest_benchmark.__version__)"
Expand Down Expand Up @@ -54,13 +55,7 @@ jobs:
# Run benchmarks with optimal performance settings (no memory debugging)
echo "Running benchmarks with performance-optimized settings..."

# Try pytest-benchmark first
if python -m pytest tests/benchmark_text_service.py -v --benchmark-autosave --benchmark-json=benchmark-results.json --tb=short -p no:cacheprovider; then
echo "✅ pytest-benchmark tests completed successfully"
else
echo "⚠️ pytest-benchmark failed, running simple performance test as fallback"
python tests/simple_performance_test.py
fi
python -m pytest tests/benchmark_text_service.py -v --benchmark-autosave --benchmark-json=benchmark-results.json --tb=short -p no:cacheprovider

- name: Check for performance regression
run: |
Expand Down
134 changes: 64 additions & 70 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: Release

on:
schedule:
# Monday-Wednesday at 2 AM UTC: alpha builds from dev
# Scheduled runs validate candidates without publishing.
- cron: "0 2 * * 1-3"
# Thursday at 2 AM UTC: beta builds from dev
# Thursday beta qualification also stops before publication.
- cron: "0 2 * * 4"
workflow_dispatch:
inputs:
Expand All @@ -19,7 +19,7 @@ on:
dry_run:
description: "Dry run (skip PyPI publish)"
required: false
default: false
default: true
type: boolean
dry_run_ref:
description: "Candidate ref to validate (allowed only with dry_run)"
Expand Down Expand Up @@ -86,6 +86,15 @@ jobs:
fetch-depth: 0
ref: ${{ steps.resolve.outputs.checkout_ref }}

- name: Reject a development branch missing stable changes
if: steps.resolve.outputs.release_type != 'stable'
run: |
git fetch origin main
git merge-base --is-ancestor origin/main HEAD || {
echo "Development source is missing stable changes; merge main into dev first"
exit 1
}

- name: Freeze source commit
id: source
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -125,7 +134,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v6
with:
Expand Down Expand Up @@ -272,7 +281,7 @@ jobs:
pip install build twine bump2version

- name: Configure git
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
run: |
git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --local user.name "github-actions[bot]"
Expand All @@ -282,57 +291,20 @@ jobs:
env:
VERSION_OVERRIDE: ${{ inputs.version_override }}
run: |
set -e
set -euo pipefail
git fetch --tags

TYPE="${{ needs.determine-release.outputs.release_type }}"
CURRENT=$(sed -n 's/^__version__ = "\([^"]*\)"/\1/p' datafog/__about__.py)
if [ -z "$CURRENT" ]; then
echo "Failed to parse current version from datafog/__about__.py"
exit 1
fi
echo "Current version: $CURRENT"

# Strip any pre-release suffix to get base version
BASE=$(echo "$CURRENT" | sed -E 's/(a|b)[0-9]+([.][0-9A-Za-z]+)?$//')
if [ -n "$VERSION_OVERRIDE" ]; then
BASE="$VERSION_OVERRIDE"
if echo "$BASE" | grep -Eq '(a|b)[0-9]+([.][0-9A-Za-z]+)?$'; then
echo "version_override must be a stable base version like 4.4.0, not a prerelease"
exit 1
fi
fi
if ! echo "$BASE" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then
echo "Release base must have numeric major.minor.patch format"
exit 1
fi
echo "Base version: $BASE"

if [ "$TYPE" = "alpha" ]; then
ALPHA_NUM=1
while git tag -l "v${BASE}a${ALPHA_NUM}" | grep -q .; do
ALPHA_NUM=$((ALPHA_NUM + 1))
done
VERSION="${BASE}a${ALPHA_NUM}"

elif [ "$TYPE" = "beta" ]; then
BETA_NUM=1
while git tag -l "v${BASE}b${BETA_NUM}" | grep -q .; do
BETA_NUM=$((BETA_NUM + 1))
done
VERSION="${BASE}b${BETA_NUM}"

else
VERSION="$BASE"
fi

VERSION=$(python scripts/release_version.py --type "$TYPE" --override "$VERSION_OVERRIDE")
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Publishing version: $VERSION"

sed -i "s/__version__ = \".*\"/__version__ = \"$VERSION\"/" datafog/__about__.py
if grep -q 'version="' setup.py 2>/dev/null; then
sed -i "s/version=\".*\"/version=\"$VERSION\"/" setup.py
fi
RELEASE_VERSION="$VERSION" python - <<'PYCODE'
import os, re
from pathlib import Path
version = os.environ["RELEASE_VERSION"]
about = Path("datafog/__about__.py")
about.write_text(re.sub(r'^__version__ = ".*"', f'__version__ = "{version}"', about.read_text(), flags=re.M))
config = Path(".bumpversion.cfg")
config.write_text(re.sub(r'^current_version = .*', f'current_version = {version}', config.read_text(), flags=re.M))
PYCODE

- name: Generate changelog
env:
Expand Down Expand Up @@ -371,7 +343,7 @@ jobs:
if-no-files-found: error

- name: Verify release branch still matches tested source
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
RELEASE_BRANCH: ${{ needs.determine-release.outputs.target_branch }}
TESTED_SHA: ${{ needs.determine-release.outputs.source_sha }}
Expand All @@ -382,30 +354,36 @@ jobs:
exit 1
fi

- name: Commit and tag the validated release before publication
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
VERSION: ${{ steps.version.outputs.version }}
BRANCH: ${{ needs.determine-release.outputs.target_branch }}
run: |
set -euo pipefail
git add datafog/__about__.py .bumpversion.cfg
if ! git diff --cached --quiet; then
git commit -m "chore: bump version to $VERSION [skip ci]"
fi
git tag -a "v$VERSION" -m "Release $VERSION"
git push --atomic origin "HEAD:$BRANCH" "v$VERSION"

- name: Publish to PyPI
if: inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
python -m twine upload dist/* --verbose

- name: Commit version bump & create release
if: inputs.dry_run != true
- name: Create GitHub release
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true
env:
GITHUB_TOKEN: ${{ secrets.GH_PAT }}
VERSION: ${{ steps.version.outputs.version }}
TYPE: ${{ needs.determine-release.outputs.release_type }}
BRANCH: ${{ needs.determine-release.outputs.target_branch }}
run: |
VERSION="${{ steps.version.outputs.version }}"
TYPE="${{ needs.determine-release.outputs.release_type }}"
BRANCH="${{ needs.determine-release.outputs.target_branch }}"

git add datafog/__about__.py setup.py
git commit -m "chore: bump version to $VERSION [skip ci]" || echo "No version changes to commit"
git push origin "HEAD:$BRANCH"

git tag -a "v$VERSION" -m "Release $VERSION"
git push origin "v$VERSION"

PRERELEASE_FLAG=""
TITLE=""
if [ "$TYPE" = "alpha" ]; then
Expand All @@ -426,17 +404,33 @@ jobs:
dist/*

- name: Dry run summary
if: inputs.dry_run == true
if: github.event_name != 'workflow_dispatch' || inputs.dry_run == true
run: |
echo "DRY RUN COMPLETE"
echo "Would have published: ${{ steps.version.outputs.version }}"
echo "Package contents:"
ls -la dist/

# ── 4. Cleanup old pre-releases ───────────────────────────────────────
sync-development:
needs: [determine-release, publish]
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true && needs.determine-release.outputs.release_type == 'stable'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: dev
fetch-depth: 0
token: ${{ secrets.GH_PAT }}
- name: Bring stable release metadata back to development
run: |
git fetch origin main
git merge --ff-only origin/main
git push origin HEAD:dev

cleanup:
needs: [determine-release, publish]
if: needs.determine-release.outputs.release_type != 'stable' && inputs.dry_run != true
if: github.event_name == 'workflow_dispatch' && inputs.dry_run != true && needs.determine-release.outputs.release_type != 'stable'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ docs/*
!docs/migration-4.8.1-contract.md
!docs/migration-4.9.md
!docs/core-0.4-integration.md
!docs/review-remediation.md
!docs/agents/
!docs/agents/**
!docs/audit/
Expand Down
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
repos:
- repo: https://github.com/PyCQA/isort
rev: 5.12.0
rev: 5.13.2
hooks:
- id: isort
args: [--profile=black]
exclude: .venv

- repo: https://github.com/psf/black
rev: 24.2.0
rev: 26.3.1
hooks:
- id: black
language_version: python3
Expand Down
22 changes: 11 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@

## Current Project Status

**Stable version: 4.9.0**
**Stable version: 4.11.0**

**Development version: 4.9.0**
**Development candidate: 4.12.0**

**Next major target: 5.0.0**

Expand Down Expand Up @@ -239,17 +239,17 @@ except ImportError:

### Workflow Architecture (3 workflows)

| Workflow | Purpose | Trigger |
| --------------- | ----------------------------------- | -------------------------- |
| `ci.yml` | Lint + Test + Coverage + Wheel size | Push/PR to main/dev |
| `release.yml` | Alpha/Beta/Stable publishing | Schedule + manual dispatch |
| `benchmark.yml` | Performance benchmarks | Push/PR/weekly |
| Workflow | Purpose | Trigger |
| --------------- | -------------------------------------- | --------------------------------------------- |
| `ci.yml` | Lint + Test + Coverage + Wheel size | Push/PR to main/dev |
| `release.yml` | Candidate qualification and publishing | Schedule validates; manual dispatch publishes |
| `benchmark.yml` | Performance benchmarks | Push/PR/weekly |

### Release Cadence

- **Alpha** (Mon-Wed 2AM UTC): Automatic from `dev`, date+commit versioning
- **Beta** (Thursday 2AM UTC): Automatic from `dev`, incremental beta numbers
- **Stable** (manual dispatch): From `main`, base version or override
- **Alpha** (Mon-Wed 2AM UTC): Candidate qualification from `dev`, without publication
- **Beta** (Thursday 2AM UTC): Candidate qualification from `dev`, without publication
- **Publication** (manual dispatch): Dry run by default; explicit non-dry runs publish alpha/beta from `dev` or stable from `main`

### Release Pipeline

Expand Down Expand Up @@ -320,7 +320,7 @@ Before beginning any task please checkout a branch from `dev` and create a pull

### Release Preparation

- Alpha/beta releases are automated via `release.yml` schedule
- Alpha/beta candidate qualification is automated via `release.yml`; publication requires an explicit manual dispatch
- Stable releases: merge `dev` → `main`, then trigger `release.yml` with `stable` type
- Use `dry_run: true` to validate before actual publish
- Performance validation on realistic data sets
Expand Down
41 changes: 41 additions & 0 deletions CHANGELOG.MD
Original file line number Diff line number Diff line change
@@ -1,5 +1,46 @@
# ChangeLog

## [Unreleased]

## [4.12.0]

#### Security

- Preserve native structured PII detection in smart, spaCy, GLiNER, legacy and
CLI paths. Normalize lone surrogates without moving source spans; retain
structured findings on bounded model inputs. Require review when hook payloads
exceed scan budgets, rather than silently allowing unscanned tails.
- Bound user allowlist matching with timeouts and remove quadratic overlap and
Python fallback regex behavior. Redact exact NER offsets and whole TextService
inputs so repeated names and chunk boundaries cannot expose PII.
- Cover LiteLLM tool arguments, prompt/input requests and complete streamed
responses, with request-scoped tokens. Hide configured API keys in CLI output.

#### Changed

- Hash APIs honor the selected algorithm while preserving existing plain-hash
marker widths and defaults. Optional `hash_key` enables full-length HMAC with
a key of at least 32 bytes. Plain hashes provide deterministic pseudonyms and
do not resist guessing of low-entropy PII.
- Reuse tokens for repeated values, avoid existing token collisions, reject
invalid explicit ranges and unsupported entity selections, and report backend
entity capabilities. Async decorators await results before filtering.
- Require the reviewed Core 1.5 minor series (`datafog-core~=1.5.0`) in the base
package and `rust` extra; review native differences in `rust-1.5.0.json`.
Core's complete mapped-IPv6 spans, email boundaries, Unicode mailboxes and
phone-plus-ZIP classification are available through the wrapper. Published
Core artifacts passed dependent installed-wheel qualification.
- Raise the Pydantic floor to 2.1,
include spaCy in `nlp-advanced`, and allow Torch 2.x versions compatible with
`datafog-core[gliner2]`. Add an isolated build declaration and runnable sdist
tests; remove unused duplicate implementation modules.
- Repair Pydantic optional defaults, legacy anonymizer enum filters and overlaps,
Spark label aliases, implicit pipeline scanning and invalid CLI operations.
- Scheduled release jobs qualify candidates without publication. Manual runs
default to dry runs, reject stale version bases and development history, gate
Python 3.13, record release refs before upload, and synchronize stable metadata
back to development after publication.

## [4.11.0]

#### Changed
Expand Down
5 changes: 5 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include LICENSE README.md pyproject.toml tox.ini requirements-test.txt
recursive-include tests *.py *.json *.txt
recursive-include scripts *.py
prune tests/.datafog_env
global-exclude __pycache__ *.py[cod]
Loading
Loading