Repository navigation
✨ feat: generate PEP 770 SBOMs for Python wheels - #2
Merged
Merged
Conversation
Port the CycloneDX SBOM generator from pypa/virtualenv's hatch_build.py into a reusable hatchling plugin, with the project-specific parts (supplier, repository, bundled wheels) moved into hook options and a subclass method for per-wheel properties. The SPDX rendering and the validator ship as the hatch-pep770 spdx and validate commands.
Generate the SBOM from the wheel's METADATA so the hatchling hook and the new embed command share one code path, and expose the hook, the Project settings, the building blocks and the SPDX and validation helpers from the package root. Raise the floor to Python 3.11.
The distribution, import package, console script and hatch hook are now python-sbom (python_sbom for imports), and the build hook class is SbomBuildHook.
Join the build script path as a string so a drive letter survives on Windows, and admit simplint 0.7.1 past the seven-day cutoff so the cooled hook job can install it.
A pure-Python package needs every supported Python on Linux, the oldest and newest on Windows, the newest on macOS, and the tooling once. ty checks every platform in its single job instead.
The committed wheel tripped Scorecard's Binary-Artifacts check and could not be reviewed; a fixture now writes the same small wheel with fixed timestamps, so its hashes stay stable.
The core (collect, model, formats, wheel, validate, cli) no longer imports hatchling; the build hook moves to python_sbom.hatch behind a Backend ABC, with hatchling as the hatch extra. Build facts are gathered once at the edge and passed in. Also fixes five review findings: unique SPDX ids for conditional requirements, distinct refs for bundled wheel variants, marker-aware runtime dependencies, structural validation before deeper checks, and SPDX rendering of versionless roots.
Split settings and build facts, the document model, wheel archives, formats and validation into subpackages that import in one direction, with build_sbom as the only pipeline. A Backend now just returns the request, project content comes from a ContentProvider passed in, and the hatch hook writes the SBOM in finalize from the built wheel, so a later hook can no longer make it stale. Address the first review's correctness and security findings: bundled wheels are checked against their RECORD, embedded entries are the same on every OS and keep the wheel's mode, repeated requirements collapse, other tools' SBOMs get the schema check alone, and bad input raises an error the CLI reports instead of a traceback.
A custom hook loads the shipped hook from the source tree, since the package cannot be a build requirement of itself, and pkg_meta fails when the built wheel lacks .dist-info/sboms/.
Cover the content provider, the backends and the pipeline, the source revision chain, embedding after auditwheel, and why build tools are listed down to their files. Redraw the architecture diagram from the import graph, render type aliases and the TypedDicts, and keep hatchling's lifecycle methods out of the API reference.
Trim .gitignore and CODEOWNERS, run simplint between pyproject-fmt and ruff, point Homepage at the documentation and add the Python modules classifier.
The public API is now the content provider surface the hatchling hook calls, plus the types its signatures reach; embedding, archives, SPDX and validation run through the command line. A provider receives a context with the project root, settings, identity and a reader for the artifact, so one provider serves the wheel and an archive. The new archive command describes a zipapp with the same bundled-wheel identities the wheel SBOM uses. Address the second review: build facts survive egg-info and broken requirements, git is used only for a tracked project without running a parent repository's hooks, an sdist carries its commit, bundled wheels must match every RECORD row, decoy dist-info and duplicate members are refused, and validation reports hostile documents instead of raising.
Work in progress, split across agents from here: - core uses installer, packageurl-python, dep-logic, license-expression and packaging.direct_url; build-system requires them for the self-SBOM - validate caps components at 5,000, raised with --max-components - the sdist revision file is written as bytes, so it holds LF on Windows - bench/ benchmark harness scaffolding
for more information, see https://pre-commit.ci
A cap of zero rejects every SBOM with components and a negative one rejects all of them, and the cap also sets the size limit. argparse now reports anything but a positive integer as a usage error.
python-sbom spdx crashed with a traceback on a missing file, invalid JSON, input that is not UTF-8, deep nesting, or a timestamp that is no ISO date. It now reports each as an error, and refuses input over the size validate accepts at its default component cap, so the JSON parser never holds more than validate lets it.
The hook cached the revision for the whole process, so a build after a commit in the same process recorded the earlier commit and dirty state. Each build now reads it in initialize, before it writes the artifact, and finalize reuses that value.
initialize wrote the revision file into a directory from tempfile.mkdtemp that nothing removed, so every sdist build left one behind. finalize now removes it once the sdist holds the file. hatchling skips finalize when the build fails, so the directory then goes with the hook, at the latest at exit.
The resolver of the content provider costs a third of a millisecond on top of hatchling, so it loads with the hook instead of twice behind a suppression. finalize's build_data now carries one suppression per line, each under 120 columns.
The self-build hook delegated only finalize, so python-sbom's sdist carried no .python-sbom-revision and a wheel built from it named no commit. One helper now builds the SbomBuildHook for the whole build, and both initialize and finalize go to it.
The embed test and the retagged wheel test each asserted three behaviours; fixtures now run the command or build once per test, and each test checks one. The archive compression test takes its flagged archive from a fixture instead of patching bytes in its body.
The parser imported _validate._document, and with it _formats._spdx, only to show the --max-components default. The cap and the size limit it sets now live in _validate._limits, which imports nothing.
SettingsError and WheelError, and the ValueError and ImportError bases, had no caller that told them apart from SbomError. Every error is now an SbomError; MissingExtraError stays, as validation lets it through where it reports every other error per document.
The diagram missed _errors and version, drew the hook's lazy import of _generate as a load-time one, and lacked the CLI's imports of _wheel and, for types, _model. The text still described one pipeline, the generate package docstring named backends, the contributing guide pointed at the removed _cli/_commands.py, and the threat model claimed spdx caps nesting.
proselint read the diagram's model --> errors after wheel --> model as a repeated word, and covdefaults excludes only the bare TYPE_CHECKING guard, so the API test's typing.TYPE_CHECKING import counted as uncovered.
packageurl only lowercases a pypi name and turns underscores into dashes, so jaraco.context and jaraco_context got two purls, and a build tool's bom-ref, built from its normalized name, disagreed with its own purl (tool:pkg:pypi/boolean-py vs pkg:pypi/boolean.py). Names now go through canonicalize_name first, as before the switch to packageurl. The pinned SBOM changes only in the project's purl and the serial derived from it.
license-expression and its SPDX index took 9-10 ms of every cold validate, though only --allow-license reads them.
The census pushed every value of the document onto its stack. Only lists and objects go there now; numbers are judged as their container is read, and the depth check moves to the container, so a value still nests too deep exactly when it did. The census of virtualenv's zipapp SBOM drops from about 12 to 8 ms.
Each distribution's WHEEL went through the email parser for one field, and each Requires-Dist line was parsed and its marker evaluated again for every distribution declaring it. Root-Is-Purelib is now read from the header lines directly, agreeing with the email parser on 104,668 WHEEL files, and a requirement's verdict is cached for the process.
The SBOM followed the compression most of the archive's members used, so virtualenv's zipapp, which stores all of them, took it stored, megabytes of JSON. It now always goes in deflated at level 9, with the fixed time and attributes as before, and the other members keep their bytes.
The archive command now only parses options and prints, like embed; the orchestrator reads, builds and writes, and the archive's document is built by a pure model function as the wheel's is. The pipeline module folds into embed, its only user.
The built wheel's .whl members are read by _wheel and handed to the source tree reader, so embed no longer rebuilds the source tree to rename them.
Keeps the order the archive path asked content providers in before the model split, names the digest lookup for what it returns, and lets the benchmark find the bundled wheel reader in _wheel.
Two .dist-info in one directory left one distribution out of the archive SBOM and gave its files to the other, so that layout is now refused. Identical bundled wheel copies took one path in the built wheel and broke the build on a repeated bom-ref; each copy now takes its own. A .whl at the top level of an archive crashed generation; it is now described as a bundled wheel under the root component.
Fixes the second external review and restructures for review. The built artifact is now the only source of the bytes an SBOM describes: one reader keeps it open for the whole build, and bundled wheels are found in it by digest, so a source wheel changed or left out since the build is refused. One module defines what an artifact holds and ships and checks references, so generation, validation and the license policy judge the same set: top-level archive wheels meet the policy and get their vendored edges, and a repeated dependency entry is refused. A provider module another project already imported is refused. The wheel and archive pipelines now read alike: open the artifact, read it into a tree, map the tree to an Artifact with a pure function, assemble, write. Zip reading and writing are separate modules, one helper hashes, and the archive SBOM file name, bundled order and bundled marking each live in one place. Also: archive refuses to write over its own input, SPDX ids carry the full digest, RECORD rows match by parsed path, the scheduled check runs, malformed tools fail cleanly, archive ownership looks up ancestors, and docs/contributing.rst names what each layer guarantees and the test that holds it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Projects that ship wheels have no reusable way to put a software bill of materials where PEP 770 tells installers to keep it, in
.dist-info/sboms/. virtualenv carries its own generator, and the other tox-dev projects carry none. python-sbom turns that generator into a package: a hatchling project enables a build hook with one table inpyproject.toml, a project on another backend such as meson-python runspython-sbom embedon the built wheel, andpython-sbom archivedescribes other zip artifacts, such as virtualenv's zipapp. 📦The CycloneDX 1.6 document records the project, the runtime dependencies it declares, the wheels it bundles, and the build toolchain down to file hashes. Nothing about the build machine goes in, so a rebuild with the same inputs reproduces it byte for byte.
python-sbom spdxrenders it as SPDX 2.3.python-sbom validatetakes CycloneDX input only: it checks the document against the official CycloneDX schema, shipped with the package so validation works offline. For documents python-sbom wrote, it also checks the references and the SPDX rendering, plus an optional licence allow-list.Architecture
The hatch hook and the command line sit at the edge: they gather the options, the content provider, the build scripts and the build facts, then call the pipeline for their artifact. Each pipeline opens the wheel or archive once and reads it into a tree, then a pure function of the model maps that tree to components. One shared step adds what every artifact of the project carries, and the result is written back. Nothing below the hook imports hatchling, so the command line installs without it.
flowchart LR hook[hatch hook] --> pipelines cli[CLI: embed / archive / spdx / validate] --> pipelines pipelines[wheel and archive pipelines] --> collect[collect: settings, facts, revision, wheel or archive tree] collect --> reader[artifact reader: one checked handle per build] pipelines --> model[model: pure document building, shared graph rules] pipelines --> formats[CycloneDX JSON / SPDX 2.3] pipelines --> write[wheel and archive writing] provider[ContentProvider] --> pipelines cli --> validate[validate: schemas, references, licences] validate --> model classDef edge fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a; classDef core fill:#d8f3dc,stroke:#2d6a4f,color:#081c15; class hook,cli,provider edge; class pipelines,collect,reader,model,formats,write,validate core;The SBOM describes the bytes the artifact ships. Bundled wheels are matched to the artifact's own copies by digest and read from there, so a source wheel that changed after the build is refused rather than described. Generation, validation and the licence policy share one definition of what an artifact holds and one reference check, so they judge the same components wherever the document places them.
Projects extend the document through one seam: a
ContentProviderreceives aProviderContext(project root, settings, identity, and a reader for the artifact's members) and returnsContributions: extra components, dependencies, root properties and tools. virtualenv uses it to describe the pip and setuptools wheels it seeds, in both its wheel and its zipapp, with no private imports. The public API is that seam plus every type it exposes, and a test fails if a public signature reaches a private type.Behaviour worth knowing
.git_archival.txt; a modified checkout is marked dirty.--max-components).