diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 7c74548..2a2e19a 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -19,6 +19,7 @@ Found a Python behavior that surprised you? Please add it.
for f in examples/*.py; do python "$f" || echo "FAILED: $f"; done
```
-4. Open a pull request. CI runs the examples on CPython 3.10 to 3.14.
+4. If there's a documented real-world bug, CVE or fix caused by the pitfall, add an **In the wild** paragraph linking to it.
+5. Open a pull request. CI runs the examples on CPython 3.10 to 3.14.
If behavior differs between Python versions, say which versions in the README and guard the example with `sys.version_info`.
diff --git a/README.md b/README.md
index fd816e0..5f23cdd 100644
--- a/README.md
+++ b/README.md
@@ -6,13 +6,29 @@
Examining the gap between Python's readable interfaces and the underlying complexity, implicit state, and unexpected behavior they can conceal.
-A catalog of 52 Python gotchas, traps and surprising behaviors, from beginner-level surprises to asyncio, threading, CPython internals and data-engineering bugs. Each comes with a minimal reproduction, an explanation of *why* it happens, and the fix.
+A catalog of 59 Python gotchas, traps and surprising behaviors, from beginner-level surprises to asyncio, threading, CPython internals, data-engineering bugs and security vulnerabilities. Each comes with a minimal reproduction, an explanation of *why* it happens, the fix, and, where one exists, the real-world bug or CVE it caused.
Python's syntax prioritizes readability, but readable code does not always make runtime behavior predictable. Some pitfalls arise from implicit state, others from surprising semantics or abstractions that leak underlying details. John Ousterhout's *A Philosophy of Software Design* calls a particularly troublesome symptom of complexity *unknown unknowns*: behavior you cannot predict from the interface in front of you. When the language's mechanics don't match its readable syntax, developers may discover those differences only through runtime bugs.
-Here is a collection of Python pitfalls developers often discover the hard way.
+Here is a collection of Python pitfalls developers often discover the hard way, and the evidence that they don't stay in tutorials.
-> Every snippet here and in [`docs/`](docs/) has a runnable, self-checking twin in [`examples/`](examples/). CI runs them on CPython 3.10 to 3.14, so the outputs shown here are verified, not remembered.
+> Every pitfall except two security entries has a runnable, self-checking example in [`examples/`](examples/). CI runs them on CPython 3.10 to 3.14, so the outputs shown here are verified, not remembered.
+
+## How the complexity shows up in production
+
+Each row is a kind of hidden complexity, a pitfall that demonstrates the mechanism, and what it has caused in real code. Advisory counts are unique PyPI vulnerabilities published since 2023, from the [OSV](https://osv.dev) database ([method](docs/dependency-risk.md#method-and-limitations)).
+
+| Hidden complexity | Pitfalls | PyPI advisories since 2023 | Real-world cases |
+|---|---|---:|---|
+| A path join silently discards your prefix | [#53](docs/security.md#53-ospathjoin-discards-your-prefix), [#54](docs/security.md#54-tarfile-trusted-member-paths-until-python-314) | 507 (path traversal, the #1 weakness) | `tarfile` CVE-2007-4559, open for 15 years |
+| Parsing cost grows with the attacker's input | [#56](docs/security.md#56-nested-quantifiers-backtrack-exponentially), [#57](docs/security.md#57-decompression-bombs), [#42](docs/interpreter.md#42-strint-fails-above-4300-digits) | 549 (denial of service) | `urllib3` CVE-2021-33503, `idna` CVE-2024-3651, CPython CVE-2020-10735 |
+| Loading data runs code | [#58](docs/security.md#58-loading-data-can-run-code) | 210 (deserialization) | `torch.load` CVE-2025-32434; 58 bypasses of `picklescan` |
+| A convenience hides a protocol decision | [#55](docs/security.md#55-redirects-forward-credentials-to-another-host) | 60 (redirect-related leaks) | `requests` CVE-2018-18074, `urllib3` CVE-2023-43804 |
+| One parameter, two meanings | [#59](docs/security.md#59-one-key-parameter-two-meanings) | 41 (signature verification) | PyJWT: 15 key or algorithm advisories since 2023 |
+| Hash values change per process | [#43](docs/data-systems.md#43-hash-changes-between-processes) | | CPython CVE-2012-1150, the reason `hash()` is randomized |
+| Wall-clock vs elapsed time | [#48](docs/data-systems.md#48-aware-datetime-arithmetic-is-wall-clock-arithmetic), [#49](docs/data-systems.md#49-a-utc-offset-is-not-a-time-zone) | | Home Assistant: [alert loop](https://github.com/home-assistant/core/pull/188537), [backups an hour off](https://github.com/home-assistant/core/pull/188836) |
+| The event loop is shared by everything | [#20](docs/concurrency.md#20-fire-and-forget-tasks-get-garbage-collected), [#22](docs/concurrency.md#22-blocking-calls-freeze-the-event-loop), [#24](docs/concurrency.md#24-loop-methods-arent-thread-safe) | | Home Assistant's [guard rails](#in-the-wild-home-assistant) and [Hikvision fix](https://github.com/home-assistant/core/pull/160027) |
+| Code you never chose | [Dependency risk](docs/dependency-risk.md) | | `urllib3` and `certifi` reach 145 and 144 of the top 1,000 packages, 86–87% indirectly |
## Contents
@@ -54,9 +70,16 @@ Here is a collection of Python pitfalls developers often discover the hard way.
| [Object model & imports](docs/object-model.md) | 31–37 | Properties vs `__getattr__`, `@lru_cache` leaks, `__eq__` vs `__hash__`, `__exit__` swallowing errors, weak refs to bound methods, `mock.patch` targets, double-imported `__main__` |
| [Interpreter internals](docs/interpreter.md) | 38–42 | `python -O` stripping asserts, class-body comprehension scope, `locals()` writes, `id()` reuse, the 4300-digit `int`/`str` limit |
| [Data systems: time, numbers, bytes](docs/data-systems.md) | 43–52 | `hash()` sharding, NaN in containers, nanosecond timestamps in floats and `datetime`, float prices, DST arithmetic, fixed UTC offsets, `utcnow()`, `struct` alignment, non-standard JSON |
+| [Security: where complexity becomes a CVE](docs/security.md) | 53–59 | `os.path.join` with absolute paths, `tarfile` extraction, credentials forwarded on redirect, regex backtracking, decompression bombs, unsafe deserialization, signing-key confusion |
See also [In the wild](#in-the-wild-home-assistant) for how a large production codebase hits and guards against these.
+### Part III: Research
+
+| Page | Covers |
+|------|--------|
+| [Complexity in the wild](docs/dependency-risk.md) | What Python's vulnerability data shows, organized by mechanism: 7,032 OSV advisories joined with the dependency graphs of the top 1,000 packages, yanked releases, OpenSSF Scorecard and typosquatting data, with charts |
+
---
## Iterators & laziness
diff --git a/analysis/README.md b/analysis/README.md
new file mode 100644
index 0000000..4b29e11
--- /dev/null
+++ b/analysis/README.md
@@ -0,0 +1,38 @@
+# Dependency Risk Analysis
+
+Scripts behind [docs/dependency-risk.md](../docs/dependency-risk.md). They download public data, join it, and print every number used in the report.
+
+`dependency_risk.csv` is the joined snapshot from 30 October 2026: one row per package that at least 5 of the top 1,000 PyPI packages depend on, with reach, advisory counts, yanked releases and Scorecard.
+
+## Reproduce
+
+Downloaded data is untrusted, so keep it in its own directory and run the scripts from here with `python -I`. Network scripts need Python 3.11+ with working CA certificates. `fetch_yanked.py` also needs `packaging`, and `charts.py` needs `matplotlib`.
+
+```sh
+DATA=/tmp/pypi-risk-data
+mkdir -p $DATA/osv $DATA/top $DATA/deps $DATA/simple $DATA/scorecard
+
+curl -L -o $DATA/osv/all.zip https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip
+unzip -q $DATA/osv/all.zip -d $DATA/osv/json
+curl -L -o $DATA/top/top30d.json https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json
+
+python -I osv_agg.py $DATA # Merge advisories into unique vulnerabilities
+python -I fetch_deps.py $DATA 1000 # Resolved dependency graphs from deps.dev
+python -I fetch_yanked.py $DATA 1000 # Yanked releases from PyPI (needs packaging)
+python -I analyze.py $DATA # Reach x vulnerabilities x yanks
+python -I fetch_scorecard.py $DATA # OpenSSF Scorecard for the hidden layer
+python -I report.py $DATA # Every table in the report
+python -I charts.py $DATA ../docs/img # Light and dark SVG charts (needs matplotlib)
+```
+
+The fetch scripts cache one JSON file per package, so they can be rerun after a failure. Expect about five minutes in total. Results will drift as OSV, PyPI and deps.dev update.
+
+| Script | Output |
+|---|---|
+| `osv_agg.py` | `vulns.json`: unique vulnerabilities (aliases merged) and malicious-package counts |
+| `fetch_deps.py` | `deps/*.json`: resolved graph per top package, falling back to a recent release if deps.dev hasn't resolved the newest |
+| `fetch_yanked.py` | `simple/*.json`: release count and yanked versions with reasons |
+| `analyze.py` | `joined.json`: per-package reach, direct and indirect counts, advisories, yanks |
+| `fetch_scorecard.py` | `scorecard/*.json`: Scorecard for packages with reach ≥ 20 and mostly indirect use |
+| `report.py` | Prints the hidden-layer, burstiness, CWE, yanked, malicious-package, monthly and Scorecard tables |
+| `charts.py` | `docs/img/*-{light,dark}.svg`: advisories per year, reach vs advisories, weakness mix. Colors are a palette validated for color-vision deficiency in both themes |
diff --git a/analysis/analyze.py b/analysis/analyze.py
new file mode 100644
index 0000000..6700d1a
--- /dev/null
+++ b/analysis/analyze.py
@@ -0,0 +1,69 @@
+"""Join vulnerabilities, dependency reach, downloads and yanked releases."""
+import collections, glob, json, os, re, sys
+
+data = sys.argv[1]
+norm = lambda n: re.sub(r"[-_.]+", "-", n).lower()
+top = json.load(open(f"{data}/top/top30d.json"))["rows"]
+downloads = {norm(r["project"]): r["download_count"] for r in top}
+rank = {norm(r["project"]): i + 1 for i, r in enumerate(top)}
+roots = {norm(r["project"]) for r in top[:1000]}
+
+# Reach: how many top-1000 packages pull each package in, directly or only indirectly.
+direct, indirect, nodes_per_root = collections.defaultdict(set), collections.defaultdict(set), []
+for f in glob.glob(f"{data}/deps/*.json"):
+ d = json.load(open(f))
+ if "graph" not in d:
+ continue
+ root = norm(os.path.basename(f)[:-5])
+ if root not in roots: # Ignore graphs cached for packages no longer in the top 1,000.
+ continue
+ nodes = d["graph"]["nodes"]
+ nodes_per_root.append(len(nodes) - 1)
+ for node in nodes:
+ name = norm(node["versionKey"]["name"])
+ if node["relation"] == "DIRECT":
+ direct[name].add(root)
+ elif node["relation"] == "INDIRECT":
+ indirect[name].add(root)
+reach = {p: direct[p] | indirect[p] for p in set(direct) | set(indirect)}
+only_indirect = {p: reach[p] - direct[p] for p in reach}
+
+v = json.load(open(f"{data}/vulns.json"))
+vulns, malicious = v["vulns"], v["malicious"]
+by_pkg = collections.defaultdict(list)
+for x in vulns:
+ for p in x["pkgs"]:
+ by_pkg[p].append(x)
+
+yanked = {}
+for f in glob.glob(f"{data}/simple/*.json"):
+ d = json.load(open(f))
+ if "error" not in d:
+ yanked[norm(os.path.basename(f)[:-5])] = d
+
+def row(p):
+ vs = by_pkg.get(p, [])
+ recent = [x for x in vs if x["published"] >= "2023-01-01"]
+ y = yanked.get(p, {})
+ return dict(
+ pkg=p, rank=rank.get(p), downloads=downloads.get(p, 0),
+ reach=len(reach.get(p, ())), direct=len(direct.get(p, ())), only_indirect=len(only_indirect.get(p, ())),
+ vulns=len(vs), vulns_2023=len(recent),
+ high_crit=sum(1 for x in vs if x["sev"] >= 3),
+ cwes=collections.Counter(c for x in vs for c in x["cwes"]).most_common(3),
+ yanked=len(y.get("yanked", [])), versions=y.get("n_versions"),
+ )
+
+rows = [row(p) for p in reach]
+json.dump({"rows": rows, "roots_with_graph": len(nodes_per_root),
+ "median_deps": sorted(nodes_per_root)[len(nodes_per_root) // 2],
+ "mean_deps": sum(nodes_per_root) / len(nodes_per_root)}, open(f"{data}/joined.json", "w"))
+
+print(f"roots with graphs: {len(nodes_per_root)}, median transitive deps {sorted(nodes_per_root)[len(nodes_per_root)//2]}, mean {sum(nodes_per_root)/len(nodes_per_root):.1f}")
+fmt = "{pkg:22} rank={rank!s:>5} reach={reach:>4} direct={direct:>4} only_ind={only_indirect:>4} vulns={vulns:>3} since23={vulns_2023:>3} hi/crit={high_crit:>3} yanked={yanked:>3}/{versions!s:<4} {cwes}"
+print("\n== Dependencies with reach >= 10, by all-time vulns")
+for r in sorted((r for r in rows if r["reach"] >= 10 and r["vulns"]), key=lambda r: -r["vulns"])[:30]:
+ print(fmt.format(**r))
+print("\n== Exposure (vulns since 2023 x reach), reach >= 10")
+for r in sorted((r for r in rows if r["reach"] >= 10 and r["vulns_2023"]), key=lambda r: -r["vulns_2023"] * r["reach"])[:20]:
+ print(fmt.format(**r), "score=", r["vulns_2023"] * r["reach"])
diff --git a/analysis/charts.py b/analysis/charts.py
new file mode 100644
index 0000000..51c8ce5
--- /dev/null
+++ b/analysis/charts.py
@@ -0,0 +1,211 @@
+"""Render the dependency-risk charts as light and dark SVGs.
+
+Usage: python -I charts.py DATA_DIR OUT_DIR (needs matplotlib)
+
+Reads vulns.json and joined.json produced by osv_agg.py and analyze.py.
+Colors are the validated reference palette (blue/orange pass all-pairs CVD
+checks in both modes); text uses ink tokens, never series colors.
+"""
+
+import collections
+import json
+import sys
+
+import matplotlib
+
+matplotlib.use("svg")
+import matplotlib.pyplot as plt # noqa: E402
+from matplotlib.path import Path # noqa: E402
+from matplotlib.patches import PathPatch # noqa: E402
+from matplotlib.transforms import IdentityTransform # noqa: E402
+
+data, out = sys.argv[1], sys.argv[2]
+
+THEMES = {
+ "light": dict(surface="#fcfcfb", ink="#0b0b0b", ink2="#52514e", muted="#898781",
+ grid="#e1e0d9", axis="#c3c2b7", s1="#2a78d6", s2="#eb6834"),
+ "dark": dict(surface="#1a1a19", ink="#f0efec", ink2="#c3c2b7", muted="#898781",
+ grid="#2c2c2a", axis="#383835", s1="#3987e5", s2="#d95926"),
+}
+plt.rcParams.update({
+ "svg.fonttype": "none",
+ "font.family": "sans-serif",
+ "font.sans-serif": ["Helvetica", "Arial", "DejaVu Sans"],
+ "font.size": 10,
+})
+
+DOS = {"CWE-400", "CWE-770", "CWE-1333", "CWE-407", "CWE-409", "CWE-835", "CWE-674",
+ "CWE-1050", "CWE-834", "CWE-789", "CWE-776"}
+CLASSES = [
+ ("Resource exhaustion (DoS)", DOS),
+ ("Information exposure", {"CWE-200", "CWE-201", "CWE-209", "CWE-532", "CWE-522", "CWE-669", "CWE-359"}),
+ ("Auth & signature checks", {"CWE-287", "CWE-285", "CWE-862", "CWE-863", "CWE-347", "CWE-345",
+ "CWE-295", "CWE-306", "CWE-639", "CWE-1390"}),
+ ("Path traversal", {"CWE-22", "CWE-23", "CWE-24", "CWE-29", "CWE-36", "CWE-73"}),
+ ("Code & command injection", {"CWE-94", "CWE-95", "CWE-77", "CWE-78", "CWE-89", "CWE-1336", "CWE-74", "CWE-917"}),
+ ("Cross-site scripting", {"CWE-79", "CWE-80"}),
+ ("Server-side request forgery", {"CWE-918"}),
+ ("Unsafe deserialization", {"CWE-502"}),
+ ("HTTP request smuggling", {"CWE-444"}),
+]
+
+vulns = json.load(open(f"{data}/vulns.json"))["vulns"]
+rows = {r["pkg"]: r for r in json.load(open(f"{data}/joined.json"))["rows"]}
+deps = {p for p, r in rows.items() if r["reach"] >= 10}
+
+
+def style_axes(ax, t):
+ ax.set_facecolor(t["surface"])
+ for side in ("top", "right", "left"):
+ ax.spines[side].set_visible(False)
+ ax.spines["bottom"].set_color(t["axis"])
+ ax.tick_params(colors=t["muted"], length=0, labelsize=9)
+
+
+def rounded_hbar(ax, y, x0, x1, height, color, radius=3.0):
+ """Horizontal bar, square at the baseline, rounded at the data end (display units)."""
+ (ax0, ay0), (ax1, ay1) = ax.transData.transform([(x0, y - height / 2), (x1, y + height / 2)])
+ # Sort, since an inverted axis flips display order (and would give a negative radius).
+ px0, px1, py0, py1 = min(ax0, ax1), max(ax0, ax1), min(ay0, ay1), max(ay0, ay1)
+ r = min(radius, (px1 - px0) / 2, (py1 - py0) / 2)
+ k = 0.5523 * r # Bezier circle approximation.
+ verts = [(px0, py0), (px1 - r, py0), (px1 - r + k, py0), (px1, py0 + r - k), (px1, py0 + r),
+ (px1, py1 - r), (px1, py1 - r + k), (px1 - r + k, py1), (px1 - r, py1), (px0, py1), (px0, py0)]
+ codes = [Path.MOVETO, Path.LINETO, Path.CURVE4, Path.CURVE4, Path.CURVE4,
+ Path.LINETO, Path.CURVE4, Path.CURVE4, Path.CURVE4, Path.LINETO, Path.CLOSEPOLY]
+ ax.add_patch(PathPatch(Path(verts, codes), facecolor=color, edgecolor="none", transform=IdentityTransform()))
+
+
+def rounded_vbar(ax, x, y0, y1, width, color, radius=3.0):
+ """Vertical bar, square at the baseline, rounded at the top."""
+ (ax0, ay0), (ax1, ay1) = ax.transData.transform([(x - width / 2, y0), (x + width / 2, y1)])
+ px0, px1, py0, py1 = min(ax0, ax1), max(ax0, ax1), min(ay0, ay1), max(ay0, ay1)
+ r = min(radius, (px1 - px0) / 2, (py1 - py0) / 2)
+ k = 0.5523 * r
+ verts = [(px0, py0), (px0, py1 - r), (px0, py1 - r + k), (px0 + r - k, py1), (px0 + r, py1),
+ (px1 - r, py1), (px1 - r + k, py1), (px1, py1 - r + k), (px1, py1 - r), (px1, py0), (px0, py0)]
+ codes = [Path.MOVETO, Path.LINETO, Path.CURVE4, Path.CURVE4, Path.CURVE4,
+ Path.LINETO, Path.CURVE4, Path.CURVE4, Path.CURVE4, Path.LINETO, Path.CLOSEPOLY]
+ ax.add_patch(PathPatch(Path(verts, codes), facecolor=color, edgecolor="none", transform=IdentityTransform()))
+
+
+def weakness_mix(mode):
+ t = THEMES[mode]
+ recent = [v for v in vulns if v["published"] >= "2023" and v["cwes"]]
+ pops = {"dependencies": [v for v in recent if deps & set(v["pkgs"])], "all": recent}
+ share = {k: [100 * sum(1 for v in vs if cls & set(v["cwes"])) / len(vs) for _, cls in CLASSES]
+ for k, vs in pops.items()}
+ order = sorted(range(len(CLASSES)), key=lambda i: -share["dependencies"][i])
+
+ fig = plt.figure(figsize=(8, 4.6), facecolor=t["surface"])
+ ax = fig.add_axes([0.30, 0.10, 0.66, 0.74])
+ style_axes(ax, t)
+ n = len(CLASSES)
+ ax.set_xlim(0, 40)
+ ax.set_ylim(n - 0.5, -0.5)
+ ax.set_yticks(range(n), [CLASSES[i][0] for i in order], color=t["ink2"], fontsize=9.5)
+ ax.set_xticks([0, 10, 20, 30, 40], ["0%", "10%", "20%", "30%", "40%"])
+ ax.grid(axis="x", color=t["grid"], linewidth=0.75)
+ ax.set_axisbelow(True)
+ fig.canvas.draw()
+ bar = 0.30 # Two bars per band with a gap between them; each well under 24px.
+ for row, i in enumerate(order):
+ rounded_hbar(ax, row - bar / 2 - 0.02, 0, share["dependencies"][i], bar, t["s1"])
+ rounded_hbar(ax, row + bar / 2 + 0.02, 0, share["all"][i], bar, t["s2"])
+ # Selective labels: only the headline pair.
+ i = order[0]
+ ax.text(share["dependencies"][i] + 0.6, 0 - bar / 2 - 0.02, f"{share['dependencies'][i]:.0f}%",
+ va="center", fontsize=9, color=t["ink"], fontweight="bold")
+ ax.text(share["all"][i] + 0.6, 0 + bar / 2 + 0.02, f"{share['all'][i]:.0f}%", va="center", fontsize=9, color=t["ink2"])
+
+ fig.text(0.02, 0.95, "How Python vulnerabilities fail: dependencies vs all of PyPI",
+ fontsize=12.5, fontweight="bold", color=t["ink"])
+ fig.text(0.02, 0.905, f"Share of advisories published since 2023 with each weakness type (CWE); an advisory can have several. "
+ f"Dependencies: {len(pops['dependencies'])}; all PyPI: {len(pops['all']):,}.",
+ fontsize=8.5, color=t["muted"])
+ legend(fig, t, [(t["s1"], "Dependencies pulled in by ≥10 of the top 1,000 packages"), (t["s2"], "All PyPI")], y=0.035)
+ fig.savefig(f"{out}/weakness-mix-{mode}.svg", facecolor=t["surface"])
+ plt.close(fig)
+ return {CLASSES[i][0]: (round(share["dependencies"][i], 1), round(share["all"][i], 1)) for i in order}
+
+
+def legend(fig, t, items, y):
+ x = 0.02
+ for color, label in items:
+ fig.patches.append(matplotlib.patches.FancyBboxPatch((x, y), 0.014, 0.022, boxstyle="round,pad=0,rounding_size=0.004",
+ transform=fig.transFigure, facecolor=color, edgecolor="none"))
+ txt = fig.text(x + 0.022, y + 0.011, label, fontsize=9, color=t["ink2"], va="center")
+ fig.canvas.draw()
+ x = fig.transFigure.inverted().transform(txt.get_window_extent().get_points())[1][0] + 0.03
+
+
+def reach_scatter(mode):
+ t = THEMES[mode]
+ pts = [r for r in rows.values() if r["reach"] >= 10]
+ hidden = lambda r: r["only_indirect"] / r["reach"] >= 0.5
+
+ fig = plt.figure(figsize=(8, 5.2), facecolor=t["surface"])
+ ax = fig.add_axes([0.09, 0.15, 0.87, 0.68])
+ style_axes(ax, t)
+ ax.spines["left"].set_visible(True)
+ ax.spines["left"].set_color(t["axis"])
+ ax.set_yscale("symlog", linthresh=1, linscale=0.6)
+ ax.set_ylim(-0.25, 110)
+ ax.set_xlim(0, 240)
+ ax.set_yticks([0, 1, 10, 100], ["0", "1", "10", "100"])
+ ax.grid(color=t["grid"], linewidth=0.75)
+ ax.set_axisbelow(True)
+ ax.set_xlabel("Reach: how many of the top 1,000 PyPI packages depend on it", color=t["ink2"], fontsize=9.5)
+ ax.set_ylabel("Advisories since 2023", color=t["ink2"], fontsize=9.5)
+ for group, color in ((False, t["s2"]), (True, t["s1"])):
+ sel = [r for r in pts if hidden(r) == group]
+ ax.scatter([r["reach"] for r in sel], [r["vulns_2023"] for r in sel], s=46, color=color,
+ edgecolors=t["surface"], linewidths=1.5, zorder=3)
+ # Offsets in points; negative dx right-aligns the label to the left of its dot.
+ labels = {"urllib3": (6, 4), "cryptography": (6, 4), "pillow": (6, 0), "aiohttp": (6, -4),
+ "requests": (-7, 3), "pyjwt": (6, 1), "certifi": (6, -3), "idna": (6, 3), "filelock": (6, -9),
+ "tornado": (-7, -2), "pyasn1": (6, 2), "anyio": (6, 2), "typing-extensions": (-6, 7),
+ "charset-normalizer": (-6, 7)}
+ for r in pts:
+ if r["pkg"] in labels:
+ dx, dy = labels[r["pkg"]]
+ ax.annotate(r["pkg"], (r["reach"], r["vulns_2023"]), xytext=(dx, dy), textcoords="offset points",
+ fontsize=8.5, color=t["ink2"], ha="left" if dx > 0 else "right", va="bottom", zorder=4)
+ fig.text(0.02, 0.95, "Reach vs advisories: the hidden layer sits far right", fontsize=12.5, fontweight="bold", color=t["ink"])
+ fig.text(0.02, 0.905, f"{len(pts)} packages that at least 10 of the top 1,000 depend on. Vertical axis is logarithmic above 1.",
+ fontsize=8.5, color=t["muted"])
+ legend(fig, t, [(t["s1"], "Hidden layer: mostly pulled in indirectly"), (t["s2"], "Mostly a direct dependency")], y=0.035)
+ fig.savefig(f"{out}/reach-vs-advisories-{mode}.svg", facecolor=t["surface"])
+ plt.close(fig)
+
+
+def per_year(mode):
+ t = THEMES[mode]
+ counts = collections.Counter(v["published"][:4] for v in vulns)
+ years = [str(y) for y in range(2016, 2027)]
+ fig = plt.figure(figsize=(8, 3.8), facecolor=t["surface"])
+ ax = fig.add_axes([0.09, 0.12, 0.88, 0.68])
+ style_axes(ax, t)
+ ax.set_xlim(-0.6, len(years) - 0.4)
+ ax.set_ylim(0, 3000)
+ ax.set_xticks(range(len(years)), years)
+ ax.set_yticks([0, 1000, 2000, 3000], ["0", "1,000", "2,000", "3,000"])
+ ax.grid(axis="y", color=t["grid"], linewidth=0.75)
+ ax.set_axisbelow(True)
+ fig.canvas.draw()
+ for i, y in enumerate(years):
+ rounded_vbar(ax, i, 0, counts[y], 0.42, t["s1"])
+ ax.text(len(years) - 1, counts["2026"] + 60, f"{counts['2026']:,}\nto 30 Oct", ha="center", va="bottom", fontsize=9, color=t["ink"], fontweight="bold")
+ ax.text(len(years) - 2, counts["2025"] + 60, f"{counts['2025']:,}", ha="center", va="bottom", fontsize=9, color=t["ink2"])
+ fig.text(0.02, 0.93, "Unique Python vulnerabilities published per year", fontsize=12.5, fontweight="bold", color=t["ink"])
+ fig.text(0.02, 0.875, "PyPI advisories in OSV, duplicates across databases merged. 2026 is a partial year.", fontsize=8.5, color=t["muted"])
+ fig.savefig(f"{out}/advisories-per-year-{mode}.svg", facecolor=t["surface"])
+ plt.close(fig)
+ return {y: counts[y] for y in years}
+
+
+for mode in THEMES:
+ mix = weakness_mix(mode)
+ reach_scatter(mode)
+ years = per_year(mode)
+print(json.dumps({"weakness_mix": mix, "per_year": years}, indent=1))
diff --git a/analysis/dependency_risk.csv b/analysis/dependency_risk.csv
new file mode 100644
index 0000000..a951f2b
--- /dev/null
+++ b/analysis/dependency_risk.csv
@@ -0,0 +1,223 @@
+package,download_rank,reach_top1000,direct,indirect_only,vulns_all,vulns_since_2023,vulns_2026,high_or_critical,yanked_releases,releases,scorecard
+typing-extensions,3,233,96,137,0,0,0,0,0,52,8.7
+idna,4,177,8,169,2,2,1,0,0,42,8.5
+urllib3,5,145,21,124,22,14,6,12,4,105,8.6
+certifi,6,144,19,125,4,3,1,2,1,78,6.3
+packaging,2,140,72,68,0,0,0,0,1,54,
+charset-normalizer,8,125,4,121,0,0,0,0,1,66,
+requests,7,121,62,59,8,4,1,1,2,160,
+pycparser,19,100,1,99,0,0,0,0,0,27,5.9
+cffi,11,99,8,91,0,0,0,0,2,80,5.1
+six,15,83,9,74,0,0,0,0,0,29,3.7
+cryptography,9,82,25,57,24,19,7,12,3,160,8.3
+opentelemetry-api,60,72,44,28,1,1,1,0,0,87,
+anyio,22,72,18,54,3,3,3,2,1,72,5.6
+python-dateutil,14,71,27,44,0,0,0,0,0,27,4.4
+h11,27,69,4,65,1,1,0,1,0,13,4.4
+protobuf,32,63,39,24,5,2,1,4,5,223,
+pydantic-core,21,61,1,60,0,0,0,0,3,162,6.8
+annotated-types,26,61,1,60,0,0,0,0,0,9,4.2
+typing-inspection,28,61,7,54,0,0,0,0,0,10,
+pydantic,18,60,47,13,2,1,0,0,2,210,
+pyyaml,13,58,31,27,4,0,0,4,1,31,
+attrs,29,58,14,44,0,0,0,0,1,36,8.8
+pygments,12,56,10,46,5,2,1,3,0,69,6.2
+numpy,23,53,46,7,8,0,0,5,1,138,
+click,20,47,26,21,0,0,0,0,1,65,
+tzdata,73,44,5,39,0,0,0,0,4,39,7
+pyasn1,65,43,5,38,5,5,5,5,0,42,6.6
+opentelemetry-semantic-conventions,102,40,20,20,0,0,0,0,0,64,8.2
+truststore,124,39,3,36,0,0,0,0,1,17,5.7
+pyasn1-modules,84,38,2,36,0,0,0,0,0,27,
+markupsafe,34,38,5,33,0,0,0,0,0,36,5.6
+httpcore2,130,37,1,36,1,1,1,1,0,22,
+google-auth,75,36,27,9,0,0,0,0,3,207,
+wrapt,70,36,17,19,0,0,0,0,2,122,3.8
+httpx2,129,36,9,27,5,5,5,2,0,22,
+googleapis-common-protos,64,34,7,27,0,0,0,0,1,87,8.3
+platformdirs,38,33,13,20,0,0,0,0,0,94,
+httpx,33,33,18,15,1,0,0,1,0,76,
+httpcore,35,33,0,33,0,0,0,0,0,64,4.7
+rpds-py,49,31,2,29,0,0,0,0,2,72,
+referencing,56,30,5,25,0,0,0,0,0,108,6.7
+jinja2,41,30,17,13,10,5,0,3,0,52,
+grpcio,89,29,21,8,6,5,0,4,11,284,
+jsonschema-specifications,57,28,2,26,0,0,0,0,0,25,4.5
+mdurl,58,28,1,27,0,0,0,0,1,4,3.4
+sniffio,72,28,9,19,0,0,0,0,0,6,4
+jsonschema,48,27,14,13,0,0,0,0,1,109,
+markdown-it-py,54,27,3,24,2,2,0,2,0,45,5.3
+rich,52,25,16,9,0,0,0,0,1,207,
+proto-plus,113,24,16,8,0,0,0,0,0,86,
+websockets,68,24,6,18,2,0,0,2,0,55,5.5
+google-api-core,101,23,21,2,0,0,0,0,3,175,
+multidict,47,23,4,19,1,1,1,0,8,156,7.3
+propcache,55,22,2,20,0,0,0,0,2,13,
+tenacity,81,22,8,14,0,0,0,0,0,61,6.7
+yarl,50,21,6,15,0,0,0,0,7,137,6.4
+opentelemetry-instrumentation,185,20,18,2,1,1,0,1,0,76,
+pluggy,16,20,5,15,0,0,0,0,1,25,6.8
+opentelemetry-sdk,109,20,16,4,0,0,0,0,0,86,
+requests-toolbelt,144,20,5,15,0,0,0,0,0,20,4.7
+fsspec,39,19,9,10,1,1,1,1,3,102,
+distro,108,19,4,15,0,0,0,0,0,16,
+traitlets,243,19,13,6,0,0,0,0,2,56,
+pyjwt,37,19,10,9,25,23,21,11,0,58,
+jmespath,42,19,5,14,0,0,0,0,0,27,
+filelock,44,18,8,10,20,20,19,1,2,138,
+grpcio-status,85,18,2,16,0,0,0,0,10,209,
+python-dotenv,36,18,8,10,1,1,1,0,0,52,
+jsonpointer,224,18,1,17,0,0,0,0,0,35,
+xxhash,219,17,4,13,0,0,0,0,3,54,
+zstandard,177,17,3,14,0,0,0,0,0,42,
+pillow,59,17,14,3,119,66,60,88,0,108,
+frozenlist,61,17,2,15,0,0,0,0,1,18,
+isodate,160,16,11,5,0,0,0,0,1,18,
+wcwidth,118,16,3,13,0,0,0,0,0,49,
+aiosignal,66,16,1,15,0,0,0,0,0,10,
+aiohappyeyeballs,67,16,1,15,0,0,0,0,1,46,
+setuptools,10,16,11,5,6,4,2,4,12,627,
+tqdm,62,15,11,4,2,1,0,1,0,150,
+orjson,153,15,3,12,2,2,1,2,0,151,
+defusedxml,143,15,7,8,0,0,0,0,1,13,
+iniconfig,25,15,1,14,0,0,0,0,0,11,
+botocore,17,15,10,5,0,0,0,0,0,2543,
+aiohttp,51,15,11,4,55,54,40,7,7,314,
+pandas,40,14,12,2,1,0,0,0,1,120,
+pytest,24,14,14,0,1,1,1,0,1,193,
+uuid-utils,329,14,2,12,0,0,0,0,0,36,
+soupsieve,98,14,1,13,4,4,4,2,2,61,
+jupyter-core,312,13,8,5,2,1,0,2,2,63,
+tornado,262,13,6,7,26,24,15,13,0,90,
+oauthlib,127,13,2,11,4,3,3,1,0,53,
+langchain-protocol,411,13,2,11,0,0,0,0,0,13,
+beautifulsoup4,99,13,7,6,0,0,0,0,1,54,
+annotated-doc,69,13,2,11,0,0,0,0,0,5,
+jsonpatch,256,13,1,12,0,0,0,0,0,47,
+cloudpickle,163,13,4,9,0,0,0,0,0,41,
+pyparsing,90,13,6,7,0,0,0,0,0,86,
+langsmith,307,13,3,10,4,4,4,2,9,549,
+webencodings,246,13,5,8,0,0,0,0,0,8,
+ptyprocess,229,13,2,11,0,0,0,0,0,10,
+fastjsonschema,254,12,2,10,0,0,0,0,1,48,
+starlette,45,12,5,7,10,10,5,5,2,206,
+langchain-core,195,12,12,0,9,9,4,4,4,318,
+azure-core,173,12,11,1,1,1,1,1,1,81,
+psutil,103,12,5,7,1,0,0,1,0,102,
+uvicorn,46,12,5,7,2,0,0,2,1,204,
+opentelemetry-proto,133,12,5,7,0,0,0,0,0,76,
+pyzmq,284,11,3,8,0,0,0,0,0,107,
+s3transfer,30,11,1,10,0,0,0,0,1,64,
+google-cloud-core,176,11,6,5,0,0,0,0,0,59,
+requests-oauthlib,115,11,4,7,0,0,0,0,3,23,
+opentelemetry-util-http,250,11,10,1,0,0,0,0,0,66,
+narwhals,247,11,4,7,0,0,0,0,5,246,
+more-itertools,125,11,5,6,0,0,0,0,1,57,
+hyperframe,198,11,1,10,0,0,0,0,0,21,
+hpack,193,11,1,10,2,1,1,1,0,13,
+msgpack,145,11,2,9,2,2,2,2,0,36,
+grpc-google-iam-v1,264,11,8,3,0,0,0,0,0,25,
+shellingham,110,11,2,9,0,0,0,0,2,23,
+sqlparse,252,10,5,5,9,8,6,5,0,39,
+jupyter-client,327,10,3,7,0,0,0,0,3,99,
+boto3,1,10,8,2,0,0,0,0,0,2145,
+scipy,97,10,9,1,3,2,0,1,2,103,
+h2,192,10,1,9,2,2,1,0,0,47,
+sqlalchemy,87,10,8,2,3,0,0,3,3,338,
+pathspec,43,10,6,4,0,0,0,0,1,38,
+prompt-toolkit,151,10,4,6,0,0,0,0,0,137,
+babel,260,9,3,6,1,0,0,1,0,38,
+nbformat,316,9,4,5,0,0,0,0,3,42,
+pytz,71,9,7,2,0,0,0,0,0,126,
+typer,114,9,7,2,0,0,0,0,0,91,
+cachetools,119,9,5,4,0,0,0,0,1,84,
+google-crc32c,207,9,3,6,0,0,0,0,1,30,
+tinycss2,285,9,2,7,0,0,0,0,0,18,
+zipp,76,9,1,8,1,1,0,0,0,66,
+rfc3339-validator,377,9,2,7,0,0,0,0,0,5,
+websocket-client,167,8,3,5,0,0,0,0,0,90,
+pyarrow,86,8,6,2,5,3,1,4,0,65,
+mistune,372,8,2,6,21,18,18,8,0,57,
+pydantic-settings,79,8,5,3,1,1,1,0,0,48,
+werkzeug,149,8,6,2,15,10,3,4,0,106,
+hf-xet,148,8,1,7,0,0,0,0,2,51,
+importlib-metadata,78,8,3,5,0,0,0,0,2,133,
+nbclient,416,8,2,6,0,0,0,0,4,46,
+terminado,591,7,2,5,0,0,0,0,1,36,
+lxml,91,7,6,1,7,1,1,1,0,130,
+pure-eval,280,7,1,6,0,0,0,0,0,10,
+regex,93,7,4,3,0,0,0,0,1,369,
+opentelemetry-exporter-otlp-proto-common,142,7,2,5,0,0,0,0,0,39,
+huggingface-hub,122,7,6,1,0,0,0,0,8,338,
+nvidia-nvjitlink,667,7,3,4,0,0,0,0,0,12,
+networkx,141,7,4,3,0,0,0,0,2,87,
+mypy-extensions,111,7,4,3,0,0,0,0,0,10,
+python-json-logger,291,7,1,6,0,0,0,0,0,38,
+tomlkit,53,7,5,2,0,0,0,0,0,57,
+python-multipart,83,7,2,5,9,9,7,5,2,32,
+argon2-cffi-bindings,364,7,1,6,0,0,0,0,0,4,
+asttokens,272,7,1,6,0,0,0,0,1,36,
+parso,267,7,1,6,1,0,0,0,0,30,
+prometheus-client,216,7,2,5,0,0,0,0,1,60,
+colorama,112,7,4,3,0,0,0,0,1,46,
+pandocfilters,431,7,1,6,0,0,0,0,0,16,
+google-resumable-media,209,7,2,5,0,0,0,0,1,56,
+nvidia-cuda-nvrtc,701,7,2,5,0,0,0,0,0,14,
+itsdangerous,175,7,2,5,0,0,0,0,0,27,
+sortedcontainers,135,7,4,3,0,0,0,0,0,40,
+bleach,373,7,1,6,8,3,3,2,0,57,
+jupyterlab-pygments,433,7,1,6,0,0,0,0,0,7,
+docstring-parser,137,7,4,3,0,0,0,0,0,26,
+rfc3986-validator,554,7,1,6,0,0,0,0,0,2,
+asgiref,238,7,3,4,0,0,0,0,1,73,
+executing,259,7,1,6,0,0,0,0,1,37,
+ipython-pygments-lexers,390,6,1,5,0,0,0,0,0,3,
+argon2-cffi,375,6,1,5,0,0,0,0,0,18,
+smmap,187,6,1,5,0,0,0,0,1,18,
+jedi,268,6,1,5,0,0,0,0,0,37,
+dnspython,126,6,2,4,2,2,1,1,0,32,
+jupyter-events,605,6,1,5,0,0,0,0,0,17,
+send2trash,563,6,1,5,0,0,0,0,0,18,
+nbconvert,422,6,1,5,5,4,3,2,1,104,
+jupyter-server-terminals,609,6,1,5,0,0,0,0,0,16,
+fonttools,146,6,3,3,2,2,0,1,2,238,
+jaraco-functools,201,6,2,4,0,0,0,0,0,52,
+opentelemetry-exporter-otlp-common,1344,6,2,4,0,0,0,0,0,,
+text-unidecode,351,6,1,5,0,0,0,0,0,5,
+watchfiles,94,6,1,5,0,0,0,0,0,30,
+distlib,131,6,1,5,0,0,0,0,0,36,
+arrow,378,6,1,5,0,0,0,0,0,65,
+pexpect,234,6,1,5,0,0,0,0,0,18,
+joblib,168,6,2,4,1,0,0,1,0,106,
+stack-data,289,6,1,5,0,0,0,0,0,21,
+nvidia-cublas,684,6,2,4,0,0,0,0,2,26,
+matplotlib-inline,278,6,2,4,0,0,0,0,1,10,
+python-discovery,162,6,2,4,0,0,0,0,0,28,
+blinker,227,5,1,4,0,0,0,0,0,17,
+threadpoolctl,159,5,1,4,0,0,0,0,0,13,
+mpmath,230,5,1,4,1,0,0,1,0,41,
+cuda-pathfinder,566,5,1,4,0,0,0,0,0,34,
+virtualenv,116,5,3,2,24,23,22,12,3,311,
+ipython,263,5,3,2,8,1,0,4,7,201,
+wsproto,359,5,2,3,0,0,0,0,0,14,
+graphql-core,368,5,3,2,2,2,2,2,0,112,
+sse-starlette,123,5,1,4,0,0,0,0,0,83,
+nvidia-nccl-cu13,617,5,2,3,0,0,0,0,0,13,
+dill,180,5,3,2,0,0,0,0,0,30,
+opentelemetry-exporter-http-transport,1477,5,1,4,0,0,0,0,0,,
+jupyter-server,555,5,5,0,17,13,9,9,4,161,
+ruamel-yaml,181,5,3,2,0,0,0,0,2,273,
+databricks-sdk,302,5,3,2,0,0,0,0,0,183,
+gitdb,196,5,1,4,0,0,0,0,1,22,
+python-slugify,357,5,1,4,0,0,0,0,0,51,
+nvidia-cuda-cccl,2472,5,1,4,0,0,0,0,0,,
+tzlocal,170,5,5,0,0,0,0,0,2,54,
+jeepney,202,5,2,3,0,0,0,0,0,14,
+mako,152,5,1,4,5,3,3,3,4,83,
+jiter,92,5,2,3,0,0,0,0,0,23,
+isoduration,518,5,0,5,0,0,0,0,0,1,
+opentelemetry-exporter-otlp-proto-http,157,5,4,1,0,0,0,0,0,59,
+nvidia-cusparse,705,5,1,4,0,0,0,0,2,15,
+google-auth-oauthlib,249,5,5,0,0,0,0,0,1,32,
+lark,354,5,1,4,0,0,0,0,0,20,
+mcp-types,311,5,2,3,0,0,0,0,0,12,
diff --git a/analysis/fetch_deps.py b/analysis/fetch_deps.py
new file mode 100644
index 0000000..825db41
--- /dev/null
+++ b/analysis/fetch_deps.py
@@ -0,0 +1,48 @@
+"""Fetch resolved dependency graphs from deps.dev for the top-N PyPI packages."""
+import json, os, sys, urllib.parse, urllib.request
+from concurrent.futures import ThreadPoolExecutor
+
+data, n = sys.argv[1], int(sys.argv[2])
+rows = json.load(open(f"{data}/top/top30d.json"))["rows"][:n]
+API = "https://api.deps.dev/v3/systems/pypi/packages/"
+
+def get(url):
+ with urllib.request.urlopen(url, timeout=60) as r:
+ return json.load(r)
+
+def cached(path):
+ """A cache hit only counts if it holds data; failed fetches are retried."""
+ try:
+ return "error" not in json.load(open(path))
+ except (OSError, ValueError):
+ return False
+
+
+def fetch(name):
+ out = f"{data}/deps/{name}.json"
+ if cached(out):
+ return "cached"
+ try:
+ pkg = get(API + urllib.parse.quote(name, safe=""))
+ default = [v for v in pkg["versions"] if v.get("isDefault")]
+ recent = sorted((v for v in pkg["versions"] if not v.get("isDefault")), key=lambda v: v.get("publishedAt", ""), reverse=True)
+ last_error = None
+ # deps.dev may not have resolved the newest release yet; fall back to recent ones.
+ for v in (default + recent)[:8]:
+ version = v["versionKey"]["version"]
+ try:
+ graph = get(API + urllib.parse.quote(name, safe="") + "/versions/" + urllib.parse.quote(version, safe="") + ":dependencies")
+ except Exception as e:
+ last_error = e
+ continue
+ json.dump({"version": version, "fallback": not v.get("isDefault"), "graph": graph}, open(out, "w"))
+ return "ok"
+ raise last_error
+ except Exception as e:
+ json.dump({"error": repr(e)}, open(out, "w"))
+ return "error"
+
+with ThreadPoolExecutor(4) as pool:
+ results = list(pool.map(fetch, [r["project"] for r in rows]))
+from collections import Counter
+print(Counter(results))
diff --git a/analysis/fetch_scorecard.py b/analysis/fetch_scorecard.py
new file mode 100644
index 0000000..c892e6e
--- /dev/null
+++ b/analysis/fetch_scorecard.py
@@ -0,0 +1,42 @@
+"""Fetch OpenSSF Scorecard data via deps.dev for hidden-layer packages."""
+import json, sys, urllib.parse, urllib.request
+from concurrent.futures import ThreadPoolExecutor
+d = sys.argv[1]
+rows = json.load(open(f"{d}/joined.json"))["rows"]
+targets = [r for r in rows if r["reach"] >= 20 and r["only_indirect"] / r["reach"] >= 0.5]
+q = lambda s: urllib.parse.quote(s, safe="")
+def get(u):
+ with urllib.request.urlopen(u, timeout=60) as r: return json.load(r)
+def cached(path):
+ """A cache hit only counts if it holds data; failed fetches are retried."""
+ try:
+ return "error" not in json.load(open(path))
+ except (OSError, ValueError):
+ return False
+
+def version_for(p):
+ """The version whose graph was analyzed, or the default if that fetch failed or never ran."""
+ try:
+ return json.load(open(f"{d}/deps/{p}.json"))["version"]
+ except (OSError, ValueError, KeyError):
+ pkg = get(f"https://api.deps.dev/v3/systems/pypi/packages/{q(p)}")
+ return next(v["versionKey"]["version"] for v in pkg["versions"] if v.get("isDefault"))
+
+def fetch(r):
+ p = r["pkg"]; out = f"{d}/scorecard/{p}.json"
+ if cached(out): return
+ try:
+ version = version_for(p)
+ ver = get(f"https://api.deps.dev/v3/systems/pypi/packages/{q(p)}/versions/{q(version)}")
+ proj = next((x["projectKey"]["id"] for x in ver.get("relatedProjects", []) if x.get("relationType") == "SOURCE_REPO"), None)
+ res = {"project": proj}
+ if proj:
+ pj = get(f"https://api.deps.dev/v3/projects/{q(proj)}")
+ sc = pj.get("scorecard") or {}
+ res.update(stars=pj.get("starsCount"), score=sc.get("overallScore"), date=sc.get("date"),
+ checks={c["name"]: c["score"] for c in sc.get("checks", [])})
+ json.dump(res, open(out, "w"))
+ except Exception as e:
+ json.dump({"error": repr(e)}, open(out, "w"))
+with ThreadPoolExecutor(4) as pool: list(pool.map(fetch, targets))
+print(len(targets), "targets")
diff --git a/analysis/fetch_yanked.py b/analysis/fetch_yanked.py
new file mode 100644
index 0000000..0b6d8c5
--- /dev/null
+++ b/analysis/fetch_yanked.py
@@ -0,0 +1,50 @@
+"""Count yanked releases per package using PyPI's PEP 691 JSON simple index."""
+import json, os, sys, urllib.request
+from concurrent.futures import ThreadPoolExecutor
+from packaging.utils import parse_sdist_filename, parse_wheel_filename, InvalidSdistFilename, InvalidWheelFilename
+
+data, n = sys.argv[1], int(sys.argv[2])
+names = [r["project"] for r in json.load(open(f"{data}/top/top30d.json"))["rows"][:n]]
+
+def version_of(fn):
+ try:
+ if fn.endswith(".whl"):
+ return str(parse_wheel_filename(fn)[1])
+ return str(parse_sdist_filename(fn)[1])
+ except (InvalidSdistFilename, InvalidWheelFilename, Exception):
+ return None
+
+def cached(path):
+ """A cache hit only counts if it holds data; failed fetches are retried."""
+ try:
+ return "error" not in json.load(open(path))
+ except (OSError, ValueError):
+ return False
+
+
+def fetch(name):
+ out = f"{data}/simple/{name}.json"
+ if cached(out):
+ return "cached"
+ req = urllib.request.Request(f"https://pypi.org/simple/{name}/", headers={"Accept": "application/vnd.pypi.simple.v1+json"})
+ try:
+ with urllib.request.urlopen(req, timeout=60) as r:
+ idx = json.load(r)
+ except Exception as e:
+ json.dump({"error": repr(e)}, open(out, "w")); return "error"
+ versions, yanked, reasons = set(idx.get("versions", [])), {}, {}
+ for f in idx["files"]:
+ v = version_of(f["filename"])
+ # PEP 691: `yanked` is false/absent, true, or a reason string. Any string
+ # (even an empty one) means yanked, so test against false, not truthiness.
+ flag = f.get("yanked", False)
+ if v and flag is not False and flag is not None:
+ yanked[v] = True
+ if isinstance(flag, str) and flag:
+ reasons[v] = flag
+ json.dump({"n_versions": len(versions), "yanked": sorted(yanked), "reasons": reasons}, open(out, "w"))
+ return "ok"
+
+with ThreadPoolExecutor(4) as pool:
+ from collections import Counter
+ print(Counter(pool.map(fetch, names)))
diff --git a/analysis/osv_agg.py b/analysis/osv_agg.py
new file mode 100644
index 0000000..cefca6f
--- /dev/null
+++ b/analysis/osv_agg.py
@@ -0,0 +1,57 @@
+"""Aggregate OSV PyPI advisories into unique vulnerabilities per package."""
+import glob, json, re, sys, collections
+
+data = sys.argv[1]
+norm = lambda n: re.sub(r"[-_.]+", "-", n).lower()
+
+recs, mal = {}, collections.Counter()
+for f in glob.glob(f"{data}/osv/json/*.json"):
+ r = json.load(open(f))
+ if r.get("withdrawn"):
+ continue
+ if r["id"].startswith("MAL-"):
+ for a in r.get("affected", []):
+ mal[norm(a["package"]["name"])] += 1
+ continue
+ recs[r["id"]] = r
+
+# Union-find over ids and aliases so PYSEC/GHSA copies of one CVE merge.
+parent = {}
+def find(x):
+ parent.setdefault(x, x)
+ while parent[x] != x:
+ parent[x] = parent[parent[x]]; x = parent[x]
+ return x
+def union(a, b): parent[find(a)] = find(b)
+for rid, r in recs.items():
+ for a in r.get("aliases", []):
+ union(rid, a)
+
+clusters = collections.defaultdict(list)
+for rid in recs:
+ clusters[find(rid)].append(recs[rid])
+
+SEV = {"LOW": 1, "MODERATE": 2, "MEDIUM": 2, "HIGH": 3, "CRITICAL": 4}
+vulns = []
+for group in clusters.values():
+ pkgs, cwes, sev, published, ids, summary = set(), set(), 0, None, [], ""
+ for r in group:
+ ids.append(r["id"])
+ for a in r.get("affected", []):
+ if a["package"].get("ecosystem") == "PyPI":
+ pkgs.add(norm(a["package"]["name"]))
+ ds = r.get("database_specific") or {}
+ cwes.update(ds.get("cwe_ids") or [])
+ sev = max(sev, SEV.get(str(ds.get("severity", "")).upper(), 0))
+ p = r.get("published", "")
+ published = min(published, p) if published else p
+ summary = summary or r.get("summary", "")
+ cve = sorted({a for r in group for a in r.get("aliases", []) if a.startswith("CVE-")})
+ vulns.append(dict(ids=sorted(ids), cves=cve, pkgs=sorted(pkgs), cwes=sorted(cwes),
+ sev=sev, published=published, summary=summary))
+
+json.dump({"vulns": vulns, "malicious": mal}, open(f"{data}/vulns.json", "w"))
+print("raw non-MAL records:", len(recs), "unique vulns:", len(vulns), "malicious pkgs:", len(mal), "MAL entries:", sum(mal.values()))
+per = collections.Counter(p for v in vulns for p in v["pkgs"])
+print("packages with >=1 vuln:", len(per))
+print("top 25 all-time:", per.most_common(25))
diff --git a/analysis/report.py b/analysis/report.py
new file mode 100644
index 0000000..d1b1bb6
--- /dev/null
+++ b/analysis/report.py
@@ -0,0 +1,95 @@
+import collections, datetime as dt, glob, json, os, re, sys
+d = sys.argv[1]
+norm = lambda n: re.sub(r"[-_.]+", "-", n).lower()
+J = json.load(open(f"{d}/joined.json")); rows = {r["pkg"]: r for r in J["rows"]}
+vulns = json.load(open(f"{d}/vulns.json"))
+V, MAL = vulns["vulns"], vulns["malicious"]
+top = json.load(open(f"{d}/top/top30d.json"))["rows"]
+top_names = [norm(r["project"]) for r in top[:1000]]
+top_set = set(top_names)
+
+hidden = [r for r in rows.values() if r["reach"] >= 10 and r["only_indirect"] / r["reach"] >= 0.5]
+print("== HIDDEN LAYER: reach>=10 and >=50% of reach is indirect-only:", len(hidden))
+for r in sorted(hidden, key=lambda r: -r["reach"])[:40]:
+ print(f" {r['pkg']:24} reach={r['reach']:>3} indirect%={100*r['only_indirect']/r['reach']:>3.0f} rank={r['rank']} vulns={r['vulns']} since23={r['vulns_2023']} hi/crit={r['high_crit']} yanked={r['yanked']}")
+
+# Burstiness: share of a package's vulns published in its busiest 7-day window.
+print("\n== BURSTINESS (packages with >=8 vulns since 2023, reach>=10 or rank<=1000)")
+by_pkg = collections.defaultdict(list)
+for v in V:
+ for p in v["pkgs"]:
+ if v["published"] >= "2023": by_pkg[p].append(dt.date.fromisoformat(v["published"][:10]))
+for p, dates in sorted(by_pkg.items(), key=lambda kv: -len(kv[1])):
+ r = rows.get(p)
+ if len(dates) < 8 or not r or r["reach"] < 10: continue
+ dates.sort(); best = max(sum(1 for x in dates if 0 <= (x - s).days < 7) for s in dates)
+ print(f" {p:16} since23={len(dates):>3} busiest_week={best:>3} ({100*best/len(dates):.0f}%)")
+
+# Year trend for the hidden layer vs everything.
+hid = {r["pkg"] for r in hidden}
+yr_all, yr_hid = collections.Counter(), collections.Counter()
+for v in V:
+ y = v["published"][:4]; yr_all[y] += 1
+ if hid & set(v["pkgs"]): yr_hid[y] += 1
+print("\n== PER YEAR (all unique vulns / hidden-layer vulns)")
+print(" ", {y: (yr_all[y], yr_hid[y]) for y in sorted(yr_all) if y >= "2018"})
+
+# CWE mix in the hidden layer + other popular dependencies (reach>=10).
+deps = {r["pkg"] for r in rows.values() if r["reach"] >= 10}
+cw = collections.Counter(); n_with = 0
+for v in V:
+ if deps & set(v["pkgs"]) and v["published"] >= "2023":
+ if v["cwes"]: n_with += 1
+ cw.update(set(v["cwes"]))
+print(f"\n== CWE mix, dependencies with reach>=10, since 2023 ({n_with} vulns with CWE data)")
+for c, n in cw.most_common(15): print(f" {c:10} {n}")
+
+# Yanked releases among the top 1000 (bug proxy).
+print("\n== YANKED RELEASES, top 1000 packages")
+ys = []
+for f in glob.glob(f"{d}/simple/*.json"):
+ y = json.load(open(f)); p = norm(os.path.basename(f)[:-5])
+ if p in top_set and y.get("n_versions"): ys.append((p, len(y["yanked"]), y["n_versions"], y["reasons"]))
+print(" packages with >=1 yanked release:", sum(1 for x in ys if x[1]), "of", len(ys), "| total yanked:", sum(x[1] for x in ys))
+for p, n, tot, reasons in sorted(ys, key=lambda x: -x[1])[:15]:
+ r = rows.get(p, {}); sample = next(iter(reasons.values()), "")[:70]
+ print(f" {p:26} yanked={n:>3}/{tot:<5} reach={r.get('reach', 0):>3} e.g. {sample!r}")
+
+# Malicious packages: year trend and typosquat targets among the top 1000.
+def dl1(a, b):
+ if abs(len(a) - len(b)) > 1 or a == b: return False
+ if len(a) == len(b):
+ diff = [i for i in range(len(a)) if a[i] != b[i]]
+ return len(diff) == 1 or (len(diff) == 2 and diff[1] == diff[0] + 1 and a[diff[0]] == b[diff[1]] and a[diff[1]] == b[diff[0]])
+ if len(a) > len(b): a, b = b, a
+ return any(b[:i] + b[i+1:] == a for i in range(len(b)))
+mal_years = collections.Counter()
+for f in glob.glob(f"{d}/osv/json/MAL-*.json"):
+ mal_years[json.load(open(f)).get("published", "")[:4]] += 1
+targets = collections.Counter()
+by_len = collections.defaultdict(list)
+for t in top_names: by_len[len(t)].append(t)
+squats = 0
+for m in MAL:
+ hits = [t for L in (len(m)-1, len(m), len(m)+1) for t in by_len.get(L, []) if dl1(m, t)]
+ if hits: squats += 1; targets.update(hits)
+print("\n== MALICIOUS: per year", dict(sorted(mal_years.items())))
+print(f" {len(MAL)} malicious package names; {squats} are one edit from a top-1000 name")
+print(" most-targeted:", targets.most_common(15))
+
+# Monthly trend, DoS share by population, and Scorecard by advisory status.
+import statistics
+monthly = collections.Counter(v["published"][:7] for v in V if v["published"] >= "2025-01")
+print("\n== MONTHLY unique vulns since 2025:", dict(sorted(monthly.items())))
+print(" packages with advisories: 2025 =", len({p for v in V if v["published"][:4] == "2025" for p in v["pkgs"]}),
+ "| 2026 =", len({p for v in V if v["published"][:4] == "2026" for p in v["pkgs"]}))
+DOS = {"CWE-400", "CWE-770", "CWE-1333", "CWE-407", "CWE-409", "CWE-835", "CWE-674", "CWE-1050", "CWE-834", "CWE-789", "CWE-776"}
+for label, pop in [("dependencies (reach>=10)", deps), ("hidden layer", hid), ("all PyPI", None)]:
+ vs = [v for v in V if v["published"] >= "2023" and v["cwes"] and (pop is None or pop & set(v["pkgs"]))]
+ dos = sum(1 for v in vs if DOS & set(v["cwes"]))
+ print(f" {label}: {len(vs)} vulns with CWE since 2023, DoS-class {dos} ({100 * dos / len(vs):.0f}%)")
+sc = {os.path.basename(f)[:-5]: json.load(open(f)) for f in glob.glob(f"{d}/scorecard/*.json")}
+sc = {p: s for p, s in sc.items() if p in rows and s.get("score") is not None}
+with_v = [s["score"] for p, s in sc.items() if rows[p]["vulns"]]
+without = [s["score"] for p, s in sc.items() if not rows[p]["vulns"]]
+print(f"\n== SCORECARD mean: with advisories {statistics.mean(with_v):.1f} (n={len(with_v)}), without {statistics.mean(without):.1f} (n={len(without)})")
diff --git a/docs/data-systems.md b/docs/data-systems.md
index 2742937..3b0360c 100644
--- a/docs/data-systems.md
+++ b/docs/data-systems.md
@@ -41,6 +41,8 @@ It also makes `set` iteration order of strings differ between runs, which makes
**The fix:** Use a stable hash for anything that crosses a process boundary or gets persisted: `zlib.crc32(key.encode())`, or `hashlib.blake2b(key.encode(), digest_size=8)` for a better distribution. `PYTHONHASHSEED=0` disables randomization, but it's a workaround, not a design.
+**In the wild:** Randomization exists because of CVE-2012-1150. Attackers could send request parameters chosen to collide in a dict, turning each insert into a linear scan and letting a single HTTP request occupy a server's CPU. Python added randomization in 2.6.8, 2.7.3 and 3.2.3 behind `-R`, and turned it on by default in 3.3.
+
[Example](../examples/43_hash_randomization_sharding.py)
### 44. NaN Breaks `in`, `max()` and `sorted()`
diff --git a/docs/dependency-risk.md b/docs/dependency-risk.md
new file mode 100644
index 0000000..93358da
--- /dev/null
+++ b/docs/dependency-risk.md
@@ -0,0 +1,217 @@
+# Complexity in the Wild: What Python's Vulnerability Data Shows
+
+[← Back to all pitfalls](../README.md)
+
+The rest of this repo argues that Python's readable surface hides behavior you can't predict from the interface: *unknown unknowns*. This page tests that argument against data. It joins every Python vulnerability in the [OSV database](https://osv.dev) with the resolved dependency graphs of the 1,000 most-downloaded PyPI packages, PyPI's record of yanked releases, and OpenSSF Scorecard maintenance scores.
+
+The result is organized by **mechanism**: the kind of hidden complexity that produced each vulnerability. Each section links to the pitfalls that show that mechanism in a few lines of code.
+
+The scripts, a CSV snapshot and the chart source are in [`analysis/`](../analysis/), so every number and chart can be regenerated. *Snapshot: 30 October 2026. PyPI download rankings from 1 October 2026 (30-day window).*
+
+## Key findings
+
+1. **Most of your exposure is code you never chose.** `urllib3`, `certifi`, `idna`, `cryptography`, `anyio` and `pyasn1` are each pulled in by 43–177 of the top 1,000 packages, 70–95% of the time *indirectly*. Weighted by reach, `urllib3` and `cryptography` are the highest-exposure dependencies on PyPI.
+2. **Libraries fail at the parsing boundary.** 34% of vulnerabilities in widely used dependencies are resource exhaustion (ReDoS, decompression bombs, unbounded allocation, deep recursion), against 13% across PyPI. Applications fail through path traversal, injection and SSRF instead.
+3. **Conveniences hide decisions.** Following redirects, trusting a certificate bundle, parsing HTTP framing: each looks like plumbing, and each has produced a run of CVEs in the libraries that do it for you.
+4. **The layer below Python is a bug source of its own.** Pillow's C decoders lead PyPI libraries with 119 advisories; `cryptography` inherits OpenSSL's; compiled extensions account for the segfaults and leaks behind many yanked releases.
+5. **Known risk is growing fast, and it's measured by who looks.** 2026 has 2,637 unique advisories so far, more than three times any earlier year, arriving in audit-sized bursts. Meanwhile some of the most-depended-on packages have no advisories and weak maintenance signals.
+
+
+
+
+
+
+---
+
+## 1. Code you never chose
+
+A package's **reach** is the number of top-1,000 packages whose resolved dependency tree includes it. A package is in the **hidden layer** when more than half of that reach is indirect: it's a dependency of a dependency.
+
+
+
+
+
+
+The chart shows the shape of dependency risk. The packages with the most advisories (Pillow, aiohttp, PyJWT, Tornado) are ones people install deliberately. The packages with the most *reach* are infrastructure almost nobody installs on purpose. Weighting advisories since 2023 by reach combines the two:
+
+| Package | Reach | Indirect | Vulns since 2023 | Of which 2026 | Exposure | Top weaknesses |
+|---|---:|---:|---:|---:|---:|---|
+| `urllib3` | 145 | 86% | 14 | 6 | 2,030 | Information exposure, certificate validation |
+| `cryptography` | 82 | 70% | 19 | 7 | 1,558 | NULL dereference, vulnerable bundled component |
+| `pillow` | 17 | 18% | 66 | 60 | 1,122 | Out-of-bounds read, resource exhaustion |
+| `aiohttp` | 15 | 27% | 54 | 40 | 810 | Unbounded allocation, request smuggling |
+| `requests` | 121 | 49% | 4 | 1 | 484 | Information exposure, credential leaks |
+| `pyjwt` | 19 | 47% | 23 | 21 | 437 | Improper signature verification |
+| `certifi` | 144 | 87% | 3 | 1 | 432 | Insufficient authenticity verification |
+| `filelock` | 18 | 56% | 20 | 19 | 360 | TOCTOU races, symlink following |
+| `idna` | 177 | 95% | 2 | 1 | 354 | ReDoS, resource exhaustion |
+| `tornado` | 13 | 54% | 24 | 15 | 312 | Resource exhaustion, unbounded allocation |
+| `anyio` | 72 | 75% | 3 | 3 | 216 | Unbounded allocation, certificate validation |
+| `pyasn1` | 43 | 88% | 5 | 5 | 215 | Resource exhaustion, algorithmic complexity |
+
+Every hidden-layer package with reach of at least 20 and at least one advisory:
+
+| Package | Download rank | Reach | Indirect | Vulns (all time) | Vulns in 2026 | High/critical | Scorecard |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `cryptography` | 9 | 82 | 70% | 24 | 7 | 12 | 8.3 |
+| `urllib3` | 5 | 145 | 86% | 22 | 6 | 12 | 8.6 |
+| `pygments` | 12 | 56 | 82% | 5 | 1 | 3 | 6.2 |
+| `pyasn1` | 65 | 43 | 88% | 5 | 5 | 5 | 6.6 |
+| `httpx2` | 129 | 36 | 75% | 5 | 5 | 2 | – |
+| `certifi` | 6 | 144 | 87% | 4 | 1 | 2 | 6.3 |
+| `anyio` | 22 | 72 | 75% | 3 | 3 | 2 | 5.6 |
+| `idna` | 4 | 177 | 95% | 2 | 1 | 0 | 8.5 |
+| `websockets` | 68 | 24 | 75% | 2 | 0 | 2 | 5.5 |
+| `markdown-it-py` | 54 | 27 | 89% | 2 | 0 | 2 | 5.3 |
+| `httpcore2` | 130 | 37 | 97% | 1 | 1 | 1 | – |
+| `multidict` | 47 | 23 | 83% | 1 | 1 | 0 | 7.3 |
+| `h11` | 27 | 69 | 94% | 1 | 0 | 1 | 4.4 |
+
+For comparison, the packages with the most advisories overall are applications and frameworks you install deliberately: TensorFlow (435), Django (162), Open WebUI (161), Apache Airflow (158), vLLM (111) and MLflow (87).
+
+## 2. Parsers on untrusted input
+
+*Pitfalls: [#56 regex backtracking](security.md#56-nested-quantifiers-backtrack-exponentially), [#57 decompression bombs](security.md#57-decompression-bombs), [#42 the 4,300-digit limit](interpreter.md#42-strint-fails-above-4300-digits), [#53 path joins](security.md#53-ospathjoin-discards-your-prefix).*
+
+
+
+
+
+
+Libraries sit where untrusted bytes become Python objects: URLs, domain names, HTTP framing, ASN.1, Markdown, images, compressed streams. Their characteristic bug is letting an attacker make that conversion arbitrarily expensive, so resource exhaustion makes up a third of their advisories. The interface is "parse this", and nothing at the call site says the cost depends on the input's shape rather than its size.
+
+- **`idna`** (95% indirect) had catastrophic regex backtracking on crafted domain names (CVE-2024-3651), and **`urllib3`** on URLs with many `@` characters (CVE-2021-33503). Both sit under nearly every HTTP request made from Python.
+- **`urllib3`** has since fixed decompression bombs in its streaming API three times between December 2025 and May 2026.
+- **`pyasn1`**, which arrives almost entirely through `pyasn1-modules` (39 of its 43 dependents), had five advisories in 2026, all high or critical and all denial of service in its BER/DER decoder: unbounded recursion, quadratic OID parsing and oversized tags.
+
+Applications, by contrast, fail where untrusted input becomes an *action*: a file path (path traversal is the #1 weakness on PyPI, 507 advisories since 2023), a query, a template, a URL to fetch.
+
+**What this means for your code:** treat input *size and shape* as part of validation. Cap body sizes, header counts, nesting depth and decompression ratios at your outermost layer, because the library several levels down is where resource exhaustion lands.
+
+## 3. Conveniences that hide protocol decisions
+
+*Pitfall: [#55 redirects forward credentials](security.md#55-redirects-forward-credentials-to-another-host).*
+
+High-level HTTP APIs make decisions on your behalf, and each decision is a place where the safe answer depends on details the interface hides.
+
+- **Following redirects.** `urllib3` advisories cover `Cookie`, `Authorization` and `Proxy-Authorization` headers forwarded across origins (CVE-2018-20060, CVE-2023-43804, CVE-2024-37891, CVE-2026-44431), and `requests` has had the same class (CVE-2018-18074, CVE-2023-32681). The standard library's `urllib.request` still forwards these headers today.
+- **Trusting certificates.** `certifi`'s advisories are about *trust*, not code: the bundled root list included authorities that were later distrusted (TrustCor, e-Tugra, GLOBALTRUST), and in 2026 roots whose intermediate keys were compromised (emSign). `certifi` is 87% indirect, so most of its users never chose which authorities they trust.
+- **Parsing HTTP framing.** Request smuggling (CWE-444) is 6% of dependency advisories against under 1% across PyPI: 18 advisories, 12 of them in `aiohttp`. `h11`, the HTTP/1.1 parser under `httpx`, `httpcore` and `uvicorn`, had one in 2025 for accepting malformed chunked bodies. Smuggling happens when two parsers disagree about where one request ends, which is invisible to anyone calling `client.get()`.
+
+## 4. One parameter, two meanings
+
+*Pitfall: [#59 signing-key confusion](security.md#59-one-key-parameter-two-meanings).*
+
+**PyJWT** has had 23 advisories since 2023. 15 of them involve keys or algorithms: public keys in DER form, JWK containers or reformatted PEM accepted as HMAC secrets, and empty keys accepted. Each is the same shallow interface, a `key` argument whose meaning depends on an algorithm the caller may not control.
+
+**`filelock`** shows the same pattern on the filesystem. Its interface is "a lock". Its 20 advisories since 2023 (18 in one week of October 2026) are about what that hides when the lock file lives in a shared directory: symlinks swapped in between check and use, hard links, and lock files created by another user.
+
+## 5. The layer below Python
+
+*Pitfalls: [#25 `+=` is not atomic](concurrency.md#25--is-not-atomic) and [#29 `fork()` copies locked locks](concurrency.md#29-fork-copies-locked-locks) show how C-level details leak into Python semantics.*
+
+- **Pillow** is the most vulnerable *library* on PyPI: 119 advisories, 88 of them high or critical, led by out-of-bounds reads and writes in its C image decoders. `Image.open()` looks like reading a file; underneath, it's C parsing untrusted binary formats.
+- **`cryptography`** has a recurring weakness, CWE-1395, *dependency on a vulnerable third-party component*: its wheels bundle OpenSSL, so OpenSSL advisories become `cryptography` advisories. Upgrading `cryptography` is how Python users patch OpenSSL, often without knowing they have it.
+- **Yanked releases** of compiled extensions show the same layer failing in ways pure Python can't: `grpcio` ("Segfaults", "Deadlock observed in Apache Beam"), `multidict` (memory leaks) and `yarl` ("Cython generated files are broken"). See [Appendix A](#yanked-releases).
+
+## What to do
+
+1. **Know your hidden layer.** Run `uv tree` or `pipdeptree` and look at what you didn't choose. The packages in section 1 are probably in your environment.
+2. **Scan continuously, including transitive dependencies.** `pip-audit` or `osv-scanner` in CI checks the whole resolved environment against the same OSV data used here. Given the 2026 volume, a weekly scan is already stale.
+3. **Lock with hashes.** `uv lock`, or `pip-compile --generate-hashes` with `pip install --require-hashes`, pins every transitive dependency to an exact, verified artifact. That protects against typosquats ([Appendix B](#appendix-b-malicious-packages-and-typosquatting)) and keeps a bad release from arriving silently.
+4. **Bound untrusted input at the edge.** Resource exhaustion is the leading dependency failure, so cap sizes before data reaches the parsers.
+5. **Isolate the riskiest parsers.** Process untrusted images, PDFs and archives in a separate worker with memory and CPU limits.
+6. **Don't read "no advisories" as "safe".** For quiet, critical dependencies, check maintenance signals as well as advisories ([Appendix A](#the-quiet-ones)).
+
+---
+
+## Appendix A: Measuring risk
+
+### The 2026 surge and bursty disclosure
+
+Monthly volume climbed from about 60 in mid-2025 to between 210 and 390 every month since April 2026, and the number of distinct packages receiving advisories more than doubled (310 in 2025, 725 in 2026). This dataset can't say why. Faster, more automated vulnerability discovery would produce this pattern, but so would changes in disclosure practice; treat any explanation as a hypothesis.
+
+What the data does show is that packages receive advisories in batches, consistent with a single audit or researcher working through a codebase:
+
+| Package | Advisories since 2023 | Busiest 7 days | Share |
+|---|---:|---:|---:|
+| `filelock` | 20 | 18 | 90% |
+| `pillow` | 66 | 41 | 62% |
+| `pyjwt` | 23 | 14 | 61% |
+| `sqlparse` | 8 | 4 | 50% |
+| `starlette` | 10 | 4 | 40% |
+| `aiohttp` | 54 | 10 | 19% |
+| `cryptography` | 19 | 3 | 16% |
+
+So a package's advisory count says as much about *who has looked* as about how buggy it is.
+
+### The quiet ones
+
+Of the 42 hidden-layer packages with reach of at least 20, most have **no advisories at all**. OpenSSF Scorecard (0–10, covering maintenance activity, code review, fuzzing, branch protection and similar practices) suggests that some are quiet because nobody is looking rather than because they're solid.
+
+| Package | Reach | Indirect | Scorecard | Maintained | Code review | Stars |
+|---|---:|---:|---:|---:|---:|---:|
+| `mdurl` | 28 | 96% | 3.4 | 0 | 0 | 15 |
+| `six` | 83 | 89% | 3.7 | 0 | 2 | 1,033 |
+| `wrapt` | 36 | 53% | 3.8 | 10 | 0 | 2,313 |
+| `sniffio` | 28 | 68% | 4.0 | 0 | 8 | 155 |
+| `annotated-types` | 61 | 98% | 4.2 | 2 | 8 | 616 |
+| `h11` | 69 | 94% | 4.4 | 0 | 5 | 571 |
+| `python-dateutil` | 71 | 62% | 4.4 | 1 | 1 | 2,641 |
+| `httpcore` | 33 | 100% | 4.7 | 0 | 7 | 551 |
+
+Across the 34 hidden-layer packages that have a Scorecard, those **with** advisories average 6.6 and those **without** average 5.8: the projects that publish advisories tend to be the ones with security processes. Two caveats: a "Maintained" score of 0 only means little recent activity, which is legitimate for finished software such as `six`; and the sample is small.
+
+### Yanked releases
+
+PyPI lets maintainers *yank* a release: it stays downloadable for exact pins but is skipped by resolvers. Maintainers yank a release when it's broken badly enough to warn everyone off, which makes yanks a measurable trace of bugs that reached users.
+
+- **331 of the top 1,000 packages (33%) have yanked at least one release**, 940 releases in total.
+- The heaviest yankers are fast-moving tools and frameworks, led by `polars` (67 yanked), `claude-agent-sdk` (25, mostly scheduled removals of old builds), `omegaconf` (19), `setuptools-scm` (18), `setuptools` (12) and `grpcio` (11).
+
+Among widely used dependencies (reach ≥ 10):
+
+| Package | Yanked | Reach | Example reasons |
+|---|---:|---:|---|
+| `setuptools` | 12 of 627 | 16 | Build regressions (pypa/setuptools#2500, #2885, #3002) |
+| `grpcio` | 11 of 284 | 29 | "Segfaults", "Deadlock observed in Apache Beam" |
+| `multidict` | 8 of 156 | 23 | Memory leaks (aio-libs/multidict#1117, #1131) |
+| `yarl` | 7 of 137 | 21 | "Cython generated files are broken", performance regression |
+| `aiohttp` | 7 of 314 | 15 | Regressions, an over-tight `multidict` pin that broke resolution |
+| `protobuf` | 5 of 223 | 63 | Wrong Python-version metadata, superseded release candidates |
+| `tzdata` | 4 of 39 | 44 | (no reason given) |
+| `urllib3` | 4 of 105 | 145 | "Truncated response bodies when streaming a large compressed body" |
+
+Besides compiled extensions (section 5), **packaging metadata** is a bug surface of its own: several yanks fixed version pins or Python-version claims rather than code.
+
+## Appendix B: Malicious packages and typosquatting
+
+OSV's `MAL-` records track packages uploaded to PyPI with malicious intent. They aren't vulnerabilities in real libraries, so they're excluded from everything above, but they're the largest category in the database. Here the hidden complexity is the interface of `pip install` itself: a package name is the entire trust decision.
+
+| 2022 | 2023 | 2024 | 2025 | 2026 (to Oct 30) |
+|---:|---:|---:|---:|---:|
+| 20 | 6,468 | 2,646 | 1,245 | 1,588 |
+
+739 of the 11,942 malicious names are a single edit (one insertion, deletion, substitution or swap of adjacent letters) away from a top-1,000 package. The most impersonated: `requests` (49 lookalikes), `requests-toolbelt` (43), `colorama` (42), `websocket-client` (40), `pycodestyle` and `prompt-toolkit` (32 each), `progressbar2` (31) and `scikit-learn` (30).
+
+## Method and limitations
+
+**Sources**
+
+- Vulnerabilities: the full [OSV PyPI export](https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip) (26,753 records). Records sharing an identifier or alias were merged, so PYSEC and GHSA copies of one CVE count once, giving 7,032 unique vulnerabilities. Withdrawn records are excluded, and `MAL-` records are analyzed separately.
+- Popularity: [hugovk/top-pypi-packages](https://hugovk.github.io/top-pypi-packages/), 30-day downloads for the top 15,000 projects.
+- Dependency graphs: [deps.dev](https://deps.dev) resolved graphs for the default version of each of the top 1,000 packages. For 88 fast-releasing packages (such as `boto3` and `pydantic`), deps.dev hadn't yet resolved the newest release, so the most recent resolved release was used.
+- Yanked releases: PyPI's JSON simple index ([PEP 691](https://peps.python.org/pep-0691/)).
+- Maintenance: OpenSSF Scorecard via deps.dev.
+- Weakness classes in the chart group related CWEs, for example resource exhaustion = CWE-400, 407, 409, 674, 770, 835, 1333 and similar. The exact mapping is in [`analysis/charts.py`](../analysis/charts.py).
+
+**Limitations**
+
+- *Reach* counts only packages within the top 1,000 and only default dependencies on deps.dev's resolution platform. Optional extras (`pip install pkg[extra]`), platform-specific dependencies and vendored copies (`pip` bundles its own `urllib3` and `requests`) are not counted.
+- Severity comes from GitHub advisories (available for 6,534 of 7,032 vulnerabilities) and CWE data from 6,320. PYSEC-only records have neither.
+- Publication dates are when an advisory was published, which can trail the fix by months or years, and GitHub sometimes imports older CVEs with new dates.
+- Advisory counts reflect scrutiny and disclosure practice as well as code quality ([Appendix A](#the-2026-surge-and-bursty-disclosure)). Treat them as a measure of *known* risk.
+- Yanking is a maintainer's choice, so packages that never yank aren't necessarily bug-free.
+- "One edit away" typosquat matching only checks top-1,000 names and misses other tricks, such as added suffixes (`requests-secure`) or swapped separators.
+
+To reproduce, see [`analysis/README.md`](../analysis/README.md).
diff --git a/docs/img/advisories-per-year-dark.svg b/docs/img/advisories-per-year-dark.svg
new file mode 100644
index 0000000..17180ec
--- /dev/null
+++ b/docs/img/advisories-per-year-dark.svg
@@ -0,0 +1,289 @@
+
+
+
diff --git a/docs/img/advisories-per-year-light.svg b/docs/img/advisories-per-year-light.svg
new file mode 100644
index 0000000..a52c5ca
--- /dev/null
+++ b/docs/img/advisories-per-year-light.svg
@@ -0,0 +1,289 @@
+
+
+
diff --git a/docs/img/reach-vs-advisories-dark.svg b/docs/img/reach-vs-advisories-dark.svg
new file mode 100644
index 0000000..4db782d
--- /dev/null
+++ b/docs/img/reach-vs-advisories-dark.svg
@@ -0,0 +1,407 @@
+
+
+
diff --git a/docs/img/reach-vs-advisories-light.svg b/docs/img/reach-vs-advisories-light.svg
new file mode 100644
index 0000000..b8cf873
--- /dev/null
+++ b/docs/img/reach-vs-advisories-light.svg
@@ -0,0 +1,407 @@
+
+
+
diff --git a/docs/img/weakness-mix-dark.svg b/docs/img/weakness-mix-dark.svg
new file mode 100644
index 0000000..62905b8
--- /dev/null
+++ b/docs/img/weakness-mix-dark.svg
@@ -0,0 +1,389 @@
+
+
+
diff --git a/docs/img/weakness-mix-light.svg b/docs/img/weakness-mix-light.svg
new file mode 100644
index 0000000..7f37da5
--- /dev/null
+++ b/docs/img/weakness-mix-light.svg
@@ -0,0 +1,389 @@
+
+
+
diff --git a/docs/security.md b/docs/security.md
new file mode 100644
index 0000000..a29cc7e
--- /dev/null
+++ b/docs/security.md
@@ -0,0 +1,171 @@
+# Security Pitfalls: Where Hidden Complexity Becomes a CVE
+
+[← Back to all pitfalls](../README.md)
+
+Every pitfall in this repo is a gap between what code *looks* like it does and what it actually does. On this page, that gap is exploitable. Each entry is one of the weakness types that dominates real Python vulnerability data (see [Dependency risk](dependency-risk.md)), shown through the standard-library behavior underneath it, with the real CVEs it produced.
+
+Counts are unique PyPI advisories published since 2023, from the [OSV](https://osv.dev) database.
+
+## Contents
+
+| # | Pitfall | Weakness (CWE) | PyPI advisories since 2023 | The fix |
+|---|---------|----------------|---------------------------:|---------|
+| 53 | [`os.path.join` discards your prefix](#53-ospathjoin-discards-your-prefix) | Path traversal (22) | 444, the most common weakness on PyPI | Resolve, then check containment |
+| 54 | [`tarfile` trusted member paths](#54-tarfile-trusted-member-paths-until-python-314) | Path traversal (22) | (CVE-2007-4559, in CPython) | `filter="data"` |
+| 55 | [Redirects forward credentials](#55-redirects-forward-credentials-to-another-host) | Information exposure (200) | 60 redirect-related | Strip headers across origins |
+| 56 | [Regex backtracking](#56-nested-quantifiers-backtrack-exponentially) | ReDoS (1333) | 80 | Remove nested quantifiers |
+| 57 | [Decompression bombs](#57-decompression-bombs) | Compressed data (409) | 31 | Cap output size |
+| 58 | [Loading data runs code](#58-loading-data-can-run-code) | Deserialization (502) | 210 | Use data-only formats |
+| 59 | [One key parameter, two meanings](#59-one-key-parameter-two-meanings) | Signature verification (347) | 41 | Pin the algorithm |
+
+Entries 53–57 have runnable examples. Entries 58 and 59 are documented without examples, because demonstrating them means writing a working attack.
+
+---
+
+### 53. `os.path.join` Discards Your Prefix
+
+`os.path.join(base, name)` reads like "put `name` inside `base`". But if `name` is absolute, join throws `base` away entirely, and `pathlib`'s `/` operator does the same. Relative `..` segments pass through untouched, because join never normalizes.
+
+```python
+# The Trap
+upload_dir = "/srv/app/uploads"
+os.path.join(upload_dir, "report.pdf") # '/srv/app/uploads/report.pdf'
+os.path.join(upload_dir, "/etc/hosts") # '/etc/hosts'
+Path(upload_dir) / "/etc/hosts" # PosixPath('/etc/hosts')
+os.path.join(upload_dir, "../config.py") # '/srv/app/uploads/../config.py'
+```
+
+**Why:** `join` is documented string surgery on path syntax, not a security boundary. On Windows, a drive letter (`C:`) resets it too.
+
+**The fix:** Resolve the final path and check that it's still inside the base directory:
+
+```python
+def safe_join(base, name):
+ base = Path(base).resolve()
+ target = (base / name).resolve()
+ if not target.is_relative_to(base):
+ raise ValueError(f"{name!r} escapes {base}")
+ return target
+```
+
+`.resolve()` also follows symlinks, so a link planted inside the directory can't point outside it. Better still, don't use client-supplied names as paths at all: generate the filename yourself and store the original as metadata.
+
+**In the wild:** Path traversal (CWE-22 and its variants) is the most common weakness in PyPI advisories, with 507 since 2023. It shows up in file servers, archive extraction, model and plugin loaders, and anything that turns a request parameter into a path.
+
+[Example](../examples/53_path_join_absolute.py)
+
+### 54. `tarfile` Trusted Member Paths Until Python 3.14
+
+A tar archive stores its own file names, and before Python 3.14 `tarfile.extractall()` wrote them where they said, including `../../etc/cron.d/job` or `/home/user/.ssh/authorized_keys`. Extracting an archive meant letting its author choose where files go.
+
+```python
+# The Trap (Python < 3.14)
+with tarfile.open("upload.tar") as tar:
+ tar.extractall("dest") # A member named "../outside.txt" lands outside dest.
+```
+
+Confusingly, `zipfile.extractall()` has always sanitized names: the same `../outside.txt` member becomes `dest/outside.txt`. Two standard-library archive modules gave opposite answers to the same question.
+
+**Why:** Tar preserves full Unix filesystem semantics (absolute paths, links, device files, permissions) because it was designed for backups, where that's exactly what you want.
+
+**The fix:** Always pass `filter="data"`, which rejects absolute paths, `..` escapes, links pointing outside the destination, and device files. It was added by [PEP 706](https://peps.python.org/pep-0706/) in Python 3.12, backported to 3.8.17, 3.9.17, 3.10.12 and 3.11.4, and became the default in 3.14. Passing it explicitly makes your code safe on every supported version.
+
+**In the wild:** CVE-2007-4559 was reported in 2007 and stayed open for over 15 years, partly because changing the default would break legitimate backup tools. It was finally addressed by PEP 706 in 2023, after renewed attention to how widely `extractall()` was used on untrusted archives.
+
+[Example](../examples/54_tarfile_extraction_filter.py)
+
+### 55. Redirects Forward Credentials to Another Host
+
+HTTP libraries follow redirects for you, and that's exactly the problem: the request you configured for `api.example.com` gets replayed, headers and all, to wherever the `Location` header points. Python's own `urllib.request` forwards `Authorization` and `Cookie` headers to a different host, as verified on Python 3.13 and 3.14.
+
+```python
+# The Trap
+req = urllib.request.Request("https://api.example.com/data",
+ headers={"Authorization": "Bearer api-token"})
+urllib.request.urlopen(req)
+# A 302 to https://attacker.example/ resends "Authorization: Bearer api-token".
+```
+
+**Why:** "Follow redirects" hides a policy decision, *which* parts of the request should be resent, behind a convenience. The safe answer depends on whether the scheme, host and port changed, and older code didn't ask.
+
+**The fix:** Use a client that strips credentials on cross-origin redirects (current `requests`, `urllib3` and `httpx` do), disable redirects for authenticated calls, or install a redirect handler that removes `Authorization`, `Cookie` and `Proxy-Authorization` when the origin changes. The example has a 15-line handler.
+
+**In the wild:** The leading Python HTTP libraries have all fixed variations on this, one header at a time:
+
+- `requests`: CVE-2018-18074 (`Authorization` kept on an HTTPS-to-HTTP redirect), CVE-2023-32681 (`Proxy-Authorization` leak)
+- `urllib3`: CVE-2018-20060 (`Authorization`), CVE-2023-43804 (`Cookie`), CVE-2024-37891 (`Proxy-Authorization`), CVE-2026-44431 (sensitive headers in proxied redirects)
+
+[Example](../examples/55_redirect_forwards_credentials.py)
+
+### 56. Nested Quantifiers Backtrack Exponentially
+
+Python's `re` engine is a backtracking matcher. A pattern such as `(a+)+$` can split a run of `a`s in exponentially many ways, and on a near-miss input it tries every split before giving up. Each extra character roughly doubles the time: in the example, 20 characters take about 50 times longer than 14. A few dozen characters can pin a CPU core for minutes.
+
+```python
+# The Trap
+re.match(r"(a+)+$", "a" * 14 + "!") # ~0.0005 s
+re.match(r"(a+)+$", "a" * 20 + "!") # ~0.03 s
+re.match(r"(a+)+$", "a" * 40 + "!") # Hours.
+```
+
+Real patterns rarely look this obvious. The usual shape is two adjacent sub-patterns that can match the same characters, such as `(\w+\s?)+`, `(.*,)*` or `^(\d+)*$`, applied to a URL, email address or header.
+
+**The fix:** Remove the ambiguity (`a+$` matches the same strings in linear time), cap input length before matching, and on Python 3.11+ use possessive quantifiers (`a++`) or atomic groups (`(?>...)`), which never backtrack. For untrusted input at scale, the third-party `re2` bindings guarantee linear-time matching.
+
+**In the wild:** `urllib3` (CVE-2021-33503, URLs with many `@` characters) and `idna` (CVE-2024-3651, crafted domain names) both had catastrophic backtracking in parsers that sit under almost every Python HTTP request. There were 80 ReDoS advisories on PyPI since 2023.
+
+[Example](../examples/56_regex_backtracking.py)
+
+### 57. Decompression Bombs
+
+`zlib.decompress()`, `gzip.decompress()` and friends return the *whole* output, however large. Compression ratios of 1,000:1 are trivial to produce, and nesting layers takes them far higher, so a request body of a few kilobytes can allocate gigabytes.
+
+```python
+# The Trap
+bomb = zlib.compress(b"\0" * 50_000_000, 9) # 48,610 bytes
+zlib.decompress(bomb) # 50,000,000 bytes in memory
+```
+
+**Why:** Size limits on the wire say nothing about size after decoding. Every layer that transparently decompresses (HTTP `Content-Encoding`, gzip files, zip archives, image formats) is a place where a small input becomes a large one.
+
+**The fix:** Decompress incrementally with an output cap, using `zlib.decompressobj().decompress(data, max_length)` and checking `unconsumed_tail`, and apply limits after decoding as well as before.
+
+**In the wild:** `urllib3` fixed decompression bombs in its streaming API three times between December 2025 and May 2026 (CVE-2025-66471, then CVE-2026-21441 for a bypass via redirects, then a further bypass). There were 31 advisories for this weakness (CWE-409) on PyPI since 2023.
+
+[Example](../examples/57_decompression_bomb.py)
+
+### 58. Loading Data Can Run Code
+
+`pickle.load()` doesn't just read data. The pickle format can instruct the loader to import any importable callable and call it with arguments, which is how it rebuilds arbitrary objects. Loading a pickle from an untrusted source is equivalent to running a program from that source. The same is true of `yaml.load()` with PyYAML's full loader, `shelve`, and every machine-learning checkpoint format built on pickle, including PyTorch's `.pt` and `.pth` files and many `joblib` files.
+
+**Why:** The interface is named `load` and returns data, but the format is a small stack-based language. Nothing in the call site tells you that.
+
+**The fix:**
+
+- Use formats that can only describe data: JSON, MessagePack, Protocol Buffers, or `safetensors` for model weights. Use `yaml.safe_load()`.
+- If you must pickle, only load files you produced yourself, and authenticate them (for example with an HMAC over the bytes) so a swapped file is rejected before it's loaded.
+- Don't rely on scanners to make untrusted pickles safe (see below).
+
+**In the wild:** Deserialization of untrusted data (CWE-502) accounts for 210 PyPI advisories since 2023, concentrated in machine-learning tools that load models from the internet. [PyTorch 2.6 changed `torch.load`](https://dev-discuss.pytorch.org/t/bc-breaking-change-torch-load-is-being-flipped-to-use-weights-only-true-by-default-in-the-nightlies-after-137602/2573) to `weights_only=True` by default, and CVE-2025-32434 then showed that even that restricted mode could be bypassed in earlier versions. `picklescan`, a tool built to detect malicious pickles, has had 58 advisories of its own, most of them ways to get a malicious file past it undetected.
+
+*No runnable example: demonstrating this means writing a working payload.*
+
+### 59. One Key Parameter, Two Meanings
+
+Token libraries often take a single `key` argument whose meaning depends on the algorithm: for HMAC it's a shared *secret*, for RSA or ECDSA it's a *public* key. If the algorithm is read from the token itself, an attacker can choose it. A token labeled with an HMAC algorithm, verified against a server's *public* key, is checked using a value the attacker already knows.
+
+```python
+# The Trap: the token decides how it gets verified.
+jwt.decode(token, public_key_pem) # Older APIs inferred the algorithm.
+
+# The fix: the server decides.
+jwt.decode(token, public_key_pem, algorithms=["RS256"])
+```
+
+**Why:** One loosely typed parameter (`str | bytes`) carries two security-critical meanings, and the choice between them is made by untrusted input. The interface is shallow in exactly the sense this repo is about: simple to call, impossible to reason about without knowing the internals.
+
+**The fix:** Always pass an explicit allow-list of algorithms, never one taken from the token. Use typed key objects rather than raw strings where the library supports them, and keep HMAC secrets and asymmetric keys in separate code paths.
+
+**In the wild:** PyJWT has had 23 advisories since 2023, and 15 of them involve keys or algorithms: public keys in DER form, JWK containers or whitespace-mutated PEM accepted as HMAC secrets, and empty HMAC keys accepted. That's 41 signature-verification advisories (CWE-347) across PyPI.
+
+*No runnable example: demonstrating this means forging a token.*
diff --git a/examples/53_path_join_absolute.py b/examples/53_path_join_absolute.py
new file mode 100644
index 0000000..417273f
--- /dev/null
+++ b/examples/53_path_join_absolute.py
@@ -0,0 +1,35 @@
+"""os.path.join and pathlib discard everything before an absolute component."""
+
+import os
+import tempfile
+from pathlib import Path
+
+# The trap: a "safe" upload directory plus a user-supplied filename.
+upload_dir = "/srv/app/uploads"
+assert os.path.join(upload_dir, "report.pdf") == "/srv/app/uploads/report.pdf"
+print(os.path.join(upload_dir, "/etc/hosts")) # /etc/hosts (the prefix is gone)
+assert os.path.join(upload_dir, "/etc/hosts") == "/etc/hosts"
+assert Path(upload_dir) / "/etc/hosts" == Path("/etc/hosts")
+
+# Relative segments escape too, and join() never normalizes them.
+assert os.path.join(upload_dir, "../config.py") == "/srv/app/uploads/../config.py"
+
+
+# The fix: resolve the final path and check it is still inside the base directory.
+def safe_join(base, name):
+ base = Path(base).resolve()
+ target = (base / name).resolve()
+ if not target.is_relative_to(base):
+ raise ValueError(f"{name!r} escapes {base}")
+ return target
+
+
+with tempfile.TemporaryDirectory() as d:
+ assert safe_join(d, "report.pdf").name == "report.pdf"
+ for bad in ["/etc/hosts", "../config.py", "a/../../config.py"]:
+ try:
+ safe_join(d, bad)
+ except ValueError:
+ pass
+ else:
+ raise AssertionError(f"expected ValueError for {bad}")
diff --git a/examples/54_tarfile_extraction_filter.py b/examples/54_tarfile_extraction_filter.py
new file mode 100644
index 0000000..bb51f6d
--- /dev/null
+++ b/examples/54_tarfile_extraction_filter.py
@@ -0,0 +1,57 @@
+"""tarfile.extractall trusted member paths until Python 3.14 (CVE-2007-4559)."""
+
+import io
+import os
+import sys
+import tarfile
+import tempfile
+import warnings
+
+if not hasattr(tarfile, "data_filter"):
+ print("this Python predates tarfile extraction filters (PEP 706); skipping")
+ sys.exit(0)
+
+
+def archive_with(name):
+ buf = io.BytesIO()
+ with tarfile.open(fileobj=buf, mode="w") as tar:
+ info = tarfile.TarInfo(name)
+ info.size = 4
+ tar.addfile(info, io.BytesIO(b"data"))
+ buf.seek(0)
+ return buf
+
+
+with tempfile.TemporaryDirectory() as root:
+ dest = os.path.join(root, "dest")
+ os.mkdir(dest)
+ escaped = os.path.join(root, "outside.txt")
+
+ # The trap: a member named "../outside.txt" lands outside the destination
+ # on Python < 3.14, where extractall() defaults to the "fully_trusted" filter.
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore", DeprecationWarning)
+ try:
+ with tarfile.open(fileobj=archive_with("../outside.txt")) as tar:
+ tar.extractall(dest)
+ except tarfile.FilterError:
+ pass
+
+ if sys.version_info >= (3, 14):
+ print("3.14+: refused by the default 'data' filter")
+ assert not os.path.exists(escaped)
+ else:
+ print("before 3.14: written outside the destination directory")
+ assert os.path.exists(escaped)
+ os.remove(escaped)
+
+ # The fix: always pass filter="data" (backported to 3.8.17+, 3.9.17+,
+ # 3.10.12+, 3.11.4+), so behavior doesn't depend on the Python version.
+ try:
+ with tarfile.open(fileobj=archive_with("../outside.txt")) as tar:
+ tar.extractall(dest, filter="data")
+ except tarfile.OutsideDestinationError as e:
+ print(f"refused: {type(e).__name__}")
+ else:
+ raise AssertionError("expected OutsideDestinationError")
+ assert not os.path.exists(escaped)
diff --git a/examples/55_redirect_forwards_credentials.py b/examples/55_redirect_forwards_credentials.py
new file mode 100644
index 0000000..ff90900
--- /dev/null
+++ b/examples/55_redirect_forwards_credentials.py
@@ -0,0 +1,67 @@
+"""urllib.request forwards Authorization and Cookie headers to wherever a redirect points."""
+
+import threading
+import urllib.parse
+import urllib.request
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+received = {}
+
+
+class OtherHost(BaseHTTPRequestHandler):
+ def do_GET(self):
+ received["Authorization"] = self.headers.get("Authorization")
+ received["Cookie"] = self.headers.get("Cookie")
+ self.send_response(200)
+ self.end_headers()
+
+ def log_message(self, *args):
+ pass
+
+
+other = HTTPServer(("127.0.0.1", 0), OtherHost)
+threading.Thread(target=other.serve_forever, daemon=True).start()
+
+
+class Api(BaseHTTPRequestHandler):
+ def do_GET(self):
+ # Redirect to a different host ("localhost" vs "127.0.0.1") and port.
+ self.send_response(302)
+ self.send_header("Location", f"http://localhost:{other.server_port}/")
+ self.end_headers()
+
+ def log_message(self, *args):
+ pass
+
+
+api = HTTPServer(("127.0.0.1", 0), Api)
+threading.Thread(target=api.serve_forever, daemon=True).start()
+headers = {"Authorization": "Bearer api-token", "Cookie": "session=abc123"}
+
+# The trap
+urllib.request.urlopen(urllib.request.Request(f"http://127.0.0.1:{api.server_port}/", headers=headers)).read()
+print(received) # {'Authorization': 'Bearer api-token', 'Cookie': 'session=abc123'}
+assert received == headers
+
+
+# The fix: drop credentials when a redirect changes scheme, host or port.
+class SameOriginCredentials(urllib.request.HTTPRedirectHandler):
+ SENSITIVE = {"authorization", "cookie", "proxy-authorization"}
+
+ def redirect_request(self, req, fp, code, msg, hdrs, newurl):
+ new = super().redirect_request(req, fp, code, msg, hdrs, newurl)
+ old, target = urllib.parse.urlsplit(req.full_url), urllib.parse.urlsplit(newurl)
+ if new is not None and (old.scheme, old.netloc) != (target.scheme, target.netloc):
+ for name in list(new.headers):
+ if name.lower() in self.SENSITIVE:
+ del new.headers[name]
+ return new
+
+
+received.clear()
+opener = urllib.request.build_opener(SameOriginCredentials)
+opener.open(urllib.request.Request(f"http://127.0.0.1:{api.server_port}/", headers=headers)).read()
+assert received == {"Authorization": None, "Cookie": None}
+
+api.shutdown()
+other.shutdown()
diff --git a/examples/56_regex_backtracking.py b/examples/56_regex_backtracking.py
new file mode 100644
index 0000000..9097f02
--- /dev/null
+++ b/examples/56_regex_backtracking.py
@@ -0,0 +1,32 @@
+"""Nested quantifiers make regex matching time grow exponentially (ReDoS)."""
+
+import re
+import sys
+import time
+
+
+def seconds(pattern, text):
+ start = time.perf_counter()
+ re.match(pattern, text)
+ return time.perf_counter() - start
+
+
+# The trap: (a+)+ can split a run of "a"s in exponentially many ways, and tries
+# every one of them before concluding the trailing "!" doesn't match.
+nested = r"(a+)+$"
+re.match(nested, "") # Compile and cache first, so only matching is timed.
+short = seconds(nested, "a" * 14 + "!")
+longer = seconds(nested, "a" * 20 + "!")
+print(f"14 chars: {short:.4f}s, 20 chars: {longer:.4f}s ({longer / short:.0f}x slower)")
+assert longer > short * 8 # Theory says 64x; each extra character doubles the work.
+
+# The fix: remove the nesting. Same language, linear time.
+flat = r"a+$"
+assert seconds(flat, "a" * 100_000 + "!") < 0.5
+assert re.match(nested, "aaaa") and re.match(flat, "aaaa")
+
+# Python 3.11+ also has possessive quantifiers and atomic groups, which never backtrack.
+if sys.version_info >= (3, 11):
+ assert seconds(r"(?>a+)+$", "a" * 100_000 + "!") < 0.5
+
+# And bound input length before matching anything user-supplied.
diff --git a/examples/57_decompression_bomb.py b/examples/57_decompression_bomb.py
new file mode 100644
index 0000000..8a37251
--- /dev/null
+++ b/examples/57_decompression_bomb.py
@@ -0,0 +1,30 @@
+"""A few kilobytes of compressed data can expand to gigabytes in memory."""
+
+import zlib
+
+# The trap: 48 KB on the wire, 50 MB after decompress().
+bomb = zlib.compress(b"\0" * 50_000_000, 9)
+print(f"compressed {len(bomb):,} bytes") # ~48,610 bytes
+assert len(bomb) < 100_000
+assert len(zlib.decompress(bomb)) == 50_000_000 # Real payloads nest layers to reach GBs.
+
+# The fix: decompress incrementally with a hard cap on output size.
+LIMIT = 1_000_000
+
+
+def bounded_decompress(data, limit=LIMIT):
+ d = zlib.decompressobj()
+ out = d.decompress(data, limit)
+ if d.unconsumed_tail or not d.eof:
+ raise ValueError(f"decompressed size exceeds {limit:,} bytes")
+ return out
+
+
+try:
+ bounded_decompress(bomb)
+except ValueError as e:
+ print(e) # decompressed size exceeds 1,000,000 bytes
+else:
+ raise AssertionError("expected ValueError")
+
+assert bounded_decompress(zlib.compress(b"hello")) == b"hello"
+
+
+