Show how hidden complexity becomes real-world bugs: security pitfalls, vulnerability data and charts

CONTRIBUTING.md

@@ -19,6 +19,7 @@ Found a Python behavior that surprised you? Please add it.
    for f in examples/*.py; do python "$f" || echo "FAILED: $f"; done
    ```
 
-4. Open a pull request. CI runs the examples on CPython 3.10 to 3.14.
+4. If there's a documented real-world bug, CVE or fix caused by the pitfall, add an **In the wild** paragraph linking to it.
+5. Open a pull request. CI runs the examples on CPython 3.10 to 3.14.
 
 If behavior differs between Python versions, say which versions in the README and guard the example with `sys.version_info`.

README.md

@@ -6,13 +6,29 @@
 
 Examining the gap between Python's readable interfaces and the underlying complexity, implicit state, and unexpected behavior they can conceal.
 
-A catalog of 52 Python gotchas, traps and surprising behaviors, from beginner-level surprises to asyncio, threading, CPython internals and data-engineering bugs. Each comes with a minimal reproduction, an explanation of *why* it happens, and the fix.
+A catalog of 59 Python gotchas, traps and surprising behaviors, from beginner-level surprises to asyncio, threading, CPython internals, data-engineering bugs and security vulnerabilities. Each comes with a minimal reproduction, an explanation of *why* it happens, the fix, and, where one exists, the real-world bug or CVE it caused.
 
 Python's syntax prioritizes readability, but readable code does not always make runtime behavior predictable. Some pitfalls arise from implicit state, others from surprising semantics or abstractions that leak underlying details. John Ousterhout's *A Philosophy of Software Design* calls a particularly troublesome symptom of complexity *unknown unknowns*: behavior you cannot predict from the interface in front of you. When the language's mechanics don't match its readable syntax, developers may discover those differences only through runtime bugs.
 
-Here is a collection of Python pitfalls developers often discover the hard way.
+Here is a collection of Python pitfalls developers often discover the hard way, and the evidence that they don't stay in tutorials.
 
-> Every snippet here and in [`docs/`](docs/) has a runnable, self-checking twin in [`examples/`](examples/). CI runs them on CPython 3.10 to 3.14, so the outputs shown here are verified, not remembered.
+> Every pitfall except two security entries has a runnable, self-checking example in [`examples/`](examples/). CI runs them on CPython 3.10 to 3.14, so the outputs shown here are verified, not remembered.
+
+## How the complexity shows up in production
+
+Each row is a kind of hidden complexity, a pitfall that demonstrates the mechanism, and what it has caused in real code. Advisory counts are unique PyPI vulnerabilities published since 2023, from the [OSV](https://osv.dev) database ([method](docs/dependency-risk.md#method-and-limitations)).
+
+| Hidden complexity | Pitfalls | PyPI advisories since 2023 | Real-world cases |
+|---|---|---:|---|
+| A path join silently discards your prefix | [#53](docs/security.md#53-ospathjoin-discards-your-prefix), [#54](docs/security.md#54-tarfile-trusted-member-paths-until-python-314) | 507 (path traversal, the #1 weakness) | `tarfile` CVE-2007-4559, open for 15 years |
+| Parsing cost grows with the attacker's input | [#56](docs/security.md#56-nested-quantifiers-backtrack-exponentially), [#57](docs/security.md#57-decompression-bombs), [#42](docs/interpreter.md#42-strint-fails-above-4300-digits) | 549 (denial of service) | `urllib3` CVE-2021-33503, `idna` CVE-2024-3651, CPython CVE-2020-10735 |
+| Loading data runs code | [#58](docs/security.md#58-loading-data-can-run-code) | 210 (deserialization) | `torch.load` CVE-2025-32434; 58 bypasses of `picklescan` |
+| A convenience hides a protocol decision | [#55](docs/security.md#55-redirects-forward-credentials-to-another-host) | 60 (redirect-related leaks) | `requests` CVE-2018-18074, `urllib3` CVE-2023-43804 |
+| One parameter, two meanings | [#59](docs/security.md#59-one-key-parameter-two-meanings) | 41 (signature verification) | PyJWT: 15 key or algorithm advisories since 2023 |
+| Hash values change per process | [#43](docs/data-systems.md#43-hash-changes-between-processes) | | CPython CVE-2012-1150, the reason `hash()` is randomized |
+| Wall-clock vs elapsed time | [#48](docs/data-systems.md#48-aware-datetime-arithmetic-is-wall-clock-arithmetic), [#49](docs/data-systems.md#49-a-utc-offset-is-not-a-time-zone) | | Home Assistant: [alert loop](https://github.com/home-assistant/core/pull/188537), [backups an hour off](https://github.com/home-assistant/core/pull/188836) |
+| The event loop is shared by everything | [#20](docs/concurrency.md#20-fire-and-forget-tasks-get-garbage-collected), [#22](docs/concurrency.md#22-blocking-calls-freeze-the-event-loop), [#24](docs/concurrency.md#24-loop-methods-arent-thread-safe) | | Home Assistant's [guard rails](#in-the-wild-home-assistant) and [Hikvision fix](https://github.com/home-assistant/core/pull/160027) |
+| Code you never chose | [Dependency risk](docs/dependency-risk.md) | | `urllib3` and `certifi` reach 145 and 144 of the top 1,000 packages, 86–87% indirectly |
 
 ## Contents
 
@@ -54,9 +70,16 @@ Here is a collection of Python pitfalls developers often discover the hard way.
 | [Object model & imports](docs/object-model.md) | 31–37 | Properties vs `__getattr__`, `@lru_cache` leaks, `__eq__` vs `__hash__`, `__exit__` swallowing errors, weak refs to bound methods, `mock.patch` targets, double-imported `__main__` |
 | [Interpreter internals](docs/interpreter.md) | 38–42 | `python -O` stripping asserts, class-body comprehension scope, `locals()` writes, `id()` reuse, the 4300-digit `int`/`str` limit |
 | [Data systems: time, numbers, bytes](docs/data-systems.md) | 43–52 | `hash()` sharding, NaN in containers, nanosecond timestamps in floats and `datetime`, float prices, DST arithmetic, fixed UTC offsets, `utcnow()`, `struct` alignment, non-standard JSON |
+| [Security: where complexity becomes a CVE](docs/security.md) | 53–59 | `os.path.join` with absolute paths, `tarfile` extraction, credentials forwarded on redirect, regex backtracking, decompression bombs, unsafe deserialization, signing-key confusion |
 
 See also [In the wild](#in-the-wild-home-assistant) for how a large production codebase hits and guards against these.
 
+### Part III: Research
+
+| Page | Covers |
+|------|--------|
+| [Complexity in the wild](docs/dependency-risk.md) | What Python's vulnerability data shows, organized by mechanism: 7,032 OSV advisories joined with the dependency graphs of the top 1,000 packages, yanked releases, OpenSSF Scorecard and typosquatting data, with charts |
+
 ---
 
 ## Iterators & laziness

analysis/README.md

@@ -0,0 +1,38 @@
+# Dependency Risk Analysis
+
+Scripts behind [docs/dependency-risk.md](../docs/dependency-risk.md). They download public data, join it, and print every number used in the report.
+
+`dependency_risk.csv` is the joined snapshot from 30 October 2026: one row per package that at least 5 of the top 1,000 PyPI packages depend on, with reach, advisory counts, yanked releases and Scorecard.
+
+## Reproduce
+
+Downloaded data is untrusted, so keep it in its own directory and run the scripts from here with `python -I`. Network scripts need Python 3.11+ with working CA certificates. `fetch_yanked.py` also needs `packaging`, and `charts.py` needs `matplotlib`.
+
+```sh
+DATA=/tmp/pypi-risk-data
+mkdir -p $DATA/osv $DATA/top $DATA/deps $DATA/simple $DATA/scorecard
+
+curl -L -o $DATA/osv/all.zip https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip
+unzip -q $DATA/osv/all.zip -d $DATA/osv/json
+curl -L -o $DATA/top/top30d.json https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json
+
+python -I osv_agg.py $DATA              # Merge advisories into unique vulnerabilities
+python -I fetch_deps.py $DATA 1000      # Resolved dependency graphs from deps.dev
+python -I fetch_yanked.py $DATA 1000    # Yanked releases from PyPI (needs packaging)
+python -I analyze.py $DATA              # Reach x vulnerabilities x yanks
+python -I fetch_scorecard.py $DATA      # OpenSSF Scorecard for the hidden layer
+python -I report.py $DATA               # Every table in the report
+python -I charts.py $DATA ../docs/img   # Light and dark SVG charts (needs matplotlib)
+```
+
+The fetch scripts cache one JSON file per package, so they can be rerun after a failure. Expect about five minutes in total. Results will drift as OSV, PyPI and deps.dev update.
+
+| Script | Output |
+|---|---|
+| `osv_agg.py` | `vulns.json`: unique vulnerabilities (aliases merged) and malicious-package counts |
+| `fetch_deps.py` | `deps/*.json`: resolved graph per top package, falling back to a recent release if deps.dev hasn't resolved the newest |
+| `fetch_yanked.py` | `simple/*.json`: release count and yanked versions with reasons |
+| `analyze.py` | `joined.json`: per-package reach, direct and indirect counts, advisories, yanks |
+| `fetch_scorecard.py` | `scorecard/*.json`: Scorecard for packages with reach ≥ 20 and mostly indirect use |
+| `report.py` | Prints the hidden-layer, burstiness, CWE, yanked, malicious-package, monthly and Scorecard tables |
+| `charts.py` | `docs/img/*-{light,dark}.svg`: advisories per year, reach vs advisories, weakness mix. Colors are a palette validated for color-vision deficiency in both themes |

analysis/analyze.py

@@ -0,0 +1,69 @@
+"""Join vulnerabilities, dependency reach, downloads and yanked releases."""
+import collections, glob, json, os, re, sys
+
+data = sys.argv[1]
+norm = lambda n: re.sub(r"[-_.]+", "-", n).lower()
+top = json.load(open(f"{data}/top/top30d.json"))["rows"]
+downloads = {norm(r["project"]): r["download_count"] for r in top}
+rank = {norm(r["project"]): i + 1 for i, r in enumerate(top)}
+roots = {norm(r["project"]) for r in top[:1000]}
+
+# Reach: how many top-1000 packages pull each package in, directly or only indirectly.
+direct, indirect, nodes_per_root = collections.defaultdict(set), collections.defaultdict(set), []
+for f in glob.glob(f"{data}/deps/*.json"):
+    d = json.load(open(f))
+    if "graph" not in d:
+        continue
+    root = norm(os.path.basename(f)[:-5])
+    if root not in roots:  # Ignore graphs cached for packages no longer in the top 1,000.
+        continue
+    nodes = d["graph"]["nodes"]
+    nodes_per_root.append(len(nodes) - 1)
+    for node in nodes:
+        name = norm(node["versionKey"]["name"])
+        if node["relation"] == "DIRECT":
+            direct[name].add(root)
+        elif node["relation"] == "INDIRECT":
+            indirect[name].add(root)
+reach = {p: direct[p] | indirect[p] for p in set(direct) | set(indirect)}
+only_indirect = {p: reach[p] - direct[p] for p in reach}
+
+v = json.load(open(f"{data}/vulns.json"))
+vulns, malicious = v["vulns"], v["malicious"]
+by_pkg = collections.defaultdict(list)
+for x in vulns:
+    for p in x["pkgs"]:
+        by_pkg[p].append(x)
+
+yanked = {}
+for f in glob.glob(f"{data}/simple/*.json"):
+    d = json.load(open(f))
+    if "error" not in d:
+        yanked[norm(os.path.basename(f)[:-5])] = d
+
+def row(p):
+    vs = by_pkg.get(p, [])
+    recent = [x for x in vs if x["published"] >= "2023-01-01"]
+    y = yanked.get(p, {})
+    return dict(
+        pkg=p, rank=rank.get(p), downloads=downloads.get(p, 0),
+        reach=len(reach.get(p, ())), direct=len(direct.get(p, ())), only_indirect=len(only_indirect.get(p, ())),
+        vulns=len(vs), vulns_2023=len(recent),
+        high_crit=sum(1 for x in vs if x["sev"] >= 3),
+        cwes=collections.Counter(c for x in vs for c in x["cwes"]).most_common(3),
+        yanked=len(y.get("yanked", [])), versions=y.get("n_versions"),
+    )
+
+rows = [row(p) for p in reach]
+json.dump({"rows": rows, "roots_with_graph": len(nodes_per_root),
+           "median_deps": sorted(nodes_per_root)[len(nodes_per_root) // 2],
+           "mean_deps": sum(nodes_per_root) / len(nodes_per_root)}, open(f"{data}/joined.json", "w"))
+
+print(f"roots with graphs: {len(nodes_per_root)}, median transitive deps {sorted(nodes_per_root)[len(nodes_per_root)//2]}, mean {sum(nodes_per_root)/len(nodes_per_root):.1f}")
+fmt = "{pkg:22} rank={rank!s:>5} reach={reach:>4} direct={direct:>4} only_ind={only_indirect:>4} vulns={vulns:>3} since23={vulns_2023:>3} hi/crit={high_crit:>3} yanked={yanked:>3}/{versions!s:<4} {cwes}"
+print("\n== Dependencies with reach >= 10, by all-time vulns")
+for r in sorted((r for r in rows if r["reach"] >= 10 and r["vulns"]), key=lambda r: -r["vulns"])[:30]:
+    print(fmt.format(**r))
+print("\n== Exposure (vulns since 2023 x reach), reach >= 10")
+for r in sorted((r for r in rows if r["reach"] >= 10 and r["vulns_2023"]), key=lambda r: -r["vulns_2023"] * r["reach"])[:20]:
+    print(fmt.format(**r), "score=", r["vulns_2023"] * r["reach"])