[chapmanjacobd]

Summary

This PR lets independent reader workers search concurrently in threads and adds an option to skip snippets when they are not needed.

• Releases the GIL around blocking native work: archive construction, match estimation and retrieval, iterator setup and advancement, and result metadata/snippet extraction.
• Keeps Python work under the GIL, including filename encoding, decoding and object creation, plus searcher/query setup. Each search snapshots the searcher's Python archive list.
• Adds keyword-only Search.get_results(..., include_snippets=True). The default preserves existing behavior; False skips native article reads and parsing for snippets, leaving SearchResult.snippet empty.
• Documents worker-private archive, searcher, query and iterator ownership. This is not a general thread-safety guarantee for shared mutable objects.

Suggestions, item-content cache synchronization and the existing creator locking contract are unchanged.

Native dependency and thread-safety scope

Threaded reader use should be paired with openzim/libzim#1141, which fixes the existing search guard adopting mutexes that were never locked. This PR does not update the default libzim 9.8.2 bundle; coordinate a native release/bundle update before recommending threaded deployments.

This does not certify every native library version or arbitrary shared-object usage. Workers should open independent Archive handles, rather than only creating multiple Searchers from one shared archive.

Validation

• 157 reader/creator tests pass against the patched native library.
• Regressions cover four concurrent workers with independent handles, both snippet modes, result identity when snippets are disabled, and Python-thread progress during native snippet work.
• Ruff, Black and Pyright pass for the changed Python test and public stub.

Benchmark results

Skipping snippets accounts for most of the overall speedup. Compared with multiprocessing using include_snippets=False, multithreading adds 22% desktop throughput and 47% on the 4-CPU / 2 GiB profile. From multiprocessing with include_snippets=True to multithreading with include_snippets=False, sampled peak PSS falls 62% and 48%, respectively. The 29–43x total speedup is not a claim about GIL release alone.

Multiprocessing: include_snippets=True vs. include_snippets=False

The include_snippets=True configuration uses the multiprocessing approach and generates snippets that the application then discards. The comparison uses include_snippets=False while retaining multiprocessing.

Resource profile	Configuration	Searches/s	Speedup vs. include_snippets=True	CPU ms/search	Avg. CPU cores	Peak PSS MiB	p50 ms	p95 ms
Desktop, 8 workers	Multiprocessing, include_snippets=True	315	1.0x	23.898	7.48	1,484	22.56	50.42
Desktop, 8 workers	Multiprocessing, include_snippets=False	7,543	24.0x	0.703	5.30	644	0.87	1.83
4-CPU / 2 GiB profile, 4 workers	Multiprocessing, include_snippets=True	183	1.0x	21.137	3.85	1,069	20.18	46.87
4-CPU / 2 GiB profile, 4 workers	Multiprocessing, include_snippets=False	5,373	29.4x	0.665	3.58	596	0.58	1.48

Multiprocessing vs. multithreading (include_snippets=False)

Both configurations use this reader with include_snippets=False. Multithreading uses a bounded thread pool and independently opened archive/searcher handles.

Resource profile	Configuration	Searches/s	Speedup vs. multiprocessing	CPU ms/search	Avg. CPU cores	Peak PSS MiB	p50 ms	p95 ms
Desktop, 8 workers	Multiprocessing	7,543	1.0x	0.703	5.30	644	0.87	1.83
Desktop, 8 workers	Multithreading	9,197	1.2x	0.574	5.25	562	0.77	1.40
4-CPU / 2 GiB profile, 4 workers	Multiprocessing	5,373	1.0x	0.665	3.58	596	0.58	1.48
4-CPU / 2 GiB profile, 4 workers	Multithreading	7,904	1.5x	0.449	3.54	556	0.42	0.99

Method

All configurations use the same rebuilt native dependencies: Python 3.13.12, libzim 9.8.2 with openzim/libzim#1141, Xapian 1.4.31 and ICU 78.3. The controlled baseline rebuilds unchanged python-libzim 57555c2 against that stack to avoid a native dependency-version confound.

The desktop is an i5-13600KF with 20 logical CPUs and 32 GiB RAM, limited to CPUQuota=2000% and MemoryMax=8192M. The 4-CPU / 2 GiB profile uses CPUQuota=400% and MemoryMax=2048M on the same host; it is not Raspberry Pi hardware. Swap is disabled for both. Desktop runs use eight source callers and native workers; 4-CPU / 2 GiB profile runs use four of each.

Values are medians of three fresh-service trials per configuration. Each trial warms 144 searches, then repeats the 144-search batch for at least three measured seconds. A batch contains twelve queries against each of three archives (Wikipedia astronomy, Wikipedia medicine and Cooking Stack Exchange), with a sixteen-hit limit. One search is one query against one archive.

Measurement notes

CPU includes the driver and all workers. Peak PSS is sampled across the process tree and includes a retained 512 MiB server/model stand-in. Latencies include backend admission waits but exclude source-executor queueing. OS caches remain warm, and file pages may be charged outside the service cgroup. These are native-search microbenchmarks, not whole-server latency, model/index residency guarantees or Raspberry Pi performance. All 81 matrix, baseline and confirmation trials returned the same path/title digest.

[chapmanjacobd]

suggested ordering: merge/release openzim/libzim#1141, bump LIBZIM_DL_VERSION, and only then merge this PR