Repository navigation
Support à la carte dependencies #244
Description
scraperlib grew significantly over the years, with modules very useful even beyond pure scraper usages. The ZIM wrapper is very useful in itself to name only one.
We are also considering addition of the currently separate kiwixstorage module, or the Kiwix catalog reader.
To support this Kiwix Swiss-army knife layout, we'd need to be able to install only some of the dependencies.
Dependencies would thus be organized into extra groups (ZIM, image, video, etc) with an all group that installs everything of course.
In addition, we could maybe had a try block on imports in each module to print a user-actionable message when importing code for which dependencies are not installed.
Activity
benoit74 commentedon Jan 14, 2025
Agreed, especially if it allows to much clearly define system dependencies. Currently the documentation says that we need a bunch of system dependencies, but it should be clearer that these are not required in all extra. And usually they are in fact needed only for few methods. Would be cool if you could map this information somewhere in the documentation.
pawank925 commentedon Nov 13, 2026
@benoit74 I'd like to take this. I went through what
src/actually imports first, so the groups come from the graph and not from the module names, and it turned up one thing that blocks the whole idea.The easy part. The 23 runtime deps split as:
beartype, because__init__.pyarmsbeartype_this_package(). Nothing else is reachable from a bareimport zimscraperlib.filesystemvideo/never imports yt-dlp.The blocker is that the families are not leaves.
zim.creator->zim.metadata->inputs->download->yt_dlp, so a scraper that only writes a ZIM today takes yt-dlp with it;zim.metadata->image.probingtakes pillow and colorthief,zim.filesystem->htmltakes bs4 and lxml. Resolvingzimscraperlib.zimgives 13 of the 23 packages, so azimextra as things stand would buy almost nothing over installing everything.So I'd split it in two steps:
unique_valueshas nothing to do with downloading and can leaveinputs.stream_filebecomes a function-level import ininputs.py,zim/items.pyandzim/providers.py, andis_valid_imageinzim.metadatathe same way. This step changes no dependency and breaks no scraper.dependenciesinto extras named after the families, keep anallthat installs everything, and handle the "user-actionable message" with a__getattr__at each subpackage level (PEP 562) raisingImportErrornaming the extra to install: one helper instead of a try/except in every module.Tests: a CI job per extra, each in a fresh venv installing only that extra and running the matching subset of
tests/; a bare install asserting every family raises the message naming its extra rather than a rawModuleNotFoundError; and a check that.[all]resolves to the same 23 packages as this release so no scraper loses one quietly.Step 2 shrinks the default install, which breaks anyone writing
zimscraperlibwithout an extra, so it belongs to the next major. Is it fine to send step 1 now, since it is not breaking on its own?pawank925 commentedon Nov 13, 2026
Sent step 1 as #403.
Measured on a fresh interpreter rather than estimated:
import zimscraperlib.zimgoes from 19 of the 23 runtime deps to 16 (yt_dlp,bs4,lxmloff the graph),import zimscraperlib.inputsfrom 5 to 1. Correction: the 13 in my previous comment came from a static walk that missed transitive imports. The runtime figure is 19, so the graph is worse than I said.Two parts of that plan did not survive measuring, both in the PR description:
is_valid_image/is_valid_iso_639_3cannot be moved to call level whilezim/metadata.pyvalidatesDEFAULT_DEV_ZIM_METADATAduring its own import — the pillow and Babel loads happen there, not at the import statement — andunique_valuesdid not need to leaveinputsat all oncestream_filedid.