Repository navigation
Conversation
Move stream_file, get_retry_adapter and find_title_in_file into the functions that call them, so importing a family no longer loads another family's dependencies. No dependency changed and no API moved.
Contributor
Author
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #403 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 42 42
Lines 2952 2954 +2
Branches 452 452
=========================================
+ Hits 2952 2954 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of the approach in #244. No dependency changed, no name moved, nothing a scraper calls behaves differently.
Measured
sys.modulesof a fresh interpreter with every dependency installed:zimscraperlib.zimzimscraperlib.inputsGone from the
zimgraph:yt_dlp,bs4,lxml.Correction to what I quoted in #244: I said 13, from a static AST walk that missed transitive imports. The runtime number is 19 of 23, so the graph is worse than stated, not better.
Changes
inputs.py:stream_fileimported in the http branch ofhandle_user_provided_file, so a local path no longer loads the download stack.zim/items.py:stream_fileimported inURLItem.download_for_sizeandURLItem.__init__.zim/providers.py:get_retry_adapterinURLProvider.__init__,stream_fileinURLProvider.get_size_of.zim/filesystem.py:find_title_in_fileinFileItem.get_title.tests/inputs/test_inputs.py: the threemonkeypatch.setattrcalls now patchzimscraperlib.download— the module the name is looked up on at call time.tests/test_import_boundaries.py: new. Hides yt-dlp (or bs4+lxml) from a child interpreter with asys.meta_pathfinder and assertszimscraperlib.zim/zimscraperlib.inputsimport anyway, then asserts that hiding yt-dlp does breakzimscraperlib.download— without that self-check the first assertions could pass vacuously.pyproject.toml:PLC0415ignored, with the reason. The rule forbids precisely the pattern this PR introduces and step 2 spreads, so either the config says so or 6# noqas do.Two edges I did not cut, and why
zim/metadata.pykeepsis_valid_imageandis_valid_iso_639_3at module level, andunique_valuesstays ininputs.get_cleaned_valuechanges no number:metadata.pybuildsDEFAULT_DEV_ZIM_METADATAwhile it is being imported, and the illustration and language entries in it are validated through pillow and Babel at that moment. So the modules load atimport zimscraperlib.zimeither way, and the edit would be churn. Cutting that edge for real means not validating that constant at import time, which is a behavior change rather than an import move — a step 2 decision. For the size of the prize I measured it with a throwaway patch skipping the construction:zimdrops from 16 to 11, and to 8 (beartype, idna, libzim, pymupdf-lite, python-magic, regex, requests, urllib3) oncecreator.py'sPIL.Imageandi18nimports get the same call-level treatment. That patch is not in this PR.unique_valuesdid not need to leaveinputs. Oncestream_fileis call-level there,inputspulls nothing but the package itself, sometadata -> inputscosts no dependency, and moving it would be a public name move for nothing.Tests
pytest --runslow --runinstalled: 1325 passed, 37 failed, 33 skipped. The same 37 fail on a cleanmainhere (no ffmpeg, no gifsicle; exit 127), so nothing moved in either direction.ruff 0.17.3 checkandformat --check,pyright 1.1.414: clean.