Repository navigation
Fix: safe downloads and export fallback timeout; prepare 0.1.3 - #6
Merged
Merged
Conversation
download() (sync and async) wrote straight to dest, so a dropped connection or full disk left a partial file in place of the previous one. It now streams to a unique '<dest>.<random>.part' next to dest and replaces dest only when the body is complete (new _core.atomic_destination). Found in a Codex review; present since the first release.
The v3 fallback restarted the full timeout (timeout=300 could take ~600 s) and always wrote '<dest>.part', clobbering a file of that name and colliding with concurrent exports. It now gets only the time left and writes through atomic_destination. Found in a Codex review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5 (base
harden/strictness); GitHub retargets it tomainwhen #5 is merged and its branch deleted.Fixes three findings from the Codex review of #5. These change behaviour (for the better), so they are kept out of #5.
dest.client.download()(sync and async) wrote straight todest; a dropped connection or full disk left a partial file in place of the previous one. It now streams to a unique<dest>.<random>.partnext todestand replacesdestonly when the body is complete (_core.atomic_destination). Present since the first release; affects every download-based method, includingexport_to_file.export_to_filefallback stays within onetimeout. The v3 rerun on non-data-lake dataflows restarted the full timeout (timeout=300could take ~600 s); it now gets only the time left.<dest>.part, which overwrote a file of that name and collided with concurrent exports to the same destination.Each fix has tests that fail before it (
tests/test_safe_downloads.py, two new cases intests/test_export_fallback.py); both fix commits pass on their own.Also: version 0.1.3 and CHANGELOG (release after merge, as usual).