Skip to content

Keep existing L1 files when replacing them fails - #149

Open
namurphy wants to merge 1 commit into
PlasmaPy:mainfrom
namurphy:l1-replace-atomically
Open

namurphy wants to merge 1 commit into
PlasmaPy:mainfrom
namurphy:l1-replace-atomically

Conversation

@namurphy

Copy link
Copy Markdown
Member

Follow-up to #136, from its ultrareview.

Problem

With -o/--overwrite, _read_master_cdf deleted the existing L1 file before the new one was created. If writing the new file then failed (for example, a CDF library error, a full disk, or Ctrl-C), the cleanup in cdf_merge_to_daily deleted the partial file too. That left no L1 file at all for that day.

Change

  • Each L1 file is now written at a temporary path in the same directory, .<name>.<random hex>.tmp.cdf, and moved over the final path with Path.replace only once it is complete. The new file therefore replaces the old one in a single step, and if anything fails, the old file is left unchanged and the temporary file is deleted.
    • The temporary name starts with . so that it is hidden, and ends with .cdf because the CDF library adds .cdf to names that don't end with it.
  • Logical_file_id is still set to the final file name, which is now passed to _write_data_to_cdf explicitly.
  • The check for an existing file without --overwrite now happens in cdf_merge_to_daily, before anything is written.
  • Replacing a file is now noted in the "Wrote N records" log message, rather than in a separate "Overwriting" message before the file is written.
  • The README's "Differences from IDL" section notes the change. IDL's read_master_cdf copies the skeleton over the existing file.

The output files themselves are unchanged.

Testing

  • nox -s tests-3.9, nox -s tests-3.14, nox -s typecheck, nox -s lint, and nox -s docs pass. That includes the comparison with the IDL L1 files for 2026-08-03, which checks Logical_file_id.
  • New tests:
    • With overwrite, a failure after the data has been written (a CDFError or a KeyboardInterrupt) leaves the existing file unchanged and no temporary files behind.
    • A failure while writing a new file leaves nothing behind.
    • A successful overwrite leaves only the final file.

Limitations

  • Without --overwrite, the check for an existing file and the rename aren't one atomic step. A file created by another process in between would be replaced.
  • This covers part of the "make retries and parallel runs safe for Dagster" item in Add prototype l05_to_l1 tool ported from IDL #136. Log file names still only have one-second resolution.

🤖 Generated with Claude Code

With overwrite, an existing L1 file was deleted before the new one was
created, and if writing the new one then failed, the partial file was
deleted too, leaving no L1 file for that day. Each L1 file is now
written at a temporary path in the same directory and renamed over the
existing file only once it is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.45%. Comparing base (c66036e) to head (3945bfb).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #149      +/-   ##
==========================================
+ Coverage   76.19%   76.45%   +0.26%     
==========================================
  Files          15       15              
  Lines        1420     1423       +3     
  Branches      200      199       -1     
==========================================
+ Hits         1082     1088       +6     
+ Misses        298      295       -3     
  Partials       40       40              
Files with missing lines Coverage Δ
src/pyfaradaycup/l05_to_l1/cdf_to_survey.py 94.33% <100.00%> (+3.07%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant