Skip to content

Add prototype l05_to_l1 tool ported from IDL - #136

Merged
namurphy merged 24 commits into
PlasmaPy:mainfrom
namurphy:l05-to-l1
Nov 19, 2026
Merged

namurphy merged 24 commits into
PlasmaPy:mainfrom
namurphy:l05-to-l1

Conversation

@namurphy

@namurphy namurphy commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Warning

This is a prototype. It ports the IDL L0.5 → L1 step closely enough to reproduce the five IDL L1 files for one day (2026-08-03). It has not been validated against production output beyond that day, and it should not be used in the pipeline yet. The additional testing that is needed is listed below.

Summary

This adds l05_to_l1, a click command line tool that merges SPC L0.5 CDF files into daily L1 CDF files, one per APID (351, 353, 354, 35E, 35F). It's a port of this IDL code, keeping the IDL function names and structure:

Python (pyfaradaycup.l05_to_l1) IDL
spp_swp_spc_l05_to_l1_dates, spp_swp_spc_l05_to_l1_date, spp_swp_spc_which_apid spp_swp_spc_l05_to_l1_dates.pro, spp_swp_spc_l05_to_l1_date.pro, spp_swp_spc_which_apid.pro
cdf_to_survey.cdf_merge_to_daily CDF_merge_to_daily in idllib/CDF_to_survey.pro (IDL names are case-insensitive)
_read_mycdf, _read_master_cdf, _write_data_to_cdf CDAWlib's read_myCDF, read_master_cdf, write_data_to_cdf
pyfaradaycup.utils.idl: cdf_epoch_tojuldays, julday, uniq IDL built-ins CDF_EPOCH_TOJULDAYS, JULDAY, UNIQ
_l05_children the sdc_state registry (spc_rawfile unique_dates / l05_children)
l05_to_l1 --l05dir=/path/to/L05 --l1dir=/path/to/L1 --logdir=/path/to/logs -v 2026-08-03

For each date, the L0.5 files with records on that date are combined. Only the records on that date are kept, and they are sorted by time, de-duplicated, and written into the L1 skeleton. Data values aren't changed. Logical_file_id and Generation_date are set the same way as by CDAWlib's write_data_to_cdf.

This uses two helpers that were split out of this PR so both tools can share them: console_level in pyfaradaycup._logging (#146) and the cdf_metadata test fixture in tests/conftest.py (#147). The only other changes to existing code are:

  • registering the new l05_to_l1 and utils subpackages (in pyproject.toml, pyfaradaycup/__init__.py, tests/test_imports.py, and the docs);
  • adding a fixture to tests/conftest.py;
  • moving _close_cdf out of the decommutator into pyfaradaycup.utils._cdf.close_cdf, so that the decommutator, l05_to_l1, and the test fixtures share one copy of the code that hides spacepy's harmless DID_NOT_COMPRESS warning.

The Python equivalents of IDL built-in functions are in a new general module, pyfaradaycup.utils.idl, so that other code ported from IDL can use them. They're public there and keep the IDL names. Their tests are in tests/utils/test_idl.py, and the tt2000 test fixture is in tests/conftest.py.

Where this mimics IDL (deliberately, for now)

  • Duplicates are removed by comparing float64 Julian days, as in IDL. Near the present, consecutive float64 values are about 40 µs apart, so records less than about 40 µs apart count as duplicates. A TODO in the code and in src/pyfaradaycup/l05_to_l1/README.md says to compare the exact int64 TT2000 values instead.
  • Of a set of duplicates, the record that comes last in input-file order is kept, matching IDL's uniq. IDL's sort isn't stable, so which duplicate IDL itself keeps is undefined.
  • Records are assigned to days by converting TT2000 to UTC with the CDF library. A time during a leap second (e.g. 2016-12-31T23:59:60.5) ends up on the next day.

Where this differs from IDL

  • There is no sdc_state registry. The L0.5 files for a date are found by searching L05/YYYY/MM/APID*/ for that month and the months on either side, reading each file's Epoch.
    • Only files whose names have an APID in SPC_L1_APIDS are opened. IDL selects the files for each APID after looking them up, so the files used are the same, but an unreadable file for another APID (such as 352) is not an error.
    • An L0.5 file for one of these APIDs that can't be read (for example, one that is truncated or still being written) is skipped with a warning, so it doesn't stop the L1 files for every date whose search includes it.
    • The APID is taken from the file name rather than the whole path, so _APID in a directory name isn't mistaken for it. As with IDL's strpos(...) gt 0, an _APID at the very start of the name doesn't count.
    • The dates in each L0.5 file are cached, keyed by its modification time and size, in place of the registry's unique_dates. When several dates are converted, each file is read once.
  • Variables are matched to the skeleton by name, not by position, so they may be in any order, and an error is raised if the skeleton and input files don't have the same variables. Records are selected along the record dimension, not along "the first dimension with as many elements as Epoch". The two are the same for the current L1 skeletons.
  • A warning is logged for each missing input file. IDL's check only looked at the last file.
  • An existing L1 file is only replaced with -o/--overwrite. IDL always overwrites. Without --overwrite, every APID's L1 file for a date is checked before any is written, so if one exists, none are written and the date isn't left with a mix of new and old files.
  • Dates can also be given as a datetime.datetime, which is converted to UTC before its date is taken if it has a time zone. IDL only accepts dates as strings or numbers. As in IDL, spp_swp_spc_which_apid and spp_swp_spc_l05_to_l1_dates also accept a single file name or date instead of a list.
  • The exit codes are 1 for a failure and 65 (EX_DATAERR) for a date with no L0.5 data, consistent with pfc_decommutator.

Testing done

  • nox -s tests-3.9 and nox -s tests-3.14 pass (including doctests); nox -s typecheck, nox -s docs, and prek pass.
  • Regression: for all five APIDs (351, 353, 354, 35E, 35F), the L1 files for 2026-08-03 made from the L0.5 files in _psp/ are identical to the IDL L1 files. The only exception is Generation_date. The comparison covers values, dtypes, variable and global attributes, and CDF metadata.
  • Unit tests with small synthetic L0.5 files cover:
    • sorting, and selecting records on either side of midnight;
    • which duplicate is kept, and near-duplicates (pinning the IDL-like behavior);
    • single-record days, and fill/pad Epoch values;
    • finding files in neighboring months, skipping APID352, not opening files for other APIDs, and skipping unreadable files;
    • finding the APID from the file name only;
    • reading each L0.5 file's dates once, and again after the file changes;
    • missing inputs, overwrite behavior (including writing no L1 files for a date if a later APID's file exists), mismatched skeletons, and variables in a different order from the skeleton;
    • single file names and dates, and datetimes with time zones;
    • Logical_file_id and Generation_date;
    • the CLI exit codes and log files.

Suggested additional testing before using this in production

  • Reproduce all five IDL L1 files for 2026-08-03. Every production L0.5 file that contributed to them is now in _psp/data/sci/sweap/spc/L05/, and the xfail marks in test_l1_matches_idl have been removed.
  • Cross-check the time conversions against IDL. The IDL docs don't say how CDF_EPOCH_TOJULDAYS handles leap seconds or how it computes its result. CDF_merge_to_daily also calls CDF_EPOCH, /BREAKDOWN on TT2000 values without /TT2000, and it isn't clear that IDL detects the type. Run the script below on the IDL machine, compare it with the expected values in its comments, and turn the outputs into test constants.
  • Compare several production days, not just one, by running IDL and Python on the same inputs. Include:
    • a day spanning two raw files that overlap (so duplicates are actually removed);
    • a day at a month boundary (L0.5 files filed under the previous month);
    • a high-cadence encounter day (for performance and memory);
    • a day with fill values in Epoch, if one exists.
  • Check that the skeletons in src/pyfaradaycup/data/cdf_skeletons/ match the production ones. They were copied but haven't been verified.
  • Check that downstream code (IDL L1 → L2) reads the Python L1 files without problems. The files are written with spacepy's CDF library version rather than CDF 3.8.1.
  • Decide how to handle duplicates, then implement and test the exact TT2000 comparison (see the TODO).
  • Make retries and parallel runs safe for Dagster. Write each L1 file to a temporary file and rename it when complete. Log file names only have one-second resolution, and file_mode="exclusive" makes two runs for the same date in the same second fail. pfc_decommutator has the same limitation.
IDL cross-check script
; L64 integer literals (LL suffix) and '(D25.17)' prints doubles at full precision.
; pyfaradaycup gives, for these values in order:
;   CDF_EPOCH_TOJULDAYS: 2461255.49999998835846782  2461255.50000000000000000
;                        2461256.00000000000000000  2461256.10343212960287929
;                        2457754.49999421276152134  2457754.50000578723847866
;                        2457754.50000000000000000
;   dates: 2026-08-02 2026-08-03 2026-08-03 2026-08-03 2016-12-31 2017-01-01 2017-01-01
;   julday(8, 3, 2026, 0, 0, 0) = 2461255.50000000000000000
;   uniq([1, 1, 2, 2, 2, 3]) = [1, 4, 5]
e = [838987269183999999LL, 838987269184000000LL, 839030469184000000LL, $
     839039405720686976LL, 536500867684000000LL, 536500868684000000LL, $
     536500869184000000LL]
print, 'CDF_EPOCH_TOJULDAYS:'
print, cdf_epoch_tojuldays(e), format='(D25.17)'
print, 'CDF_EPOCH, /BREAKDOWN without /TT2000, as in CDF_merge_to_daily:'
cdf_epoch, e, yr, mo, dy, hr, /BREAKDOWN
for i = 0, n_elements(e)-1 do print, yr[i], mo[i], dy[i], hr[i], format='(I04,"-",I02,"-",I02," ",I02)'
print, 'CDF_EPOCH, /BREAKDOWN, /TT2000:'
cdf_epoch, e, yr, mo, dy, hr, mn, sc, milli, micro, nano, /BREAKDOWN, /TT2000
for i = 0, n_elements(e)-1 do print, yr[i], mo[i], dy[i], hr[i], mn[i], sc[i], milli[i], $
   format='(I04,"-",I02,"-",I02,"T",I02,":",I02,":",I02,".",I03)'
print, 'julday(8, 3, 2026, 0, 0, 0):'
print, julday(8, 3, 2026, 0, 0, 0), format='(D25.17)'
print, 'uniq([1, 1, 2, 2, 2, 3]):'
print, uniq([1, 1, 2, 2, 2, 3])

🤖 Generated with Claude Code

namurphy and others added 2 commits November 17, 2026 14:12
Move console_level from pfc_decommutator into pyfaradaycup._logging,
and the CDF metadata comparison helper from the decommutator tests into
a fixture in tests/conftest.py, so that the upcoming pfc_l05_to_l1 tool
and its tests can use them too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the IDL code that merges SPC L0.5 CDF files into daily L1 CDF
files (spp_swp_spc_l05_to_l1_dates.pro, spp_swp_spc_l05_to_l1_date.pro,
spp_swp_spc_which_apid.pro, and CDF_merge_to_daily from
idllib/CDF_to_survey.pro) into the new pyfaradaycup.l05_to_l1
subpackage, keeping the IDL names and structure.

This is a prototype. It mimics IDL, including removing duplicate
records by comparing float64 Julian days, and finds the L0.5 files for
a date by searching the L0.5 directory instead of the IDL sdc_state
registry. Only the APID351 L1 file from IDL can be reproduced from the
L0.5 files in _psp, so the regression tests for the other APIDs are
marked xfail until the rest of that day's L0.5 files are added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.06557% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.19%. Comparing base (5675a6a) to head (60f48a1).

Files with missing lines Patch % Lines
src/pyfaradaycup/l05_to_l1/cdf_to_survey.py 91.26% 5 Missing and 4 partials ⚠️
...c/pyfaradaycup/l05_to_l1/_spp_swp_spc_l05_to_l1.py 97.14% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #136      +/-   ##
==========================================
+ Coverage   70.99%   76.19%   +5.20%     
==========================================
  Files           8       15       +7     
  Lines        1124     1420     +296     
  Branches      163      200      +37     
==========================================
+ Hits          798     1082     +284     
- Misses        291      298       +7     
- Partials       35       40       +5     
Files with missing lines Coverage Δ
src/pyfaradaycup/__init__.py 100.00% <100.00%> (ø)
src/pyfaradaycup/decommutator/_l0_to_l05.py 72.93% <100.00%> (-0.25%) ⬇️
src/pyfaradaycup/l05_to_l1/__init__.py 100.00% <100.00%> (ø)
src/pyfaradaycup/l05_to_l1/main.py 100.00% <100.00%> (ø)
src/pyfaradaycup/utils/__init__.py 100.00% <100.00%> (ø)
src/pyfaradaycup/utils/_cdf.py 100.00% <100.00%> (ø)
src/pyfaradaycup/utils/idl.py 100.00% <100.00%> (ø)
...c/pyfaradaycup/l05_to_l1/_spp_swp_spc_l05_to_l1.py 97.14% <97.14%> (ø)
src/pyfaradaycup/l05_to_l1/cdf_to_survey.py 91.26% <91.26%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@namurphy

namurphy commented Nov 17, 2026 •

Copy link
Copy Markdown
Member Author

It's pretty awesome that Claude can do this in 14 minutes and 9 seconds, but doing code review on this is going to be quite terrible. My reaction ∈ {🫠, 😭, 🥲, 😩, 😳, 😐}

@namurphy

Copy link
Copy Markdown
Member Author

Stuff I need to do:

  • Transfer over a complete progression of data files from L0 to L3 containing exactly the files that go in and go out. (Maybe data for the start of a month?)
  • Create a git-lfs repository to contain more test data? That would reduce the limits on testing. A possibility would be to contain quick test data in the repository for CI and comprehensive test data in a separate repository.
  • Extract smaller portions of this PR into standalone PRs to facilitate code review.
  • Identify intermediate places where it would be beneficial to do comparisons with IDL functionality.
  • Better understand the issue of duplicates and de-duplication, as described above.
  • Create issues for how to improve code from this PR.

namurphy and others added 3 commits November 18, 2026 14:51
These are the L0.5 files from the raw files 0523462904, 0523462907,
and 0523462913 that have records on 2026-08-03. With them, the L1
files from IDL for APIDs 353, 354, 35E, and 35F can be reproduced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All five L1 files from IDL for 2026-08-03 can now be reproduced from the
L0.5 files in _psp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	_psp/data/sci/sweap/spc/L05/2026/08/APID353/0523462907_3_EA_APID353_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID354/0523462907_3_EA_APID354_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID354/0523462913_7_EA_APID354_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID35E/0523462907_3_EA_APID35E_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID35E/0523462913_7_EA_APID35E_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462904_2_EA_APID35F_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462907_3_EA_APID35F_L1.cdf
#	_psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462913_7_EA_APID35F_L1.cdf
Comment thread docs/source/python.rst Outdated
Converting L0.5 files into L1 files from Python
===============================================

The :ref:`pfc_l05_to_l1 <cli>` command line tool calls

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming is still up in the air; see #132.

Comment on lines +100 to +120
def _parse_date(date: datetime.date | str) -> datetime.date:
"""
Return a date given as a `datetime.date` or an ISO 8601 string.

Like ``TIMESTAMPTOVALUES`` in IDL, only the date is used from a
string like ``"2026-08-03T12:00:00Z"``.

Raises
------
ValueError
If ``date`` is a string that does not start with a date in the
format ``YYYY-MM-DD``.
"""
if isinstance(date, datetime.datetime):
return date.date()
if isinstance(date, datetime.date):
return date
if len(date) < _ISO_DATE_LENGTH:
msg = f"Invalid date {date!r}: it must be in ISO 8601 format, like 2026-08-03"
raise ValueError(msg)
return datetime.date.fromisoformat(date[:10])

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of this PR is to do, for the most part, a direct translation from IDL → Python. I suspect a lot of the datetime logic could be handled by datetime rather than with home rolled solutions, but that is for the future.

Comment on lines +123 to +126
def _months_around(date: datetime.date) -> list[tuple[int, int]]:
"""Return the ``(year, month)`` before, of, and after ``date``."""
index = date.year * 12 + date.month - 1
return [(i // 12, i % 12 + 1) for i in (index - 1, index, index + 1)]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bunch of this stuff regarding file handling should be put into separate files, most likely. Again, that's for the future.

Comment on lines +175 to +176
l05dir: str | os.PathLike[str],
l1dir: str | os.PathLike[str],

@namurphy namurphy Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not satisfied with how the current tools in pyfaradaycup are handling paths. I'm thinking that by default the files should go where they're supposed to go in the database — not the local directory. But then we'll need to have this specified somewhere, like in an environment variable...but that can be brittle and add more skills that users would need to know how to do. Then again, the CLI is for interactive use, and the database orchestrator will likely be where we set this up. This will probably be a good topic for discussion, and it may be worth designing around use cases. Sigh!

"""
Make the L1 files for one date from all L0.5 files with data on it.

For each APID in ``SPC_L1_APIDS`` (351, 353, 354, 35E, and 35F), the

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add "APID" to the glossary. In part because I keep forgetting what it stands for. 😅

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh — and perhaps define what the particular values are in the glossary entry.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we can rename this to _l05_to_l1.py. But in a future PR when we diverge more from IDL.

Comment on lines +42 to +43
SPC_L1_APIDS: tuple[str, ...] = ("351", "353", "354", "35E", "35F")
"""The APIDs that L1 files are made for, in the order they are made."""

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be defined elsewhere, since it's needed throughout the L0 → L1 steps.

Comment on lines +46 to +69
class NoL05DataError(ValueError):
"""
Raised when there are no L0.5 files with SPC data for a date.

Parameters
----------
date : datetime.date
The date that has no data.
l05dir : pathlib.Path
The L0.5 directory that was searched.
"""

def __init__(self, date: datetime.date, l05dir: pathlib.Path) -> None:
super().__init__(
f"No L0.5 files in {l05dir} have SPC data for {date.isoformat()}"
)
self.date = date
self.l05dir = l05dir

def __reduce__(
self,
) -> tuple[type[NoL05DataError], tuple[datetime.date, pathlib.Path]]:
"""Pickle with the arguments, which are not the exception's args."""
return type(self), (self.date, self.l05dir)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exception handling is likely to be one of the places where Python and IDL differ most.

Comment on lines +249 to +250
l05dir: str | os.PathLike[str],
l1dir: str | os.PathLike[str],

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paths in Linux and Windows differ, but tests are run in CI on Ubuntu, macOS, and Windows.

Comment on lines +90 to +91
The Julian day of each time. Near the present, consecutive
float64 values differ by about 40 µs.

@namurphy namurphy Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me that I ran into some issues where times differed by maybe ∼100 μs between the original L0.5 files and the newly created L0.5 files. We should return to this later.

Comment on lines +95 to +106
The documentation of ``CDF_EPOCH_TOJULDAYS`` does not say how it
handles leap seconds or exactly how it calculates the result, so
the result may differ from IDL in the last bits. The CDF library
puts times during a leap second, like 2016-12-31T23:59:60.5, in the
first second of the next day.
"""
# spacepy defines the functions in pycdf.lib at run time, which ty
# cannot follow
cdf_epoch = pycdf.lib.v_tt2000_to_epoch( # ty: ignore[missing-argument]
np.asarray(epoch, dtype=np.int64) # ty: ignore[invalid-argument-type]
)
return np.asarray(cdf_epoch, dtype=np.float64) / _MS_PER_DAY + _JD_OF_CDF_EPOCH_ZERO

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So long as this tool will not be used on data from ≤2016, leap seconds will not need to be handled.

Comment on lines +155 to +204
def _read_mycdf(
infiles: Sequence[pathlib.Path],
) -> tuple[dict[str, np.ndarray], dict[str, bool]]:
"""
Read every variable from CDF files, concatenating their records.

This replaces ``read_myCDF(vars, infiles, /all)`` from CDAWlib.

Parameters
----------
infiles : sequence of pathlib.Path
CDF files that have the same variables.

Returns
-------
data : dict of str to numpy.ndarray
The raw values of each variable, in the order of the variables
in the first file. The records of record-varying variables are
concatenated in the order of ``infiles``. Variables that are not
record-varying are taken from the first file.
record_varying : dict of str to bool
Whether each variable is record-varying.

Raises
------
ValueError
If the files do not all have the same variables.
"""
chunks: dict[str, list[np.ndarray]] = {}
record_varying: dict[str, bool] = {}
for infile in infiles:
with pycdf.CDF(str(infile)) as cdf:
names = list(cdf)
if not chunks:
chunks = {name: [] for name in names}
record_varying = {name: cdf[name].rv() for name in names}
elif set(names) != set(chunks):
msg = (
f"{infile} has variables {sorted(names)}, but "
f"{infiles[0]} has variables {sorted(chunks)}"
)
raise ValueError(msg)
for name in names:
if record_varying[name] or not chunks[name]:
chunks[name].append(np.asarray(cdf.raw_var(name)[...]))
data = {
name: np.concatenate(values) if record_varying[name] else values[0]
for name, values in chunks.items()
}
return data, record_varying

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CDF handling is similar to IDL for now, but one place where we can use spacepy.pycdf more thoroughly (or perhaps cdflib).

Comment on lines +240 to +252
def _close_cdf(cdf: pycdf.CDF) -> None:
"""
Close a CDF file.

spacepy warns when a variable that the CDF skeleton marks for
compression does not get smaller when compressed, which is harmless,
so this warning is not shown.
"""
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore", message="DID_NOT_COMPRESS", category=pycdf.CDFWarning
)
cdf.close()

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems likely better to silence this warning in the settings in pyproject.toml.

Comment on lines +286 to +294
def cdf_merge_to_daily( # ruff:ignore[PLR0913]
infiles: Sequence[str | os.PathLike[str]],
*,
skeletonfile: str | os.PathLike[str],
outname: str,
outpath: str | os.PathLike[str],
outdate: datetime.date | None = None,
outversion: int = 0,
overwrite: bool = False,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a good place to use a parameter object design pattern (i.e., pass in one compound object instead of many different individual objects. There might also be better refactorings.


The records from all of the files are combined, sorted by time, and
de-duplicated, and the records for each day are written into a new
CDF file created from ``skeletonfile``. The data values are not

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we can eliminate the need for skeleton files. The need for them seems...weird. And maintenance of the skeleton files seems very difficult too. Hm...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems much cleaner to define the skeletons in the Python code in this project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@namurphy namurphy changed the title Add prototype pfc_l05_to_l1 tool ported from IDL Add prototype l05_to_l1 tool ported from IDL Nov 19, 2026
ty 0.0.104 no longer reports these diagnostics for spacepy's pycdf.lib
functions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
namurphy and others added 10 commits November 19, 2026 11:56
# Conflicts:
#	src/pyfaradaycup/decommutator/main.py
#	tests/decommutator/test_decommutator.py
The Python equivalents of the IDL built-in functions JULDAY,
CDF_EPOCH_TOJULDAYS, and UNIQ are not specific to the L0.5 to L1 step,
so move them out of cdf_to_survey into a general module that other code
ported from IDL can use. They are public there, with the IDL names.
Their tests move into tests/utils/test_idl.py, and the tt2000 fixture
moves into tests/conftest.py so both sets of tests can use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IDL's spp_swp_spc_which_apid and spp_swp_spc_l05_to_l1_dates accept a
scalar as well as an array. Passing a single string to the Python ports
iterated over its characters, so spp_swp_spc_which_apid silently
returned a list of empty strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching for the L0.5 files with records on a date opened every CDF
file under APID*/, including those for APID 352, before selecting the
files for each APID. Selecting by APID first gives the same files, and
means that a file for another APID that cannot be read is not an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the L1 files for several dates were made, every L0.5 file in the
three months around each date was opened again to read its dates. The
dates of each file are now cached, keyed by its modification time and
size, which plays the part of the unique_dates of each raw file in the
sdc_state registry in IDL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Variables are written into the L1 file by name, as the README says, but
the check that the skeleton and the input files have the same variables
also required them to be in the same order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dates passed to spp_swp_spc_l05_to_l1_date and
spp_swp_spc_l05_to_l1_dates are documented as being in UTC, but the
date of a datetime with another time zone was its local date, so the
L1 files for the wrong day could be made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The decommutator, the L0.5 to L1 conversion, and the test fixture for
writing L0.5 files each had a copy of the code that hides spacepy's
harmless DID_NOT_COMPRESS warning when a CDF file is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same offset between Julian days and datetime.date ordinals was
defined in both cdf_to_survey and pyfaradaycup.utils.idl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sphinx could not resolve "these" as a type, which made the nitpicky
documentation build fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@namurphy

Copy link
Copy Markdown
Member Author

My goal with this PR is to get the conversion from L0.5 → L1 working in Python, while following the existing IDL structure closely when possible. This has been accomplished, and there was much rejoicing. Having this snapshot may be useful in the future because it could make it easier to ascertain when any bugs got introduced into the IDL version. It's presently useful as a way to do a code review to help plan out changes to make in subsequent PRs. There will be many. 😅

namurphy and others added 4 commits November 19, 2026 13:57
One truncated or corrupt L0.5 file, or one still being written, in the
three months around a date stopped the L1 files for every date whose
search included it from being made. Such a file is now skipped with a
warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without overwrite, the L1 files for the APIDs before the first one that
already existed were written before FileExistsError was raised, so the
date was left with a mix of new and old L1 files, and a plain retry
then failed at the first APID. Every output file is now checked before
any is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spp_swp_spc_which_apid searched the whole path for _APID, as IDL does,
so an _APID in the name of a directory above the L0.5 files gave the
wrong APID for every file, and every date failed with no L0.5 data.
Like IDL's "strpos(...) gt 0", an _APID at the start of the name is
still not counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Note that L0.5 files for other APIDs are not opened, that the dates of
each L0.5 file are cached, that variables can be in any order, and
that datetimes with time zones are converted to UTC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@namurphy
namurphy merged commit c66036e into PlasmaPy:main Nov 19, 2026
17 checks passed
@namurphy
namurphy deleted the l05-to-l1 branch November 19, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant