Repository navigation
Add prototype l05_to_l1 tool ported from IDL - #136
Conversation
Move console_level from pfc_decommutator into pyfaradaycup._logging, and the CDF metadata comparison helper from the decommutator tests into a fixture in tests/conftest.py, so that the upcoming pfc_l05_to_l1 tool and its tests can use them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the IDL code that merges SPC L0.5 CDF files into daily L1 CDF files (spp_swp_spc_l05_to_l1_dates.pro, spp_swp_spc_l05_to_l1_date.pro, spp_swp_spc_which_apid.pro, and CDF_merge_to_daily from idllib/CDF_to_survey.pro) into the new pyfaradaycup.l05_to_l1 subpackage, keeping the IDL names and structure. This is a prototype. It mimics IDL, including removing duplicate records by comparing float64 Julian days, and finds the L0.5 files for a date by searching the L0.5 directory instead of the IDL sdc_state registry. Only the APID351 L1 file from IDL can be reproduced from the L0.5 files in _psp, so the regression tests for the other APIDs are marked xfail until the rest of that day's L0.5 files are added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #136 +/- ##
==========================================
+ Coverage 70.99% 76.19% +5.20%
==========================================
Files 8 15 +7
Lines 1124 1420 +296
Branches 163 200 +37
==========================================
+ Hits 798 1082 +284
- Misses 291 298 +7
- Partials 35 40 +5
🚀 New features to boost your workflow:
|
|
It's pretty awesome that Claude can do this in 14 minutes and 9 seconds, but doing code review on this is going to be quite terrible. My reaction ∈ {🫠, 😭, 🥲, 😩, 😳, 😐} |
|
Stuff I need to do:
|
These are the L0.5 files from the raw files 0523462904, 0523462907, and 0523462913 that have records on 2026-08-03. With them, the L1 files from IDL for APIDs 353, 354, 35E, and 35F can be reproduced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All five L1 files from IDL for 2026-08-03 can now be reproduced from the L0.5 files in _psp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # _psp/data/sci/sweap/spc/L05/2026/08/APID353/0523462907_3_EA_APID353_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID354/0523462907_3_EA_APID354_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID354/0523462913_7_EA_APID354_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID35E/0523462907_3_EA_APID35E_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID35E/0523462913_7_EA_APID35E_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462904_2_EA_APID35F_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462907_3_EA_APID35F_L1.cdf # _psp/data/sci/sweap/spc/L05/2026/08/APID35F/0523462913_7_EA_APID35F_L1.cdf
| Converting L0.5 files into L1 files from Python | ||
| =============================================== | ||
|
|
||
| The :ref:`pfc_l05_to_l1 <cli>` command line tool calls |
| def _parse_date(date: datetime.date | str) -> datetime.date: | ||
| """ | ||
| Return a date given as a `datetime.date` or an ISO 8601 string. | ||
|
|
||
| Like ``TIMESTAMPTOVALUES`` in IDL, only the date is used from a | ||
| string like ``"2026-08-03T12:00:00Z"``. | ||
|
|
||
| Raises | ||
| ------ | ||
| ValueError | ||
| If ``date`` is a string that does not start with a date in the | ||
| format ``YYYY-MM-DD``. | ||
| """ | ||
| if isinstance(date, datetime.datetime): | ||
| return date.date() | ||
| if isinstance(date, datetime.date): | ||
| return date | ||
| if len(date) < _ISO_DATE_LENGTH: | ||
| msg = f"Invalid date {date!r}: it must be in ISO 8601 format, like 2026-08-03" | ||
| raise ValueError(msg) | ||
| return datetime.date.fromisoformat(date[:10]) |
There was a problem hiding this comment.
The purpose of this PR is to do, for the most part, a direct translation from IDL → Python. I suspect a lot of the datetime logic could be handled by datetime rather than with home rolled solutions, but that is for the future.
| def _months_around(date: datetime.date) -> list[tuple[int, int]]: | ||
| """Return the ``(year, month)`` before, of, and after ``date``.""" | ||
| index = date.year * 12 + date.month - 1 | ||
| return [(i // 12, i % 12 + 1) for i in (index - 1, index, index + 1)] |
There was a problem hiding this comment.
A bunch of this stuff regarding file handling should be put into separate files, most likely. Again, that's for the future.
| l05dir: str | os.PathLike[str], | ||
| l1dir: str | os.PathLike[str], |
There was a problem hiding this comment.
I'm not satisfied with how the current tools in pyfaradaycup are handling paths. I'm thinking that by default the files should go where they're supposed to go in the database — not the local directory. But then we'll need to have this specified somewhere, like in an environment variable...but that can be brittle and add more skills that users would need to know how to do. Then again, the CLI is for interactive use, and the database orchestrator will likely be where we set this up. This will probably be a good topic for discussion, and it may be worth designing around use cases. Sigh!
| """ | ||
| Make the L1 files for one date from all L0.5 files with data on it. | ||
|
|
||
| For each APID in ``SPC_L1_APIDS`` (351, 353, 354, 35E, and 35F), the |
There was a problem hiding this comment.
We should add "APID" to the glossary. In part because I keep forgetting what it stands for. 😅
There was a problem hiding this comment.
Oh — and perhaps define what the particular values are in the glossary entry.
There was a problem hiding this comment.
I'm thinking we can rename this to _l05_to_l1.py. But in a future PR when we diverge more from IDL.
| SPC_L1_APIDS: tuple[str, ...] = ("351", "353", "354", "35E", "35F") | ||
| """The APIDs that L1 files are made for, in the order they are made.""" |
There was a problem hiding this comment.
This should be defined elsewhere, since it's needed throughout the L0 → L1 steps.
| class NoL05DataError(ValueError): | ||
| """ | ||
| Raised when there are no L0.5 files with SPC data for a date. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| date : datetime.date | ||
| The date that has no data. | ||
| l05dir : pathlib.Path | ||
| The L0.5 directory that was searched. | ||
| """ | ||
|
|
||
| def __init__(self, date: datetime.date, l05dir: pathlib.Path) -> None: | ||
| super().__init__( | ||
| f"No L0.5 files in {l05dir} have SPC data for {date.isoformat()}" | ||
| ) | ||
| self.date = date | ||
| self.l05dir = l05dir | ||
|
|
||
| def __reduce__( | ||
| self, | ||
| ) -> tuple[type[NoL05DataError], tuple[datetime.date, pathlib.Path]]: | ||
| """Pickle with the arguments, which are not the exception's args.""" | ||
| return type(self), (self.date, self.l05dir) |
There was a problem hiding this comment.
Exception handling is likely to be one of the places where Python and IDL differ most.
| l05dir: str | os.PathLike[str], | ||
| l1dir: str | os.PathLike[str], |
There was a problem hiding this comment.
Paths in Linux and Windows differ, but tests are run in CI on Ubuntu, macOS, and Windows.
| The Julian day of each time. Near the present, consecutive | ||
| float64 values differ by about 40 µs. |
There was a problem hiding this comment.
This reminds me that I ran into some issues where times differed by maybe ∼100 μs between the original L0.5 files and the newly created L0.5 files. We should return to this later.
| The documentation of ``CDF_EPOCH_TOJULDAYS`` does not say how it | ||
| handles leap seconds or exactly how it calculates the result, so | ||
| the result may differ from IDL in the last bits. The CDF library | ||
| puts times during a leap second, like 2016-12-31T23:59:60.5, in the | ||
| first second of the next day. | ||
| """ | ||
| # spacepy defines the functions in pycdf.lib at run time, which ty | ||
| # cannot follow | ||
| cdf_epoch = pycdf.lib.v_tt2000_to_epoch( # ty: ignore[missing-argument] | ||
| np.asarray(epoch, dtype=np.int64) # ty: ignore[invalid-argument-type] | ||
| ) | ||
| return np.asarray(cdf_epoch, dtype=np.float64) / _MS_PER_DAY + _JD_OF_CDF_EPOCH_ZERO |
There was a problem hiding this comment.
So long as this tool will not be used on data from ≤2016, leap seconds will not need to be handled.
| def _read_mycdf( | ||
| infiles: Sequence[pathlib.Path], | ||
| ) -> tuple[dict[str, np.ndarray], dict[str, bool]]: | ||
| """ | ||
| Read every variable from CDF files, concatenating their records. | ||
|
|
||
| This replaces ``read_myCDF(vars, infiles, /all)`` from CDAWlib. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| infiles : sequence of pathlib.Path | ||
| CDF files that have the same variables. | ||
|
|
||
| Returns | ||
| ------- | ||
| data : dict of str to numpy.ndarray | ||
| The raw values of each variable, in the order of the variables | ||
| in the first file. The records of record-varying variables are | ||
| concatenated in the order of ``infiles``. Variables that are not | ||
| record-varying are taken from the first file. | ||
| record_varying : dict of str to bool | ||
| Whether each variable is record-varying. | ||
|
|
||
| Raises | ||
| ------ | ||
| ValueError | ||
| If the files do not all have the same variables. | ||
| """ | ||
| chunks: dict[str, list[np.ndarray]] = {} | ||
| record_varying: dict[str, bool] = {} | ||
| for infile in infiles: | ||
| with pycdf.CDF(str(infile)) as cdf: | ||
| names = list(cdf) | ||
| if not chunks: | ||
| chunks = {name: [] for name in names} | ||
| record_varying = {name: cdf[name].rv() for name in names} | ||
| elif set(names) != set(chunks): | ||
| msg = ( | ||
| f"{infile} has variables {sorted(names)}, but " | ||
| f"{infiles[0]} has variables {sorted(chunks)}" | ||
| ) | ||
| raise ValueError(msg) | ||
| for name in names: | ||
| if record_varying[name] or not chunks[name]: | ||
| chunks[name].append(np.asarray(cdf.raw_var(name)[...])) | ||
| data = { | ||
| name: np.concatenate(values) if record_varying[name] else values[0] | ||
| for name, values in chunks.items() | ||
| } | ||
| return data, record_varying |
There was a problem hiding this comment.
The CDF handling is similar to IDL for now, but one place where we can use spacepy.pycdf more thoroughly (or perhaps cdflib).
| def _close_cdf(cdf: pycdf.CDF) -> None: | ||
| """ | ||
| Close a CDF file. | ||
|
|
||
| spacepy warns when a variable that the CDF skeleton marks for | ||
| compression does not get smaller when compressed, which is harmless, | ||
| so this warning is not shown. | ||
| """ | ||
| with warnings.catch_warnings(): | ||
| warnings.filterwarnings( | ||
| "ignore", message="DID_NOT_COMPRESS", category=pycdf.CDFWarning | ||
| ) | ||
| cdf.close() |
There was a problem hiding this comment.
It seems likely better to silence this warning in the settings in pyproject.toml.
| def cdf_merge_to_daily( # ruff:ignore[PLR0913] | ||
| infiles: Sequence[str | os.PathLike[str]], | ||
| *, | ||
| skeletonfile: str | os.PathLike[str], | ||
| outname: str, | ||
| outpath: str | os.PathLike[str], | ||
| outdate: datetime.date | None = None, | ||
| outversion: int = 0, | ||
| overwrite: bool = False, |
There was a problem hiding this comment.
This could be a good place to use a parameter object design pattern (i.e., pass in one compound object instead of many different individual objects. There might also be better refactorings.
|
|
||
| The records from all of the files are combined, sorted by time, and | ||
| de-duplicated, and the records for each day are written into a new | ||
| CDF file created from ``skeletonfile``. The data values are not |
There was a problem hiding this comment.
I'm wondering if we can eliminate the need for skeleton files. The need for them seems...weird. And maintenance of the skeleton files seems very difficult too. Hm...
There was a problem hiding this comment.
It seems much cleaner to define the skeletons in the Python code in this project.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ty 0.0.104 no longer reports these diagnostics for spacepy's pycdf.lib functions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/pyfaradaycup/decommutator/main.py # tests/decommutator/test_decommutator.py
The Python equivalents of the IDL built-in functions JULDAY, CDF_EPOCH_TOJULDAYS, and UNIQ are not specific to the L0.5 to L1 step, so move them out of cdf_to_survey into a general module that other code ported from IDL can use. They are public there, with the IDL names. Their tests move into tests/utils/test_idl.py, and the tt2000 fixture moves into tests/conftest.py so both sets of tests can use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IDL's spp_swp_spc_which_apid and spp_swp_spc_l05_to_l1_dates accept a scalar as well as an array. Passing a single string to the Python ports iterated over its characters, so spp_swp_spc_which_apid silently returned a list of empty strings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching for the L0.5 files with records on a date opened every CDF file under APID*/, including those for APID 352, before selecting the files for each APID. Selecting by APID first gives the same files, and means that a file for another APID that cannot be read is not an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the L1 files for several dates were made, every L0.5 file in the three months around each date was opened again to read its dates. The dates of each file are now cached, keyed by its modification time and size, which plays the part of the unique_dates of each raw file in the sdc_state registry in IDL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Variables are written into the L1 file by name, as the README says, but the check that the skeleton and the input files have the same variables also required them to be in the same order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dates passed to spp_swp_spc_l05_to_l1_date and spp_swp_spc_l05_to_l1_dates are documented as being in UTC, but the date of a datetime with another time zone was its local date, so the L1 files for the wrong day could be made. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The decommutator, the L0.5 to L1 conversion, and the test fixture for writing L0.5 files each had a copy of the code that hides spacepy's harmless DID_NOT_COMPRESS warning when a CDF file is closed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same offset between Julian days and datetime.date ordinals was defined in both cdf_to_survey and pyfaradaycup.utils.idl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sphinx could not resolve "these" as a type, which made the nitpicky documentation build fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
My goal with this PR is to get the conversion from L0.5 → L1 working in Python, while following the existing IDL structure closely when possible. This has been accomplished, and there was much rejoicing. Having this snapshot may be useful in the future because it could make it easier to ascertain when any bugs got introduced into the IDL version. It's presently useful as a way to do a code review to help plan out changes to make in subsequent PRs. There will be many. 😅 |
One truncated or corrupt L0.5 file, or one still being written, in the three months around a date stopped the L1 files for every date whose search included it from being made. Such a file is now skipped with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without overwrite, the L1 files for the APIDs before the first one that already existed were written before FileExistsError was raised, so the date was left with a mix of new and old L1 files, and a plain retry then failed at the first APID. Every output file is now checked before any is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spp_swp_spc_which_apid searched the whole path for _APID, as IDL does, so an _APID in the name of a directory above the L0.5 files gave the wrong APID for every file, and every date failed with no L0.5 data. Like IDL's "strpos(...) gt 0", an _APID at the start of the name is still not counted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Note that L0.5 files for other APIDs are not opened, that the dates of each L0.5 file are cached, that variables can be in any order, and that datetimes with time zones are converted to UTC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Warning
This is a prototype. It ports the IDL L0.5 → L1 step closely enough to reproduce the five IDL L1 files for one day (2026-08-03). It has not been validated against production output beyond that day, and it should not be used in the pipeline yet. The additional testing that is needed is listed below.
Summary
This adds
l05_to_l1, aclickcommand line tool that merges SPC L0.5 CDF files into daily L1 CDF files, one per APID (351, 353, 354, 35E, 35F). It's a port of this IDL code, keeping the IDL function names and structure:pyfaradaycup.l05_to_l1)spp_swp_spc_l05_to_l1_dates,spp_swp_spc_l05_to_l1_date,spp_swp_spc_which_apidspp_swp_spc_l05_to_l1_dates.pro,spp_swp_spc_l05_to_l1_date.pro,spp_swp_spc_which_apid.procdf_to_survey.cdf_merge_to_dailyCDF_merge_to_dailyinidllib/CDF_to_survey.pro(IDL names are case-insensitive)_read_mycdf,_read_master_cdf,_write_data_to_cdfread_myCDF,read_master_cdf,write_data_to_cdfpyfaradaycup.utils.idl:cdf_epoch_tojuldays,julday,uniqCDF_EPOCH_TOJULDAYS,JULDAY,UNIQ_l05_childrensdc_stateregistry (spc_rawfileunique_dates/l05_children)For each date, the L0.5 files with records on that date are combined. Only the records on that date are kept, and they are sorted by time, de-duplicated, and written into the L1 skeleton. Data values aren't changed.
Logical_file_idandGeneration_dateare set the same way as by CDAWlib'swrite_data_to_cdf.This uses two helpers that were split out of this PR so both tools can share them:
console_levelinpyfaradaycup._logging(#146) and thecdf_metadatatest fixture intests/conftest.py(#147). The only other changes to existing code are:l05_to_l1andutilssubpackages (inpyproject.toml,pyfaradaycup/__init__.py,tests/test_imports.py, and the docs);tests/conftest.py;_close_cdfout of the decommutator intopyfaradaycup.utils._cdf.close_cdf, so that the decommutator,l05_to_l1, and the test fixtures share one copy of the code that hides spacepy's harmlessDID_NOT_COMPRESSwarning.The Python equivalents of IDL built-in functions are in a new general module,
pyfaradaycup.utils.idl, so that other code ported from IDL can use them. They're public there and keep the IDL names. Their tests are intests/utils/test_idl.py, and thett2000test fixture is intests/conftest.py.Where this mimics IDL (deliberately, for now)
src/pyfaradaycup/l05_to_l1/README.mdsays to compare the exact int64 TT2000 values instead.uniq. IDL'ssortisn't stable, so which duplicate IDL itself keeps is undefined.Where this differs from IDL
sdc_stateregistry. The L0.5 files for a date are found by searchingL05/YYYY/MM/APID*/for that month and the months on either side, reading each file'sEpoch.SPC_L1_APIDSare opened. IDL selects the files for each APID after looking them up, so the files used are the same, but an unreadable file for another APID (such as 352) is not an error._APIDin a directory name isn't mistaken for it. As with IDL'sstrpos(...) gt 0, an_APIDat the very start of the name doesn't count.unique_dates. When several dates are converted, each file is read once.Epoch". The two are the same for the current L1 skeletons.-o/--overwrite. IDL always overwrites. Without--overwrite, every APID's L1 file for a date is checked before any is written, so if one exists, none are written and the date isn't left with a mix of new and old files.datetime.datetime, which is converted to UTC before its date is taken if it has a time zone. IDL only accepts dates as strings or numbers. As in IDL,spp_swp_spc_which_apidandspp_swp_spc_l05_to_l1_datesalso accept a single file name or date instead of a list.EX_DATAERR) for a date with no L0.5 data, consistent withpfc_decommutator.Testing done
nox -s tests-3.9andnox -s tests-3.14pass (including doctests);nox -s typecheck,nox -s docs, andprekpass._psp/are identical to the IDL L1 files. The only exception isGeneration_date. The comparison covers values, dtypes, variable and global attributes, and CDF metadata.Epochvalues;datetimes with time zones;Logical_file_idandGeneration_date;Suggested additional testing before using this in production
_psp/data/sci/sweap/spc/L05/, and thexfailmarks intest_l1_matches_idlhave been removed.CDF_EPOCH_TOJULDAYShandles leap seconds or how it computes its result.CDF_merge_to_dailyalso callsCDF_EPOCH, /BREAKDOWNon TT2000 values without/TT2000, and it isn't clear that IDL detects the type. Run the script below on the IDL machine, compare it with the expected values in its comments, and turn the outputs into test constants.Epoch, if one exists.src/pyfaradaycup/data/cdf_skeletons/match the production ones. They were copied but haven't been verified.file_mode="exclusive"makes two runs for the same date in the same second fail.pfc_decommutatorhas the same limitation.IDL cross-check script
🤖 Generated with Claude Code