Repository navigation
Conversation
A curly-brace reference can be n/a or blank in a row (spec 3.2.9.3
rules 3 and 10), so the sidecar check now also tries each reference
removed and reports TAG_EXPRESSION_REPEATED for those combinations.
{HED} has the removal as its only substitution, so strings that
reference it are checked again. A repeat found by several combinations
is reported once.
A .tsv line after the first that is empty, or has a different number of tab-separated cells than the first line, is now TABULAR_INVALID (spec 3.2.10, Appendix B reasons e and f), reported by its file line in stage 0 so the rows are not validated. Lines end at LF or CRLF. The file still loads, records the lines in BaseInput.line_problems, and refuses assemble. Pins spec_tests/hed-tests to 0a07fe0. Closes hed-standard#1475.
A tabular file is validated only when it has a HED column or a sidecar with HED annotations (specification 3.2.10). Pins the existing behavior. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
pandas finds no columns in a file that starts with two or more empty lines, and the file was then treated as empty before its column names were set. The first line is one blank column name (specification 3.2.10), so it is now reported with the empty lines and cell counts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
VisLab
added
bug
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Reference expansion generates exponentially many redundant combinations and materializes the entire Cartesian product, making small sidecars expensive to validate and risking memory exhaustion.
4 open issues
What changed in this PR
This PR strengthens HED tabular validation by reporting blank and incomplete TSV lines before validating annotations. It also expands sidecar repeated-expression checks to include removed references.
Changes:
- Record empty and wrong-width TSV lines, report their file locations, and block assembly.
- Check sidecar substitutions with references removed and deduplicate repeated-expression reports.
- Add regression tests and document the validation changes.
| File | Description |
|---|---|
| tests/validator/test_spreadsheet_validator.py | Tests line errors, stage ordering, and file-line reporting. |
| tests/validator/test_sidecar_validator.py | Tests removed-reference substitutions and repeat deduplication. |
| tests/tools/bids/test_bids_file_group.py | Tests malformed lines with invalid sidecars and HED eligibility. |
| tests/models/test_base_input.py | Covers line scanning, loading, and assembly refusal. |
| tests/data/validator_tests/bids_events_invalid_columns.tsv | Adds a missing trailing cell to preserve fixture intent. |
| hed/validator/spreadsheet_validator.py | Reports line problems during initial validation. |
| hed/validator/sidecar_validator.py | Checks removal substitutions and deduplicates repeat issues. |
| hed/tools/bids/bids_file_group.py | Allows line errors to be reported despite sidecar errors. |
| hed/models/base_input.py | Records malformed lines and prevents their assembly. |
| hed/errors/error_types.py | Adds internal codes for line errors. |
| hed/errors/error_messages.py | Adds line-specific validation messages. |
| CHANGELOG.md | Documents TSV and sidecar validation changes. |
Review effort: Balanced
| - The removal is made by df_util.replace_ref with n/a, as in assembly. | ||
| - A repeat found by several combinations is reported once. | ||
| """ | ||
| refs = COLUMN_REF_PATTERN.findall(hed_string) |
| # None stands for the removal of the reference. | ||
| options = [[*refs_strings.get(ref, []), None] if ref != "HED" else [None] for ref in refs] | ||
| # The combinations without a removal go first, so a repeat they find is reported with their string. | ||
| combinations = sorted(itertools.product(*options), key=lambda combination: None in combination) |
Comment on lines
+38
to
+41
| lines = text.split("\n") | ||
| if lines and lines[-1] == "": | ||
| lines.pop() # The terminator of the last line, not an empty line. | ||
| lines = [line.removesuffix("\r") for line in lines] |
| # pandas warns when it drops the extra cells of a long row; that row is in line_problems. | ||
| warnings.simplefilter("ignore", pd.errors.ParserWarning) | ||
| self._dataframe = pd.read_csv( | ||
| io.StringIO(text), |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.