Skip to content

test: improve coverage collection and cover storage edge cases - #505

Merged
lxy-9602 merged 3 commits into
apache:mainfrom
lucasfang:codex/test-reader-commit-edge-cases
Nov 4, 2026
Merged

lxy-9602 merged 3 commits into
apache:mainfrom
lucasfang:codex/test-reader-commit-edge-cases

Conversation

@lucasfang

@lucasfang lucasfang commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Closes #506.

Add 25 focused regression tests and extend existing tests for storage edge cases identified by the coverage report:

  • Cover local parent paths, prefetch reader forwarding/metrics/batch lifecycle, realtime commit replay, snapshot statistics across schema changes, and conflict checks for snapshot bounds, row ranges, and write columns.
  • Synchronize concurrent credential callers so both observe an expired token, then verify that only one refresh occurs and both receive the refreshed credentials.
  • Cover snapshot time retention, snapshots and index manifests disappearing during expiration, deleted and out-of-order tags, unreadable tagged manifests preserving data, retained statistics, and empty-directory cleanup.
  • Cover global system-table discovery and schema loading, catalog options, schema discovery without schema-0, and ignored drops of incomplete table directories.
  • Exercise cleaner option replacement with a caller-provided memory pool and executor, string-predicate dispatch through actual bitmap-index matches, and Parquet sequential fallback for missing or malformed offset indexes while preserving selected rows and the streaming schema.

Improve coverage collection with atomic C/C++ gcov counters, isolate coverage flags from bundled dependencies, and enable Jindo, OSS, S3, Lumina, Lucene, full-text, and REST alongside the existing formats.

Tests

The following focused suites passed on the final binaries:

Command Result
./build/debug/paimon-fs-test --gtest_filter='LocalFileTest.*' 8 passed
./build/debug/paimon-common-test --gtest_filter='*PrefetchFileBatchReader*' 59 passed, 2 existing cases skipped because the read-ahead cache is disabled
./build/debug/paimon-core-test --gtest_filter='FileStoreCommitImplTest.*:ConflictDetectionTest.*' 145 passed
./build/debug/paimon-rest-test --gtest_filter='RestCredentialProviderTest.*:RestTokenFileSystemTest.*' 30 passed
./build/debug/paimon-core-test --gtest_filter='ExpireSnapshotsTest.*:FileSystemCatalogTest.*:OrphanFilesCleanerTest.*' 61 passed
./build/debug/paimon-common-test --gtest_filter='BitmapGlobalIndexTest.*' 7 passed
./build/debug/paimon-parquet-format-test --gtest_filter='PageFilteredRowGroupReaderTest.*' 50 passed

The new credential concurrency test also passed 100 repetitions. Changed-file pre-commit checks and git diff --check passed. The coverage script passed bash -n; isolated compiler/CMake probes verified atomic counters and third-party flag isolation, and a stubbed coverage-script smoke test verified its component options and command sequence.

Rerunning the four newly covered suites with an isolated GCOV_PREFIX verified 63 of the 64 requested business lines: REST 1/1, expiration 19/20, catalog 18/18, cleaner 9/9, Parquet 8/8, and predicate dispatch 8/8. The remaining expiration line 355 is a defensive continue for a missing hierarchy key; the producer always inserts contiguous keys starting at zero, so normal inputs cannot reach it. The full coverage HTML report was not regenerated; these are targeted execution counts, not a new overall coverage percentage.

The normal CMake build remains blocked by Vortex selecting stable Rust 1.67.1, below its dependency requirements. The changed C++ test objects were freshly compiled and the REST, core, common, and Parquet Debug test binaries were relinked using Ninja's generated commands with existing dependency artifacts. This does not claim a successful full dependency rebuild.

API and Format

No public API, storage format, or protocol changes.

Documentation

No new user-facing feature documentation is required. The coverage script contains the expanded component configuration.

Generative AI tooling

Generated-by: OpenAI Codex 0.166.0

@lucasfang lucasfang changed the title test: cover reader and commit edge cases test: improve coverage collection and cover reader and commit edge cases Nov 4, 2026
@lucasfang lucasfang changed the title test: improve coverage collection and cover reader and commit edge cases test: improve coverage collection and cover storage edge cases Nov 4, 2026

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 0ba6ec2 into apache:main Nov 4, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Improve coverage reliability and cover reader and commit edge cases

2 participants