Skip to content

[Pipe] Reduce DataRegion write-lock duration for historical source startup - #19353

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/pipe-historical-start-lock-20261121
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/pipe-historical-start-lock-20261121

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

Shorten DataRegion write-lock hold time during historical Pipe source startup while preserving startup lifecycle and history semantics.

  • Flush and copy consistent TsFile and deletion snapshots while holding the DataRegion write lock.
  • Check and pin resources outside that lock; refresh only a replaced partition once and release TsFile and modification-file pins independently on failure.
  • Keep realtime startup before historical startup and keep historical start() synchronous.

Example

Before: DataRegion 43 has 100,138 out-of-order files. Starting six Pipe historical sources scans and pins the history while holding the write lock, so concurrent writes wait about 18 seconds.

After: the lock covers the required flush and consistent snapshots; per-resource checks and pinning proceed after release, reducing write-lock occupancy while preserving historical filtering, ordering, and deletion handling.

Testing

  • mvn spotless:apply -P with-integration-tests -pl iotdb-core/datanode passed; zero files changed.
  • mvn clean test -pl iotdb-core/datanode -am -Dtest=PipeHistoricalDataRegionTsFileAndDeletionSourceTest -Dsurefire.failIfNoSpecifiedTests=false passed; 31 tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant