Skip to content

Fix heap accounting for sealed TsFile resources - #19347

Open
JackieTien97 wants to merge 1 commit into
masterfrom
ty/tsfile-resource-memory
Open

JackieTien97 wants to merge 1 commit into
masterfrom
ty/tsfile-resource-memory

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Description

The DataNode uses the estimated heap of sealed TsFile resources to decide when to degrade device indexes. The previous estimate omitted the file path, per-resource locks and atomic objects, and lazily created modification-file handles. It also cached the first device-index estimate across changes. On removal, the manager recalculated the charge, so a resource whose progress index had grown could release bytes belonging to another file.

TsFileResource now measures resident metadata separately from the current time index. ArrayDeviceTimeIndex includes the device map's buckets and entries alongside the device IDs and time arrays. TsFileResourceManager records each registration's charge, applies changes as deltas, and releases the remaining balance on removal. Metadata-only changes reuse the registered index size. Compaction accounts for the device index it actually caches, and LOAD accounts for a shared file handle once.

Example

For a sealed resource with a 54-character file path on JDK 17.0.5 with compressed references, creating the exclusive mods handle previously left the estimate at 272 bytes.

Resource state Previous estimate Updated estimate Measured owned heap after the change
Degraded to a file-level index 272 B 528 B 472 B
After creating the exclusive mods handle 272 B 848 B 768 B

The measurements count each owned object once. The budget uses estimates for strings and hash-table capacity.

Verification

  • Self-reviewed the registration, degradation, and removal paths.
  • Added regression coverage for index growth, degradation before the first estimate, metadata changes after registration, and release of LOAD last values.
  • Ran 92 tests across six DataNode test classes covering resources, progress indexes, compaction, and LOAD.
  • Packaged all 50 Maven reactor modules with both the default English messages and the with-zh-locale profile on JDK 17.
  • Ran 32 concurrent degradation, metadata-update, and removal checks in an isolated JVM. Each removal returned the memory balance to its starting value.

Estimate resource metadata and device indexes separately. Track the charge for each registered file so metadata changes, degradation, and removal update the time-index budget consistently.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.53975% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.44%. Comparing base (dd450f7) to head (aeb877d).

Files with missing lines Patch % Lines
...ueryengine/plan/relational/sql/ast/LoadTsFile.java 0.00% 7 Missing ⚠️
...torageengine/dataregion/tsfile/TsFileResource.java 93.33% 7 Missing ⚠️
.../main/java/org/apache/iotdb/db/utils/MemUtils.java 62.50% 6 Missing ⚠️
...ageengine/rescon/memory/TsFileResourceManager.java 95.12% 4 Missing ⚠️
...compaction/schedule/CompactionScheduleContext.java 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             master   #19347    +/-   ##
==========================================
  Coverage     49.44%   49.44%            
  Complexity      718      718            
==========================================
  Files          5656     5656            
  Lines        418222   418390   +168     
  Branches      55296    55345    +49     
==========================================
+ Hits         206769   206886   +117     
- Misses       211453   211504    +51     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant