Repository navigation
[Performance] Reduce aligned memtable write overhead on wide devices - #19310
luoluoyuyu wants to merge 5 commits into
Conversation
Caideyipi
left a comment
There was a problem hiding this comment.
Static review of commit 3aa3bef9aaf0650508deb9fc76000daadbce0683 (the local PR snapshot and confirmed remote head match). Two RAM-accounting findings are attached inline. I did not rerun the Maven tests; the PR description reports 51 passing tests.
| private final List<Integer> materializedColumns = new ArrayList<>(); | ||
| // Marks which columns belong to the sparse write currently in progress, so the null-marking pass | ||
| // can skip the columns whose nulls are handled together with their values. | ||
| private long[] columnWriteEpoch; |
There was a problem hiding this comment.
[P2] Include the new write metadata in RAM accounting. getStructureRamSize() delegates to calculateContainerRamCost(), but that calculation does not include this long[] or the new materializedColumns list. A 10,000-column TVList now carries about 80 KB of epoch entries alone that calculateRamSize() does not report, so memory reservations can undercount wide lists. Please account for these structures and add a test against the expected size.
| List<Object> columnValue = | ||
| existingBlockCount == 0 | ||
| ? new ArrayList<>() | ||
| : new NullPrefixedValueList(existingBlockCount); |
There was a problem hiding this comment.
[P2] Account for the lazy prefix as allocated, not logical, slots. calculateContainerRamCost() measures every value column with listRamCostWithReferences(), which sizes an Object[] from List.size(). Here the size includes implicitNullCount even while prefix is null, so adding C columns after B existing blocks still reports O(C * B) references despite allocating only O(C) lazy wrappers. Since this estimate feeds TVList memory accounting, sparse extensions can trigger premature memory pressure and lose the intended benefit. Please account for the actual prefix/tail storage and invalidate the cached cost when the prefix is materialized.
04a1f88 to
529b7e4
Compare
Sparse aligned writes used to materialize device-wide Object[C]/BitMap[C] reorder arrays per row and then scan every column to mask the unwritten cells, so writing one measurement to a wide device cost O(C) per row. Each AlignedTVList now keeps only per-column open-block state: the last row a column wrote (short[C]), a bitset of the columns that own a value array in the open block (C/8 bytes) and a global hasNull flag. Rows are append-only, so a column's nulls are exactly the rows it skipped: the gap is masked when the next value arrives, and the block tail when the block closes or is read. Readers keep the existing representation (null array = all null, marked bitmap = null) and skip null handling entirely while hasNull is false. Also: - new columns use an implicit null prefix that is never materialized (no Object[B] allocation on the first write), - the per-row write metadata shrinks from 8C + 4*capacity to 2C + C/8, - the flush encoder materializes the open block once before reading the raw value/bitmap containers. Measured on a 3000-column device with one measurement per row: 8.8us -> 0.9us and 13KB -> 1KB allocated per row for a fixed column, 10.0us -> 1.6us for rotating columns; dense rows stay within noise. The per-list metadata of a 3000-column list drops from 40416 to 6408 bytes. Tests: TsFileProcessorTest 30/30, AlignedTVListTest 34/34, AlignedWritableMemChunkTest 7/7, AlignedTVListIteratorTest 14/14, MemTableFlushTaskTest 15/15.
529b7e4 to
ab1d3a5
Compare
When an insert plan covers every device column in device order, the sparse column mapping is pure overhead: it allocates int[columns] and Object[columns] per row only to pass an identity mapping to the TVList. Detect that case and write the row through the existing dense aligned path instead. Measured on a 3000-column device with one row writing every column: the per-row allocation drops from 67982 to 55960 bytes (the Object[columns] temporary disappears) while the sparse paths are unchanged. Tests: TsFileProcessorTest 30/30, AlignedTVListTest 34/34, AlignedWritableMemChunkTest 7/7, AlignedTVListIteratorTest 14/14, MemTableFlushTaskTest 15/15.
Add a correctness UT for the write patterns users hit in practice: dense rows, one sparse measurement per row, a column that stays null for a long run of consecutive rows and is written again later, rotating sparse columns, explicit nulls and a block boundary. The same content is verified on the flush snapshot and on both clone flavours. Throughput measurements are kept out of the repository and are run locally.
b1c70bb to
1ff0ab0
Compare
AlignedTVList caches calculateContainerRamCost(null) and invalidates it on structural changes. sort() can lazily materialize the indices container and the moved-segment bitmap, which are both charged by the container cost, but the invalidation in updateSegmentMovedMap() was only reached when indices already existed. Invalidate unconditionally so the cached size can never outlive a sort. Add a regression test that compares the cached structure size with a fresh calculation before and after sorting, and after a repeated no-op sort.
Null keeps exactly the representation it had before the sparse-write rework: a value array is allocated only for a block that actually holds a value, and a null cell is recorded in the bitmap (or implied by a null block) instead of being stored as 0/false in a default array. Add tests for both sides of that contract: - an all-null column never gets a value array (so it cannot be read back as a zero-filled block) and stays null after its first value appears in a later block; - an explicit null on a column that already has an array is bitmap-marked and is not read back as a value; - the never-written column of the mixed write-pattern test keeps null blocks in every block, including after the block is closed.
Summary
This PR reduces the write-path cost of aligned (multi-measurement) timeseries on devices with many columns. The previous code made several per-row operations scale with the device-wide column count
C, even when an insert touched one measurement.AlignedWritableMemChunknow passes a compact column-index mapping (destColumns) toAlignedTVListinstead of materializing device-wideObject[C]/BitMap[C]reorder arrays. Value copying and bitmap masking iterate only the columns in the insert plan. Columns that already hold a value array in the target block are still explicitly null-marked for the rows this write does not touch.AbstractMemTable.createAlignedMemChunkGroupIfNotExistAndGetbuilds its filtered schema list only inside thecomputeIfAbsentlambda, so existing devices no longer pay a stream/lambda allocation per insert.AlignedTVListcachescalculateContainerRamCost(null)and invalidates it on structural changes (extendColumns,expandValues,clearValue,clearBitMap,deleteColumn,releaseNonQueryColumns, clone/move, sort metadata).getStructureRamSize()is now O(1) on the common path instead of two O(C) estimates per row.NullPrefixedValueListthat represents existing blocks as implicit nulls and only materializes a block when it is first written.Example
For a device with 1000 measurements, writing one row with one measurement:
new Object[1000]andnew BitMap[1000](about 16 KB of reference arrays), then scanned all 1000 device columns twice for value copy/null marking.Object[1],BitMap[1]andint[1], and copies/masks only the written column. The firstgetStructureRamSize()after a structural change still computes the full cost, but subsequent calls in the same write return the cached value.Behavior is unchanged: unwritten measurements remain null for the new row, including when their value arrays were materialized earlier in the same block.
Validation
mvn -pl iotdb-core/datanode -am \ -Dtest=AlignedTVListTest,AlignedWritableMemChunkTest,AlignedTVListIteratorTest \ -Dsurefire.failIfNoSpecifiedTests=false testResult: 51 tests run, 0 failures, 0 errors.
New tests cover sparse column mapping with bitmaps/results across a block boundary, structure-cost cache invalidation, lazy prefixes over multiple blocks, and single-measurement writes to a 10,000-column device.