[lxy-9602]

Purpose

Linked issue: #158

Add predicate indexes for realtime append-only and deduplicate reads. Queries can use indexed key and value fields to select candidate offsets before reading and filtering rows.

• Introduce pluggable realtime index interfaces and per-field configuration through realtime.index.<plugin>.columns.
• Provide an append-only hash index supporting equality and IN predicates, with concurrent readers sharing postings without copying index data.
• Restrict index queries to [disk snapshot committed end, pinned memory end) and combine candidates with the pinned deletion vector.
• Preserve candidate bitmaps through batch slicing and projection, skip non-matching batches, and evaluate residual predicates only on selected rows.
• Keep memory indexes independent of persisted file indexes. File indexes are built from actual output rows after deduplication, preserving correct file row positions.

Indexes remain in memory when segment data spills and are retained by pinned read views until released.

Tests

Added or extended coverage for:

• Hash postings, offset bounds, concurrent reads, plugin configuration, and conservative predicate fallback.
• Store reads across segment rotation, spill, projection, bitmap slicing, and pinned DV snapshots.
• Predicate evaluation on full, sparse, empty, and nullable selections.
• Realtime integration scenarios covering index pruning without statistics or residual filtering, file-index row remapping after deduplication, snapshot boundaries, pinned plans across updates/deletes and refresh, and residual predicate composition.

API and Format

• Add public RealtimeIndex, RealtimeIndexReader, and RealtimeIndexFactory interfaces.
• Add a required offset_range to RealtimeQueryContext; callers must supply the bounds associated with the pinned memory view and selected disk snapshot.
• No changes to persisted storage formats or protocols.

Documentation

Introduce the optional configuration:

realtime.index.hash.columns=id,payload

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

[SteNicholas]

@lxy-9602, I left comments concerning fragmented-selection performance and the documentation of the built-in hash index.

[SteNicholas]

Preserve batched predicate evaluation for fragmented selections: An alternating selection such as {0, 2, 4, ...} produces one slice per selected row, so this loop calls PredicateFilter::Test() thousands of times for a single input batch. MultiLiteralsLeafFunction::Test() rebuilds the IN value set and invokes Arrow IsIn for each call, repeatedly initializing the lookup state. The work therefore grows with the number of selected runs multiplied by the IN-list size. Since this is the common reader, existing file-index and deletion-vector reads are affected too.

An isolated C++ experiment with the locally available Arrow 19.0.1, 8,192 input rows, an alternating selection, and 1,024 IN values took about 0.035 ms for one whole-batch call versus 40 ms for the single-row calls, with identical selected results. This is a kernel-level reproduction, not a benchmark of Paimon's Arrow 17 build; it also reused the value-set array, whereas this path rebuilds it.

Could we gather fragmented candidates once, evaluate the predicate in a batch, and map the results back, or reuse the initialized predicate lookup state across slices? Please add performance coverage for a sparse selection combined with a large IN list.

[SteNicholas]

Document the built-in hash plugin and its supported operations: This asks users to substitute a registered plugin name, but the guide never identifies the built-in hash plugin introduced by this PR or describes its capabilities. Please provide a usable example such as realtime.index.hash.columns=id,payload, list equality/IN support and the accepted field types, and explicitly mention the current DECIMAL, TIMESTAMP, and nested-type restrictions.

It would also help to state that the hash index returns candidates rather than exact matches: postings are keyed only by Literal::HashCode(), so hash collisions can retain nonmatching rows. Residual filtering is needed for exact results even when batch statistics are disabled. The explanation below currently motivates filtering mainly in terms of mixed batches and statistics.