GH-3877: Add FINITE_VECTOR logical type support by divjotarora · Pull Request #3876 · apache/parquet-java

divjotarora · 2026-10-22T10:39:51Z

Rationale for this change

The FINITE_VECTOR proposal defines fixed-size vectors of non-null, finite elements on canonical LIST storage, so applications can express vector semantics while retaining existing Parquet storage and interoperability. This change brings parquet-java support in line with the proposal.

What changes are included in this PR?

Adds FINITE_VECTOR schema annotations and mandatory write validation, validates stored levels and elements during reads, and preserves LIST interoperability across existing model readers.

Are these changes tested?

The affected Maven suites for FINITE_VECTOR schemas, writing, reading, adapters, and filtered reads passed, and Spotless passed for all touched modules.

Closes #3877

Fokko

The biggest issue is around the validation. It feels to me that a lot of visitor that are meant to do other things, are now also validating if it is a valid FINITE_VECTOR. I left some other comments as well. Let me know what you think

Fokko · 2026-11-02T17:45:57Z

Eventually, these should go in https://github.com/apache/parquet-testing/

Yes, that's definitely the goal here

Co-authored-by: Isaac <no-reply@databricks.com>

Fokko

Thanks @divjotarora for iterating on this, I think we're getting close 👍 Now, my main concern is the performance penalty that we're having because of the validation.

Fokko · 2026-11-05T19:57:50Z

    RecordConsumer recordWriter = new MessageColumnIORecordConsumer(columns);
    if (DEBUG) recordWriter = new RecordConsumerLoggingWrapper(recordWriter);
-    return validating ? new ValidatingRecordConsumer(recordWriter, getType()) : recordWriter;
+    return validating || TypeUtil.hasFiniteVector(getType())


Based on the discussion on the dev-list, we could also drop the validation. One of the goals of the VECTOR is performance, so I think we can also delegate this to the engine.

Fokko · 2026-11-05T19:57:50Z

+    default Optional<T> visit(FiniteVectorLogicalTypeAnnotation vectorLogicalType) {
+      return empty();
+    }


I think we can do the fallback here:

Suggested change

default Optional<T> visit(FiniteVectorLogicalTypeAnnotation vectorLogicalType) {

return empty();

}

default Optional<T> visit(FiniteVectorLogicalTypeAnnotation vectorLogicalType) {

return visit((ListLogicalTypeAnnotation) vectorLogicalType);

}

Fokko · 2026-11-05T19:57:51Z

+    @Override
+    public OriginalType toOriginalType() {
+      return OriginalType.LIST;
+    }
+


Nit: this is inherited from ListLogicalTypeAnnotation, so we can drop it:

Suggested change

@Override

public OriginalType toOriginalType() {

return OriginalType.LIST;

}

Less is more 👍

Fokko · 2026-11-05T19:57:51Z

+          ? new GroupType(repetition, name, logicalTypeAnnotation, fields, id)
+          : new GroupType(repetition, name, getOriginalType(), fields, id);
+      if (logicalTypeAnnotation instanceof FiniteVectorLogicalTypeAnnotation) {
+        checkFiniteVector(group);


Thanks for moving this into the builder! There's still a public way around it: GroupType.withNewFields(...) keeps the annotation and calls the constructor directly. I tried it: a FINITE_VECTOR group with an optional float element (via withNewFields) writes fine, and then ParquetFileReader.open on that same file fails with Invalid FINITE_VECTOR schema in file metadata. So we can produce files we can't read ourselves.

Can we also call checkFiniteVector from TypeUtil.checkValidWriteSchema? ParquetFileWriter already calls that for every write, so it closes the gap without touching the GroupType constructors.

Ack, I think that's what I was missing earlier and why I'd put all of this in the GroupType constructors before. Your suggestion makes sense, I'll look into it.

Fokko · 2026-11-05T19:57:51Z

+      buildChildren(builder, iterator, root.getNum_children(), columnOrderIterator);
+      return builder.named(root.name);
+    } catch (InvalidSchemaException e) {
+      throw new ParquetDecodingException("Invalid FINITE_VECTOR schema in file metadata", e);


This wraps every InvalidSchemaException from the builders with a FINITE_VECTOR-specific message. That's true today, but as soon as another type adds builder validation the message is misleading. Could we keep the original message, or only rethrow with this message when the cause came from the vector check?

Fokko · 2026-11-05T19:57:51Z

+  private void checkVectorColumnCounts(ColumnChunkMetaData column, long rows) {
+    MessageType fileSchema = getFileMetaData().getSchema();
+    String[] path = column.getPath().toArray();
+    FiniteVectorLogicalTypeAnnotation vector = TypeUtil.getFiniteVectorType(fileSchema, path);


Small perf thing: this walks the schema (getFiniteVectorType + getColumnDescription) for every column chunk of every row group, and again in readAllPages. We could compute a Map<ColumnPath, Integer> of vector lengths once when the reader is constructed; for files without vectors that's a single isEmpty() check.

Fokko · 2026-11-05T19:57:51Z

    do {
      ColumnReader columnReader = currentState.column;
      int d = columnReader.getCurrentDefinitionLevel();
+      currentState.validateVector(d);


This validates every vector value on the record read path, which costs ~15–25% on vector-heavy reads in my benchmark. Based on the point raised on the dev-list, I'd remove this check for now. C++ does not validate this either.

Fokko · 2026-11-05T19:57:51Z

+      if (value.length() != 2) {
+        throw invalidVector("invalid FLOAT16 element length: " + value.length());
+      }
+      if ((value.get2BytesLittleEndian() & 0x7c00) == 0x7c00) {


The same FLOAT16 bit check is also in ValidatingRecordConsumer#addBinary. How about adding Float16.isFinite(short) next to the existing Float16.isNaN(short) and using that in both places?

More broadly, this class is already the state machine for "a valid vector level stream". If the writer also checks at the column level (see my comment on MessageColumnIO), both sides can share it, and the vector logic in ValidatingRecordConsumer can go.

Fokko · 2026-11-05T19:57:51Z

+writer changes need not produce byte-identical files.
+
+```bash
+JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 /home/div.arora/.m2/wrapper/dists/apache-maven-3.9.16/c51baac8/bin/mvn -Dthrift.executable=/home/div.arora/parquet-java-worktrees/vector/target/vector-tools/bin/thrift -pl parquet-hadoop -am -DskipTests package dependency:build-classpath -DincludeScope=test -Dmdep.outputFile=target/vector-test-classpath.txt -Dmdep.regenerateFile=true


This contains local paths (/home/..., a local Thrift binary) and some review history (pre-rename notes, "Round 2" generator). Could we trim it down to what the files contain and how to regenerate them with the repo's standard build? Once the files move to parquet-testing, this can go there too.

I plan to delete all of this and move only the Parquet files + a different cleaned up README to parquet-testing. This is primarily tracking for me (and the agents 😅 ) right now.

Fokko · 2026-11-05T20:49:24Z

Another thing, I think we don't need to collect that much statistics for the vector:

min/max don't add much value
Bloom filters even less
null-count does not make much sense, since the fields are required

We can also do this in a separate PR

divjotarora marked this pull request as ready for review October 22, 2026 10:39

divjotarora changed the title ~~MINOR: Add VECTOR logical type support~~ GH-3877: Add VECTOR logical type support Oct 22, 2026

divjotarora force-pushed the vector branch from f62107b to 3f89539 Compare October 22, 2026 19:19

divjotarora changed the title ~~GH-3877: Add VECTOR logical type support~~ GH-3877: Add FINITE_VECTOR logical type support Nov 2, 2026

Fokko requested changes Nov 2, 2026

View reviewed changes

divjotarora force-pushed the vector branch from 00177ab to 6aed376 Compare November 2, 2026 20:26

Fokko reviewed Nov 3, 2026

View reviewed changes

Comment thread parquet-column/src/main/java/org/apache/parquet/schema/MessageType.java Resolved

divjotarora requested a review from Fokko November 4, 2026 15:36

divjotarora and others added 15 commits November 5, 2026 13:15

Define the VECTOR schema contract

9ee42ed

Preserve metadata fallback for VECTOR elements

1c8d398

Enforce VECTOR validity during record writes

d7d8da6

Validate stored VECTOR boundaries during record reads

8a44a99

Reject contradictory VECTOR counts in selected chunks

4f95e9c

Read VECTOR files as Avro arrays

5e5daec

Map VECTOR files through existing model LIST paths

1b31654

Cover VECTOR file contracts and malformed boundaries

36e534d

Reject invalid VECTOR adapter interpretations

99ef065

Consolidate VECTOR schema and count validation

4b1e872

Keep distinct VECTOR validation regressions

5bdd8f1

Use fixed VECTOR resources for focused file coverage

0b4bba9

apacheGH-3877: Align vector support with FINITE_VECTOR proposal

5565d64

Co-authored-by: Isaac <no-reply@databricks.com>

apacheGH-3877: Address FINITE_VECTOR schema and adapter review

974f3fb

Co-authored-by: Isaac <no-reply@databricks.com>

apacheGH-3877: Validate FINITE_VECTOR schemas in the group builder

869eb6c

Co-authored-by: Isaac <no-reply@databricks.com>

divjotarora force-pushed the vector branch from e28aef9 to 869eb6c Compare November 5, 2026 16:37

Fokko reviewed Nov 5, 2026

View reviewed changes

Conversation

divjotarora commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

divjotarora commented Oct 22, 2026 •

edited

Loading

Fokko commented Nov 5, 2026 •

edited

Loading