Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- Parquet export writes `Struct`, `List`, `FixedSizeList` and `Map` columns as Parquet groups, and an unsupported column fails with a `VortexException` naming it. ([#721](https://github.com/dfa1/vortex-java/issues/721))
- `vortex rewrite` re-encodes a file with the current writer. ([#749](https://github.com/dfa1/vortex-java/pull/749))
- `Chunk.prefetch` and `VortexHandle.prefetch` start the range requests of several columns together over HTTP, merged into few requests, and an HTTP handle keeps the segments it has read. ([#744](https://github.com/dfa1/vortex-java/issues/744))

### Fixed
- A Rust-written `vortex.varbin` column with 8- or 16-bit offsets, as Rust writes a data buffer under 64 KiB, reads its values instead of failing. ([#721](https://github.com/dfa1/vortex-java/issues/721))
- A Parquet import no longer fails on a null struct holding a required list or struct field. ([#721](https://github.com/dfa1/vortex-java/issues/721))
- An unexpected HTTP status now reports the server's own explanation and `Retry-After`, so a Hugging Face rate limit (`429`) says what to do instead of just the code. ([#743](https://github.com/dfa1/vortex-java/issues/743))
- `vortex inspect` over HTTP fetches only the end of each segment, several at a time, instead of the whole file, and shows a progress bar. ([#743](https://github.com/dfa1/vortex-java/issues/743))
- `vortex view` no longer freezes while it fetches data over HTTP: it shows a loading spinner, `q` quits at once, and it fetches only the columns on screen. ([#744](https://github.com/dfa1/vortex-java/issues/744))
Expand Down
5 changes: 3 additions & 2 deletions docs/how-to.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,8 +410,9 @@ this project does not configure, so give such a file 8 GB. Files with narrow col

## Convert Vortex to Parquet

Flat schemas only (`Bool`, `Primitive` including `F16`, `Utf8`, `Binary`, `vortex.timestamp`); a
`Struct`/`List`/`Map` top-level column throws `UnsupportedOperationException`.
Leaves of `Bool`, `Primitive` including `F16`, `Decimal`, `Utf8`, `Binary` and `vortex.timestamp`,
flat or nested in `Struct`, `List`, `FixedSizeList` and `Map` columns (exported as Parquet groups,
`LIST` and `MAP`); any other type throws `VortexException` naming the column.

**API:**

Expand Down
11 changes: 7 additions & 4 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -570,11 +570,14 @@ Final class (private constructor); `chunkSize` and `maxChunkBytes` bound the row

### `ParquetExporter` (`io.github.dfa1.vortex.parquet.ParquetExporter`)

Supports flat schemas only: `Bool`, `Primitive` (`F16` as Parquet's `FLOAT16`), `Utf8`, `Binary`, `Decimal` (Parquet `DECIMAL` on
Supports leaves of `Bool`, `Primitive` (`F16` as Parquet's `FLOAT16`), `Utf8`, `Binary`, `Decimal` (Parquet `DECIMAL` on
`INT32`, `INT64` or `FIXED_LEN_BYTE_ARRAY` by precision), and the
`vortex.timestamp` extension over MILLIS/MICROS/NANOS resolution. `Struct`/`List`/`Map` top-level
columns throw `UnsupportedOperationException` — the inverse of `ParquetImporter`'s nested
`LIST`/`STRUCT` support does not exist yet.
`vortex.timestamp` extension over MILLIS/MICROS/NANOS resolution, flat or nested in `Struct` (a Parquet group),
`List` and `FixedSizeList` (`LIST`) and `Map` (`MAP`, its key `REQUIRED`) columns, composable to any depth. Any other
type throws `VortexException` naming the column (e.g. `emb.list.element`). A `FixedSizeList` re-imports as a `List`,
since Parquet has no fixed-size list. A present struct whose every field is null exports as a null struct: the reader
folds a struct's validity into its fields, so the two read the same. A struct, list or map whose name holds a `.`
throws, since Parquet writers address those groups by dotted path.

| Method | Notes |
|------------------------------------------------------------|-----------------------------------------------------------|
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
package io.github.dfa1.vortex.integration;

import dev.hardwood.InputFile;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;
import dev.hardwood.row.PqList;
import dev.hardwood.row.PqStruct;
import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.parquet.ParquetExporter;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.Float4Vector;
import org.apache.arrow.vector.VarBinaryVector;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.complex.FixedSizeListVector;
import org.apache.arrow.vector.complex.ListVector;
import org.apache.arrow.vector.complex.StructVector;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;

import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HexFormat;
import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

/// #721: a Rust-written file with the nested column shapes of the Raincloud datasets whose Parquet
/// export failed — `mnist`'s `struct<bytes: binary, path: string>`, `cohere-wikipedia-simple-embed`'s
/// fixed-size float embedding, `coig`'s `list<struct<question, answer>>` — exports to Parquet with
/// every row, null and empty list intact. The list of structs is decoded from a `vortex.struct`
/// encoding inside the list's segment, the path a Java-written test file need not take.
class NestedParquetExportInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
private static final int ROWS = 1000;
private static final int DIM = 4;

static {
NativeLoader.loadJni();
}

@Test
void javaExportsTheNestedColumnsJniWrote(@TempDir Path tmp) throws Exception {
// Given
Path vortex = tmp.resolve("jni.vortex");
writeWithJni(vortex);
Path parquet = tmp.resolve("out.parquet");

// When
ParquetExporter.exportParquet(vortex, parquet);

// Then
List<List<Object>> result = new ArrayList<>();
try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(parquet));
RowReader rows = reader.buildRowReader().build()) {
while (rows.hasNext()) {
rows.next();
result.add(Arrays.asList(image(rows), embedding(rows), turns(rows)));
}
}
List<List<Object>> expected = new ArrayList<>();
for (int i = 0; i < ROWS; i++) {
expected.add(Arrays.asList(expectedImage(i), expectedEmbedding(i), expectedTurns(i)));
}
assertThat(result).isEqualTo(expected);
}

// ── expected values: every fifth image and seventh embedding null, a third of the paths and a
// quarter of the answers null, and row i holding i % 3 turns, so empty lists recur ──────────

private static List<Object> expectedImage(int row) {
if (row % 5 == 0) {
return null;
}
return Arrays.asList(HexFormat.of().formatHex(imageBytes(row)), row % 3 == 0 ? null : "img/" + row + ".png");
}

private static byte[] imageBytes(int row) {
return new byte[]{(byte) row, (byte) (row >> 8), 7};
}

private static List<Float> expectedEmbedding(int row) {
if (row % 7 == 0) {
return null;
}
List<Float> values = new ArrayList<>(DIM);
for (int j = 0; j < DIM; j++) {
values.add(row * 0.5f + j);
}
return values;
}

private static List<List<String>> expectedTurns(int row) {
List<List<String>> turns = new ArrayList<>();
for (int t = 0; t < row % 3; t++) {
int element = 3 * row + t;
turns.add(Arrays.asList("q" + element, element % 4 == 0 ? null : "a" + element));
}
return turns;
}

// ── read back from Parquet in the same shape ──────────────────────────────────────────────

private static List<Object> image(RowReader rows) {
if (rows.isNull("img")) {
return null;
}
PqStruct img = rows.getStruct("img");
return Arrays.asList(HexFormat.of().formatHex(img.getBinary("bytes")), img.getString("path"));
}

private static List<Float> embedding(RowReader rows) {
if (rows.isNull("emb")) {
return null;
}
return rows.getList("emb").floats();
}

private static List<List<String>> turns(RowReader rows) {
PqList conv = rows.getList("conv");
List<List<String>> turns = new ArrayList<>();
for (PqStruct turn : conv.structs()) {
turns.add(Arrays.asList(turn.getString("question"), turn.getString("answer")));
}
return turns;
}

// ── vortex-jni writer ─────────────────────────────────────────────────────────────────────

private static void writeWithJni(Path path) throws Exception {
Field img = new Field("img", FieldType.nullable(ArrowType.Struct.INSTANCE), List.of(
Field.notNullable("bytes", ArrowType.Binary.INSTANCE),
Field.nullable("path", ArrowType.Utf8.INSTANCE)));
Field emb = new Field("emb", FieldType.nullable(new ArrowType.FixedSizeList(DIM)), List.of(
Field.notNullable("item", new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE))));
Field turn = new Field("item", FieldType.notNullable(ArrowType.Struct.INSTANCE), List.of(
Field.notNullable("question", ArrowType.Utf8.INSTANCE),
Field.nullable("answer", ArrowType.Utf8.INSTANCE)));
Field conv = new Field("conv", FieldType.notNullable(ArrowType.List.INSTANCE), List.of(turn));
Schema schema = new Schema(List.of(img, emb, conv));
try (VortexWriter writer = VortexWriter.builder(SESSION, path.toAbsolutePath().toUri().toString(), schema,
ALLOCATOR).build();
VectorSchemaRoot root = VectorSchemaRoot.create(schema, ALLOCATOR)) {
fillImages((StructVector) root.getVector("img"));
fillEmbeddings((FixedSizeListVector) root.getVector("emb"));
fillTurns((ListVector) root.getVector("conv"));
root.setRowCount(ROWS);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR);
ArrowSchema arrowSchema = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, arrowSchema);
writer.writeBatch(array.memoryAddress(), arrowSchema.memoryAddress());
}
}
}

private static void fillImages(StructVector images) {
images.allocateNew();
VarBinaryVector bytes = (VarBinaryVector) images.getChild("bytes");
VarCharVector paths = (VarCharVector) images.getChild("path");
for (int i = 0; i < ROWS; i++) {
List<Object> image = expectedImage(i);
// a null struct's required field still holds a value in Arrow; only the struct is null
bytes.setSafe(i, imageBytes(i));
if (image == null) {
images.setNull(i);
paths.setNull(i);
continue;
}
images.setIndexDefined(i);
if (image.get(1) == null) {
paths.setNull(i);
} else {
paths.setSafe(i, ((String) image.get(1)).getBytes(StandardCharsets.UTF_8));
}
}
bytes.setValueCount(ROWS);
paths.setValueCount(ROWS);
images.setValueCount(ROWS);
}

private static void fillEmbeddings(FixedSizeListVector embeddings) {
embeddings.allocateNew();
Float4Vector items = (Float4Vector) embeddings.getDataVector();
for (int i = 0; i < ROWS; i++) {
List<Float> embedding = expectedEmbedding(i);
for (int j = 0; j < DIM; j++) {
items.setSafe(i * DIM + j, embedding == null ? 0f : embedding.get(j));
}
if (embedding == null) {
embeddings.setNull(i);
} else {
embeddings.setNotNull(i);
}
}
items.setValueCount(ROWS * DIM);
embeddings.setValueCount(ROWS);
}

private static void fillTurns(ListVector conversations) {
conversations.allocateNew();
StructVector turns = (StructVector) conversations.getDataVector();
VarCharVector questions = (VarCharVector) turns.getChild("question");
VarCharVector answers = (VarCharVector) turns.getChild("answer");
int element = 0;
for (int i = 0; i < ROWS; i++) {
conversations.startNewValue(i);
List<List<String>> rowTurns = expectedTurns(i);
for (List<String> rowTurn : rowTurns) {
turns.setIndexDefined(element);
questions.setSafe(element, rowTurn.get(0).getBytes(StandardCharsets.UTF_8));
if (rowTurn.get(1) == null) {
answers.setNull(element);
} else {
answers.setSafe(element, rowTurn.get(1).getBytes(StandardCharsets.UTF_8));
}
element++;
}
conversations.endValue(i, rowTurns.size());
}
questions.setValueCount(element);
answers.setValueCount(element);
turns.setValueCount(element);
conversations.setValueCount(ROWS);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
package io.github.dfa1.vortex.integration;

import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.reader.Chunk;
import io.github.dfa1.vortex.reader.ScanIterator;
import io.github.dfa1.vortex.reader.ScanOptions;
import io.github.dfa1.vortex.reader.VortexReader;
import io.github.dfa1.vortex.reader.array.VarBinArray;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.VarBinaryVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import java.nio.file.Path;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

/// Rust narrows `vortex.varbin` offsets to the smallest integer type holding the data buffer's
/// length: u8 under 256 bytes, u16 under 64 KiB. Java read every offset type but i32/u32 as 64-bit,
/// so a small Rust-written binary column decoded as garbage offsets and failed (#721).
class VarBinNarrowOffsetsInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
private static final ColumnName B = ColumnName.of("b");

static {
NativeLoader.loadJni();
}

/// 3-byte values: 50 rows fill a 150-byte buffer (u8 offsets), 1000 a 3000-byte one (u16).
@ParameterizedTest(name = "{0} rows")
@ValueSource(ints = {50, 1000})
void javaReadsTheNarrowOffsetsJniWrote(int rows, @TempDir Path tmp) throws Exception {
// Given
Path file = tmp.resolve("jni.vortex");
List<String> expected = new ArrayList<>();
for (int i = 0; i < rows; i++) {
expected.add(HexFormat.of().formatHex(value(i)));
}
writeWithJni(file, rows);

// When
List<String> result = new ArrayList<>();
try (VortexReader reader = VortexReader.open(file);
ScanIterator scan = reader.scan(ScanOptions.all())) {
while (scan.hasNext()) {
try (Chunk chunk = scan.next()) {
VarBinArray values = chunk.column(B);
for (int i = 0; i < chunk.rowCount(); i++) {
result.add(HexFormat.of().formatHex(values.getBytes(i)));
}
}
}
}

// Then — the file must really hold a varbin, or the test proves nothing
assertThat(String.join("\n", EncodingTrees.of(file))).contains("vortex.varbin");
assertThat(result).isEqualTo(expected);
}

private static byte[] value(int row) {
return new byte[]{(byte) row, (byte) (row >> 8), 7};
}

private static void writeWithJni(Path path, int rows) throws Exception {
Schema schema = new Schema(List.of(Field.notNullable("b", ArrowType.Binary.INSTANCE)));
try (VortexWriter writer = VortexWriter.builder(SESSION, path.toAbsolutePath().toUri().toString(), schema,
ALLOCATOR).build();
VectorSchemaRoot root = VectorSchemaRoot.create(schema, ALLOCATOR)) {
VarBinaryVector vector = (VarBinaryVector) root.getVector("b");
vector.allocateNew();
for (int i = 0; i < rows; i++) {
vector.setSafe(i, value(i));
}
vector.setValueCount(rows);
root.setRowCount(rows);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR);
ArrowSchema arrowSchema = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, arrowSchema);
writer.writeBatch(array.memoryAddress(), arrowSchema.memoryAddress());
}
}
}
}
Loading
Loading