Optimize modelarrayio to-modelarray runtime

.gitignore

@@ -73,6 +73,14 @@ instance/
 docs/_build/
 docs/generated/
 docs/auto_examples/
+docs/sg_execution_times.rst
+
+# Downloaded/generated benchmark data and results (one set per suite)
+benchmarks/*/cohorts/
+benchmarks/*/data/
+benchmarks/*/results/
+# Rendered benchmark reports are regenerated from the pinned CSVs, never committed.
+benchmarks/**/*.pdf
 
 # PyBuilder
 target/

README.rst

@@ -71,6 +71,39 @@ Both backends expose a similar API:
 Notes and minor differences:
 
 * Chunking vs tiling: HDF5 uses chunks; TileDB uses tiles. We compute tile sizes analogous to chunk sizes to keep write/read patterns similar.
-* Compression: HDF5 uses ``gzip`` by default; TileDB defaults to ``zstd`` with shuffle for better speed/ratio. You can switch to ``gzip`` for parity.
-* Metadata: HDF5 stores ``column_names`` as a dataset attribute; TileDB stores names as JSON metadata on the array/group.
+* Compression: HDF5 defaults to ``gzip`` level 4; TileDB defaults to ``zstd`` level 5. Compressed outputs use byte shuffle; ``--compression none`` disables the filter pipeline. You can switch TileDB to ``gzip`` for parity.
+* Metadata: HDF5 stores ``column_names`` as a sibling dataset under each scalar group. TileDB stores names as JSON metadata, and CIFTI outputs also retain their dedicated column-name arrays.
 * Layout: Both backends keep dimensions in the same order and use zero-based indices.
+
+Fast conversion and input sources
+=================================
+
+``modelarrayio to-modelarray`` stages one scalar at a time into a typed,
+C-contiguous matrix, then writes that matrix in one backend submission. Use
+``--workers N`` to bound how many input sources are loaded concurrently. This
+meaning applies to local paths and ``s3://`` URLs, every modality, and both
+backends; it does not cap TileDB's internal pools or HDF5 gzip compression
+threads. Compression threads default to every CPU available to the process; set
+``MODELARRAYIO_HDF5_COMPRESSION_WORKERS`` to cap them, for example on shared
+cluster nodes.
+
+``--workers`` previously controlled concurrent TileDB scalar writes.
+``--s3-workers`` is a deprecated alias that will be removed in the next
+breaking release. A value matching ``--workers`` is accepted; a conflicting
+value is rejected.
+
+Local files and materialized DataLad/git-annex symlinks are read directly.
+ModelArrayIO never runs ``datalad get`` implicitly; a broken annex symlink
+reports an actionable error. S3 sources use one shared connection pool sized
+for the loader concurrency. Set ``MODELARRAYIO_S3_ANON=1`` for public buckets.
+
+``--dtype float32`` is the default. ``--dtype float64`` keeps source precision
+through decoding, staging, and storage, at twice the staging memory and bytes
+written.
+
+Pass ``--timing-jsonl PATH`` to append structured per-scalar and total runtime
+records. Outputs are written to a same-filesystem temporary path and validated
+before publication. An existing HDF5 file is replaced atomically. Converting
+into an existing TileDB directory merges into it: the scalars (and parcel
+arrays) written by the run replace their namesakes, every other scalar already
+in the directory is kept, and an interrupted publication is rolled back.

benchmarks/README.md

@@ -0,0 +1,32 @@
+# ModelArrayIO benchmarks
+
+Each benchmark suite lives in its own folder, named after the `modelarrayio` subcommand it
+measures (with underscores). A suite is self-contained: its scripts, committed fixtures, pinned
+results (`baselines/`), its own `README.md`, and the gitignored inputs and outputs it produces.
+Rendered reports (PDFs) are not committed; they are regenerated from the pinned CSVs.
+
+| Suite | Measures | Guide |
+| --- | --- | --- |
+| [`to_modelarray/`](to_modelarray/) | `modelarrayio to-modelarray` runtime and output correctness across NIfTI, CIFTI and MIF on HDF5 and TileDB, including the original-vs-optimized comparison and the CUBIC (HPC) validation ladder | [`to_modelarray/README.md`](to_modelarray/README.md) |
+
+## Suite layout
+
+Each suite keeps the same three workspace folders, ignored by Git through the
+`benchmarks/*/{data,cohorts,results}/` patterns in `.gitignore`:
+
+- `data/`: downloaded or synthesized input files.
+- `cohorts/`: generated cohort CSVs. Locally materialized cohorts contain absolute paths into
+  `data/`, so regenerate them (or rewrite the path prefix) after moving the checkout.
+- `results/<run-id>/`: per-run outputs, logs, timings and `run_meta` files. Set
+  `MODELARRAYIO_BENCHMARK_RESULTS_DIR` to write them elsewhere (for example cluster scratch).
+
+Run every command from the repository root.
+
+## Adding a suite
+
+Create `benchmarks/<subcommand>/` with its own `README.md`, `fixtures/` and `baselines/`, and
+add a row to the table above. When a second suite needs a tool or input that
+currently lives in `to_modelarray/` (the shared modules `harness_common.py` and
+`output_compare.py`, the cohort generators, `source_identity.py`, the old-commit launcher in
+`baselines/`, or a downloaded cohort), move it to `benchmarks/common/` and update both suites;
+until then, keep each tool and its data with the suite that uses it.