webdataset

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.

Unofficial reimplementation of DynaDUSt3R (Stereo4D, CVPR 2025): full training pipeline, sharded 4TB streaming data path, multi-GPU training. Weights and datasets released.

[CVPR 2021: Oral] In this work, we show that high frequency Fourier spectrum decay discrepancies are not inherent characteristics for existing CNN-based generative models.

Fast and memory-efficient webdataset shard reader - Rust core with Python bindings, streaming metadata, and a bucketed dataloader.

DALLE-tools provided useful dataset utilities to improve you workflow with WebDatasets.

High-level API for tar-based dataset

This repo is the official released code of FoPro (AAAI-2023)

A sample subset of the NIH Chest X-ray Dataset. At only 2.4% of the size of the original dataset, it allows creating an accurate classifier using the Augmented Chest X-Ray repository.

Efficiently process webdatasets

Scripts to collect data from CARLA and save them as Webdataset

High-Throughput PyTorch Sequential Data Loaders for GPU Starvation Reduction

A Rust implementation of WebDataset: stream training data from tar shards, with a drop-in Python replacement checked field by field against the reference library.

Pack dataset media into plain STORED zip shards; read them fast anywhere.

Index, group, sample, resynchronize, and repack uncompressed WebDataset TAR archives

Give AI agents (Claude Code, Cursor, MCP) eyes and ears over your video library with zero cloud dependencies.

Stream WebDataset shards straight from Backblaze B2 object storage into PyTorch training — no local staging disk. A FastAPI + Next.js sample app that packs media into .tar shards, writes them to B2, and streams them back as an IterableDataset with live throughput and a distributed worker/node shard split.

Bulk-download image-text datasets into Backblaze B2 as reproducible WebDataset tar shards with img2dataset — no local staging disk. Sample app (FastAPI + Next.js) that streams shards to S3-compatible object storage, validates download yield, and streams them back for PyTorch/JAX training.

DataComp-style image-text dataset filtering on Backblaze B2: stream WebDataset shards from object storage, score image-text alignment with CLIP (open_clip), write filtered shards + quality metrics back to B2 — no local staging, no database. Full-stack Next.js + FastAPI sample for VLM-pretraining data curation

Batch audio curation in Rust: decode, VAD-segment, quality-filter, and shard raw recordings into speech-model training data.