TinyServe

TinyServe is an educational C++/CUDA LLM inference runtime for studying transformer decoding, KV caching, quantization, and GPU bottlenecks. It prioritizes correctness, measurement, and reproducibility over production-level speed.

Status: Minimum Strong Version Complete. The repository provides a fixture-validated scalar float32 transformer path, independent reference checks, contiguous KV-cache decoding, weight-only INT8 storage, a minimal localhost server, and one reproducible benchmark entry point. The executable generates deterministic synthetic token IDs; those bytes have no language-quality meaning. Real-model validation and CUDA kernels remain explicitly deferred.

Architecture: text/raw IDs -> tokenizer boundary -> embeddings -> [RMSNorm -> RoPE/GQA attention -> residual -> RMSNorm -> SwiGLU -> residual] -> final RMSNorm -> logits -> greedy/sampling. Cached decode retains each layer's rotated keys and values; the HTTP target serializes requests through one model queue.

Build and test

Requirements:

• CMake 3.16 or newer
• A C++17 compiler
• Python 3 for fixture generation and converter tests (not required by the runtime when configured with -DBUILD_TESTING=OFF)
• Optional CUDA work requires an existing CUDA compiler/toolkit and a separately validated GPU; nothing is downloaded automatically

From the repository root:

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Run the executable without arguments to print its fixture-validation status and usage. For a typical single-configuration generator:

./build/tinyserve

For Visual Studio generators on Windows, the default configuration is commonly under build/Debug/:

.\build\Debug\tinyserve.exe

After a Release build, reproduce every locally supported benchmark in a timestamped directory:

.\bench\reproduce.ps1 -BuildDir build

Use -Publish only after reviewing the new run; it explicitly replaces canonical result files and regenerates the report/README tables. Linux/macOS users can run ./bench/reproduce.sh --build-dir build. Commands, overwrite safeguards, fixed repeat counts, and artifact validation are documented in the benchmark harness.

To omit benchmarks from a runtime-only build:

cmake -S . -B build -DTINYSERVE_BUILD_BENCHMARKS=OFF

Repository layout

• runtime/: the C++ runtime and CLI
• tools/: offline conversion and reference-generation tools
• tests/: CTest-backed tests
• bench/: reproducible benchmarks and generated results
• docs/: scope, design notes, and measurement protocol
• examples/: future small usage examples

PyTorch and Hugging Face tooling may be used offline for weight conversion and reference generation, but not as TinyServe's inference backend. See scope and the benchmark protocol for the current contract and limitations.

The final generated summary is the benchmark report, with an independent completion audit. The implementation notes walk through one token, the KV cache tradeoff, and the deferred CUDA analysis.

Model boundary

Wave 1 adds a strict line-based model configuration, a versioned binary tensor container, a standard-library-only deterministic fixture converter, a validating C++ loader, and a temporary byte tokenizer. The format and conversion command are documented in weights; the deliberate tokenizer limitation is documented in tokenizer.

These components are validated against a synthetic fixture only. No real model has been converted or loaded, and the temporary tokenizer is not compatible with Qwen or Llama vocabularies.

CPU math core

Wave 2 adds contiguous float32 tensors, RMSNorm, RoPE, stable causal attention with grouped-query head mapping, SwiGLU, residuals, and reusable single-decoder-block composition. Shape and numerical conventions are documented in CPU math. These scalar implementations prioritize clarity and correctness; no speed claim is made.

Fixture generation CLI

The test-enabled build creates build/fixtures/dummy.tserve. Run greedy fixture generation with:

./build/tinyserve generate \
  --model build/fixtures/dummy.tserve \
  --prompt "Hello" \
  --max-tokens 4

On a Visual Studio multi-configuration build, use the executable under the selected configuration directory, such as .\build\Release\tinyserve.exe, while the generated fixture remains under build\fixtures\.

The CLI also accepts --tokens 65,66, and positive-temperature sampling supports --top-k and --seed. Every run prints validation_status=synthetic_fixture_unverified. See CPU generation for semantics and limitations.

Correctness harness

The test suite compares C++ RMSNorm, RoPE, causal masking, attention, layer outputs, final hidden states, and full last-token logits with a committed reference generated by an independent standard-library Python oracle. Run the Wave 4 checks with:

ctest --test-dir build --output-on-failure -R reference

The fixture process, fixed numerical tolerances, divergence workflow, optional local Hugging Face dump command, and mandatory approval gate before model downloads are documented in correctness. The offline fixture checks pass without PyTorch, Transformers, network access, or model downloads. The real-model path has not been run and no real-model correctness is claimed.

KV cache benchmark

Use --decode-mode no-cache (the default) or --decode-mode kv-cache with the generation CLI. The cache is a contiguous single-sequence float32 allocation, and cached greedy generation is tested against the preserved no-cache path.

Measured fixture result

Mode	Median TTFT (ms)	Median decode (ms)	Decode tok/s	Persistent KV bytes
no-cache	0.170179	2.492310	2808.639375	0
kv-cache	0.183106	0.341675	20487.305188	192

For this run, cached decode throughput was 7.294x the no-cache fixture throughput. This is a tiny synthetic fixture result, not real-model performance. Full metadata and raw samples are in KV cache results.

INT8 weight storage

Precision	File bytes	Median generation (ms)	Tokens/s	Max logit error	RMSE	Greedy match
float32	9802	0.534743	14960.457640	0.000000	0.000000	true
int8-storage-float32-runtime	5999	0.525867	15212.972101	0.011267	0.005248	true

Fixture file reduction: 38.80%. The runtime dequantizes weights at load time; see quantization results.

Minimal local HTTP server

The tinyserve_server target implements only GET /health and POST /generate from the canonical API. It binds 127.0.0.1, uses a serialized single-model request queue, and is not a production server.

./build/tinyserve_server --model build/fixtures/dummy.tserve --port 8000
curl http://127.0.0.1:8000/health
curl -X POST http://127.0.0.1:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"A","max_new_tokens":2,"temperature":0,"top_k":1,"top_p":1.0}'

Requests	Concurrency	p50 ms	p95 ms	Requests/s	Aggregate tokens/s
24	4	2.009000	15.857400	719.905692	1439.811385

Fixture-only localhost result; see server results.

Optional CUDA gate

Wave 6 adds -DTINYSERVE_ENABLE_CUDA=ON as an opt-in probe. It skips cleanly and preserves the CPU build when a CUDA compiler or toolkit is unavailable. On this machine, nvcc, nvidia-smi, CUDA_PATH, and the standard toolkit directory were absent, so RMSNorm, RoPE, GEMV/cuBLAS, and runtime integration remain explicitly deferred rather than untested. See CUDA notes for commands, architecture flags, the availability target, and the exact implementation status.

Status

Wave 8 integration is complete. Hugging Face Transformers, llama.cpp, and vLLM comparisons are not run because no equivalent approved real model/tokenizer or validated CUDA environment exists; the fair, approval-gated setup is specified in the external-engine protocol. No real-model correctness or quality, process peak-memory, activation quantization, INT4, CUDA result, paged cache, continuous batching, streaming, or production-serving capability is claimed.

Future work is deliberately ordered: approve and validate a real checkpoint/tokenizer, run equivalent external baselines, optimize measured CPU bottlenecks, then resume CUDA correctness and microbenchmarks on validated hardware. Detailed completion and deferrals are tracked in tinyserve_implementation_progress.md.

License

MIT. See LICENSE.