prefix-caching

Artifact for "Marconi: Prefix Caching for the Era of Hybrid LLMs" [MLSys '25 Outstanding Paper Award, Honorable Mention]

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.

Reproducing agentic KV-cache policy claims on real traces. 68k requests from 393 Claude Code sessions. LRU is harder to beat than the papers suggest.

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

DeepSeek 1017 EXL3 on 3-4 DGX Sparks. TP4 4.0.3 + Smart CUDA Graph Controller. 4.0.2 benchmarks: 4,264 cold prefill tok/s; C16 code 320.1 / prose 285.4 decode sum. Smart KV cache + Smart Adaptive speculation. C16/R48, 9M KV, 3.5 GiB checkpoints. TP3 1.2.8 (+58% cold prefill). Exact recipes, source and benchmarks.

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

Miniature LLM serving runtime with continuous batching, chunked prefill, KV-cache management, preemption, prefix caching, and real Llama-family execution.

Adaptive Disaggregated Inference on a Role-free Fleet!

GenPark AI Agent Skill - Radix tree prefix caching simulator for prompt templates and agent system instructions, maximizing KV-cache hit rates and cutting TTFT latency.

GenPark AI Agent Skill - Multi-tier model cascading router dynamically selecting optimal models (e.g. Small vs Medium vs Large) based on task complexity, cost budgets, and SLA constraints.

GenPark AI Agent Skill - Radix tree prefix caching simulator for prompt templates and agent system instructions, maximizing KV-cache hit rates and cutting TTFT latency.

GenPark AI Agent Skill - Speculative decoding simulation engine coordinating small draft model speculative token generation and large target model parallel rejection sampling verification.

GenPark AI Agent Skill - Speculative decoding simulation engine coordinating small draft model speculative token generation and large target model parallel rejection sampling verification.

🔥🔥 Accelerating decision inference with exact state forking

GenPark AI Agent Skill - Task-aware prompt token pruning using information density scoring and semantic stop-phrase elimination to compress long context prompts up to 60%.

GenPark AI Agent Skill - Task-aware prompt token pruning using information density scoring and semantic stop-phrase elimination to compress long context prompts up to 60%.

GenPark AI Agent Skill - Mixture-of-Agents (MoA) multi-layer aggregator synthesizing diverse candidate proposals from heterogeneous sub-agents into high-consensus outputs.

GenPark AI Agent Skill - Mixture-of-Agents (MoA) multi-layer aggregator synthesizing diverse candidate proposals from heterogeneous sub-agents into high-consensus outputs.

GenPark AI Agent Skill - Multi-tier model cascading router dynamically selecting optimal models (e.g. Small vs Medium vs Large) based on task complexity, cost budgets, and SLA constraints.