Skip to content
InferMatrixLabPublic

About

No description, website, or topics provided.

Resources

Contributing

Stars

93 stars

Watchers

0 watching

Forks

Repository files navigation

InferRSI helps coding agents and engineers profile inference workloads, test optimizations against a baseline, and verify output quality before adopting a change. Bring an engine checkout, a workload, and your quality checks.

🚀 Quickstart

🖥️ Use the CLI. Install from source and list the packaged benchmarks on a development machine with Python 3.10+.

git clone https://github.com/InferMatrixLab/InferRSI.git
cd InferRSI
python -m venv .venv
source .venv/bin/activate
python -m pip install .
inferrsi benchmarks list

From a target vLLM-Omni checkout, run the CPU readiness probe.

inferrsi doctor --repo . --repo-type vllm_omni --backend cpu

Setup and task discovery run without starting a benchmark. For a target host, select its backend (cpu, cuda, rocm, ascend, or mlx). Install guide · Release wheels and checksums

🤖 Use a coding agent. Install the plugin and ask it to check the target checkout, workload, and quality criteria before starting an experiment. Codex · DeepSeek Harness · WorkSwarm

🧪 Run your first experiment. Try a packaged benchmark · Bring your own workload

🔁 How It Works

Outer loop. To improve the engine, the agent profiles the workload and proposes a change. InferRSI checks quality and measures the candidate against the baseline under the same frozen protocol. Round evidence guides the next change. A fresh process retests the best candidate at the end of the campaign.

Inner loop. To improve the optimizer, compare paired campaigns with knowledge on and off to measure whether access to the library helps the agent. Reviewed PRs add findings to the library for agents to consult in later campaigns. Knowledge comparison

  • Knowledge supplies source-linked skills, prior measurements, and implementation notes for the agent to consult.
  • Harness freezes the experiment inputs and gates, runs the rounds, and records the verdict and evidence.
  • Benchmark supplies packaged tasks, measurements, and acceptance rules. Project campaigns supply their own workload and checks.

Apps assembles these capabilities into the CLI, client integrations, and result views. Module responsibilities · Workflow diagram · Architecture decision #362 · Apps migration #371

Review a run in the dashboard. Compare baseline and candidate profiles and trace each experiment back to its evidence. From a recorded run directory, open it with inferrsi dashboard --input ., which writes dashboard.html beside the run. Dashboard guide · Download the interactive example

InferRSI dashboard showing baseline and candidate profiling captures and links to experiment and knowledge traces
Dashboard demo with synthetic data.

📊 Selected Results

Model bring-up and measured optimizations across text, image, video, and speech. Performance tables highlight metrics that improved by at least 20% against the listed baseline. The evidence column gives the scope and validation status. 🔥 marks the largest gain in its table, ✅ a row whose quality checks passed, and ⚠️ a failed or unqualified check, or an inconclusive campaign.

🧩 Model onboarding

Model Engine and hardware Milestone Evidence
DeepSeek V4.1 Flash vLLM-Ascend · 8× Ascend 910C W8A8 service on the 8K-input / 1K-output workload. Six independent runs gave 39.28 / 44.07 / 56.86 ms/token at C1 / C4 / C16; the 25 ms/token target remains unmet. Bring-up and baseline
Qwen3.8-27B Standalone nano-vLLM-derived engine · 1× H20 FP8 service passed scoped HTTP/SSE lifecycle checks; quality and performance qualification incomplete. Service bring-up · Performance campaign
LFM3-Omni-2B vLLM-Omni · 1× Ascend 910B1 ✅ Two real-weight serving tests passed; a functional eager adapter served as the offline audio baseline. Serving checks and A/B benchmark

💬 Text serving

Model Measured change and baseline Metric and hardware Validation and evidence
DeepSeek V4.1 Flash +54.45% / +36.12% / +20.80%
Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card
Per-card throughput, prefill / decode / end-to-end · Ascend A3, W4A8, 1P1D Throughput and latency rechecked at each scenario's latency limit; decode rechecked once. Long-output accuracy not verified. Workloads and evidence

🎨 Image and video generation

Model Measured change and baseline Metric and hardware Validation and evidence
MiniMax-H3 Ref2VA −73.45%
Baseline: Dense inference · 407.27 s
5 s video, HTTP latency · 8× Ascend 910B1 3.77× with lossy Cache-DiT. ✅ 14 / 14 quality comparisons passed. Campaign · Benchmark
MiniMax-H3 Ref2VA −76.3%
Baseline: In-run dense control · 701.95 s
5 s video, 50-step denoising time · 4× Ascend 950PR 🔥 4.22× in five retained A/B slots. ⚠️ Historical; performance and quality inconclusive. Archive record · Recipe benchmark
MiniMax-H3-Image −60.16%
Baseline: Original layer offload · 36.62 s
Warm HTTP latency · 2× RTX 3090 One paired comparison; ✅ 9 pixel-identical outputs. Upstream results
MiniMax-H3-Image −35.9% / −36.1%
Baseline: Same-layout SP4 control · 16.89 / 4.18 s
HTTP generation latency at 4736×3552 / 1024×1024 · 4× Ascend 910B1 ✅ 30 / 30 numerical checks passed. Latency targets unmet. Benchmark
Qwen-Image-2.1-Turbo −30.60%
Baseline: Pinned control · 384.16 ms
Warm HTTP latency · 4× B300 ✅ 44 pixel-identical outputs. InferRSI results
Qwen-Image-2.1-Turbo −22.20%
Baseline: Upstream parent · 2,921.23 ms
Warm offline generation latency · 1× Ascend 910B1 ⚠️ Upstream reference; 18 pixel-identical outputs. Teardown inconclusive. Upstream results

🔊 Speech and multimodal serving

Model Measured change and baseline Metric and hardware Validation and evidence
LFM3-Omni-2B +108.1%
Baseline: Functional eager adapter · 0.285 req/s
Offline greedy request throughput · 1× Ascend 910B1, C4 ✅ Retained audio bit-identical at every tested temperature. Upstream results
MiniCPM-o 4.5 +100.6% / −50.72%
Baseline: Pinned upstream main · 11.46 req/s / 0.65548 RTF
Request throughput / mean real-time factor · 1× L20X, C32 ⚠️ Five matched runs per arm; speech-quality non-inferiority unestablished. Campaign · Benchmark
MiniCPM-o 4.5 −27.69%
Baseline: Tuned 50-frame deployment before packed conditioning
Median time to first audio · 1× L20X, C32 ⚠️ Missed the exploratory word-error-rate margin. Benchmark
MiniCPM-o 4.5 +20.5% / +25.4%
Baseline: Pinned upstream main before TensorRT + MPS
Throughput at C16 / C32 · 1× L20X ⚠️ Speech quality check failed. Benchmark
Qwen3-TTS 1.7B +158%
Baseline: Fresh no-PR control · 1.869 audio s/s
Streaming audio throughput · 1× Ascend 910B1, C1 🔥 ⚠️ Historical upstream reference; Mandarin speaker similarity and some ASR scores regressed. Upstream benchmark
Drama 3 +31% / −23%
Baseline: Piecewise graphs · 17.31 frames/s / 1.237 RTF
Throughput / mean real-time factor · 2× Ascend 910B1, C1 Two runs per arm; real-time-factor targets unmet. Campaign · Benchmark

⚙️ Standalone engines

Engine projects report these comparisons. Speech benchmarks use one concurrent request per engine and measure audio frames per second and median time to first audio.

Engine Measured change and baseline Metric and hardware Benchmark scope and evidence
Qwen3-TTS +381% / −81%
Baseline: vLLM-Omni · 12.71 frames/s / 263.51 ms
Audio throughput / time to first audio · 1× Ascend 910B1 🔥 ✅ Three runs per engine; audio and contract checks passed. Same-card benchmark
MiniCPM-o 4.5 +112% / −57%
Baseline: vLLM-Omni · 187.52 frames/s / 295.3 ms
Audio throughput / time to first audio · 1× L20X ⚠️ Target unmet; quality unqualified. Same-card benchmark
Drama 3 +58% / −31%
Baseline: vLLM-Omni · 195.89 frames/s / 47.29 ms
Audio throughput / time to first audio · 1× B300 Three-run medians; overall target unmet. Same-card benchmark
h3-metal −59%
Baseline: h3.c · 73.3 s
Warm video generation latency · Apple M5 Max Both engines used the original MiniMax-H3 at 20 steps. Author benchmark

Run your first experiment · Benchmark details and historical records · vLLM-Omni model optimization roadmap

📚 Documentation

🤝 Contributing

🤝 People decide in issues; agents write the PRs.

People agree on the scope, resources, and acceptance criteria in the issue. Agents implement the agreed scope, run tests, and submit PRs with evidence. To merge, a PR must pass the required checks and receive two independent approvals. The issue owner uses the evidence to accept the work or reopen it.

Start with a reproducible workload, a baseline, and evidence of the bottleneck. Sign off commits with git commit -s. Open an issue · Human–agent collaboration · Contributor guide · License

Run the local development checks before submitting a change.

python -m pip install -e ".[dev]"
pytest
ruff check src tests

About

No description, website, or topics provided.

Resources

Contributing

Stars

93 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages