Speed test matrix
Trinity-Mini-FP8-Block on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_DEEP_GEMM=1 "$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Mini-FP8-Block --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context46.4K177 ms TTFT
16K context46.9K347 ms TTFT
32K context48.7K664 ms TTFT
64K context43.1K1,498 ms TTFT
123K context32.8K3,784 ms TTFT
128K context31.9K4,039 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 218.861 ms TTFT | 372.663 ms TTFT | 595.185 ms TTFT | 912.2173 ms TTFT | 1,318.4246 ms TTFT | 2,284.8348 ms TTFT | 3,840.4621 ms TTFT | 6,041.8902 ms TTFT |
| 16K | 217.289 ms TTFT | 369.3119 ms TTFT | 587.3162 ms TTFT | 895.7203 ms TTFT | 1,276.9299 ms TTFT | 2,191.4487 ms TTFT | 3,593.9829 ms TTFT | 5,423.51,467 ms TTFT |
| 32K | 213.1127 ms TTFT | 361.7125 ms TTFT | 573.9195 ms TTFT | 873.0301 ms TTFT | 1,225.1473 ms TTFT | 2,031.3848 ms TTFT | 3,167.91,400 ms TTFT | 4,488.42,833 ms TTFT |
| 64K | 207.2188 ms TTFT | 346.8245 ms TTFT | 546.0248 ms TTFT | 815.8607 ms TTFT | 1,124.61,096 ms TTFT | 1,763.21,862 ms TTFT | 2,559.23,729 ms TTFT | Over capacity |
| 123K | 196.3372 ms TTFT | 322.9404 ms TTFT | 500.7449 ms TTFT | 728.71,069 ms TTFT | 971.51,988 ms TTFT | 1,412.73,374 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 46.4K | — | 177 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.9K | — | 347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 48.7K | — | 664 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 43.1K | — | 1,498 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 32.8K | — | 3,784 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 31.9K | — | 4,039 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 218.8 | 219.5 | 61 ms | 4.6 ms | 1.0 | 1 | 6,564 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 372.6 | 186.9 | 63 ms | 5.3 ms | 2.0 | 2 | 11,178 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 595.1 | 149.3 | 85 ms | 6.7 ms | 4.0 | 4 | 17,852 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 912.2 | 114.4 | 173 ms | 8.7 ms | 8.0 | 0 | 27,360 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,318.4 | 82.7 | 246 ms | 12.1 ms | 16.0 | 0 | 39,552 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,284.8 | 71.9 | 348 ms | 13.9 ms | 32.0 | 0 | 68,545 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,840.4 | 60.4 | 621 ms | 16.6 ms | 64.0 | 0 | 115,200 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,041.8 | 47.6 | 902 ms | 21.0 ms | 128.0 | 0 | 181,248 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 217.2 | 218.1 | 89 ms | 4.6 ms | 1.0 | 1 | 6,514 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 369.3 | 185.7 | 119 ms | 5.4 ms | 2.0 | 2 | 11,080 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 587.3 | 147.8 | 162 ms | 6.8 ms | 3.9 | 4 | 17,616 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 895.7 | 112.2 | 203 ms | 8.9 ms | 8.0 | 0 | 26,872 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,276.9 | 80.2 | 299 ms | 12.5 ms | 16.0 | 0 | 38,304 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,191.4 | 68.8 | 487 ms | 14.5 ms | 32.0 | 0 | 65,729 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,593.9 | 56.4 | 829 ms | 17.7 ms | 64.0 | 0 | 107,840 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,423.5 | 42.7 | 1,467 ms | 23.4 ms | 128.0 | 0 | 162,688 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 213.1 | 214.5 | 127 ms | 4.7 ms | 1.0 | 1 | 6,396 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 361.7 | 181.9 | 125 ms | 5.5 ms | 2.0 | 2 | 10,851 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 573.9 | 144.8 | 195 ms | 6.9 ms | 4.0 | 4 | 17,215 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 873.0 | 109.3 | 301 ms | 9.1 ms | 8.0 | 0 | 26,192 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,225.1 | 76.8 | 473 ms | 13.0 ms | 16.0 | 0 | 36,752 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,031.3 | 63.7 | 848 ms | 15.7 ms | 32.0 | 0 | 60,928 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,167.9 | 49.8 | 1,400 ms | 20.1 ms | 64.0 | 0 | 95,040 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,488.4 | 35.3 | 2,833 ms | 28.3 ms | 128.0 | 0 | 134,656 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 207.2 | 209.1 | 188 ms | 4.8 ms | 1.0 | 1 | 6,217 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 346.8 | 175.3 | 245 ms | 5.7 ms | 2.0 | 2 | 10,404 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 546.0 | 137.7 | 248 ms | 7.3 ms | 4.0 | 4 | 16,376 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 815.8 | 102.0 | 607 ms | 9.8 ms | 8.0 | 0 | 24,472 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,124.6 | 70.4 | 1,096 ms | 14.2 ms | 16.0 | 0 | 33,728 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,763.2 | 55.2 | 1,862 ms | 18.1 ms | 32.0 | 0 | 52,896 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,559.2 | 40.1 | 3,729 ms | 24.9 ms | 64.0 | 0 | 76,736 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 196.3 | 199.3 | 372 ms | 5.0 ms | 1.0 | 1 | 5,889 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 322.9 | 164.2 | 404 ms | 6.1 ms | 2.0 | 2 | 9,692 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 500.7 | 127.0 | 449 ms | 7.9 ms | 3.9 | 4 | 15,019 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 728.7 | 91.0 | 1,069 ms | 11.0 ms | 8.0 | 0 | 21,856 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 971.5 | 60.6 | 1,988 ms | 16.5 ms | 16.0 | 0 | 29,136 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,412.7 | 44.1 | 3,374 ms | 22.7 ms | 32.0 | 0 | 42,368 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.