Speed test matrix
Trinity-Large-Thinking-FP8-Block on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev455+g22f931fe2 · FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev455+g22f931fe2
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
22f931fe2177
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-arcee-vllm-pr-54479-thinking-fp8-block/bin/vllm" serve arcee-ai/Trinity-Large-Thinking-FP8-Block --served-model-name trinity --trust-remote-code --tensor-parallel-size 4 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 524288 --gpu-memory-utilization 0.869020 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --hf-overrides '{"max_position_embeddings":524288}'Prefill
prompt tok/s by context length8K context31.3K262 ms TTFT
16K context33.3K488 ms TTFT
32K context32.6K993 ms TTFT
64K context30.2K2,138 ms TTFT
128K context14.7K8,755 ms TTFT
256K context20.9K12,370 ms TTFT
507K context14.5K35,228 ms TTFT
512K context14.1K36,594 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 152.980 ms TTFT | 275.7114 ms TTFT | 503.2130 ms TTFT | 834.8197 ms TTFT | 1,267.5289 ms TTFT | 2,018.2484 ms TTFT | 2,939.7805 ms TTFT | 3,864.01,438 ms TTFT |
| 16K | 151.6107 ms TTFT | 273.4127 ms TTFT | 497.7165 ms TTFT | 823.7263 ms TTFT | 1,242.9395 ms TTFT | 1,940.2672 ms TTFT | 2,811.61,243 ms TTFT | Over capacity |
| 32K | 149.4175 ms TTFT | 268.6190 ms TTFT | 486.6266 ms TTFT | Looped | Looped | 1,837.01,266 ms TTFT | Over capacity | Over capacity |
| 64K | 145.5291 ms TTFT | 260.1311 ms TTFT | 473.5478 ms TTFT | 759.6782 ms TTFT | 1,104.21,259 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 137.8518 ms TTFT | 244.4561 ms TTFT | 441.7950 ms TTFT | 687.61,545 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 124.8983 ms TTFT | 223.51,343 ms TTFT | 389.61,882 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.3K | — | 262 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.3K | — | 488 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.6K | — | 993 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.2K | — | 2,138 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.7K | — | 8,755 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.9K | — | 12,370 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 14.5K | — | 35,228 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 14.1K | — | 36,594 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 152.9 | 153.4 | 80 ms | 6.5 ms | 1.0 | 1 | 4,584 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 275.7 | 138.4 | 114 ms | 7.2 ms | 2.0 | 2 | 8,274 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 503.2 | 126.3 | 130 ms | 7.9 ms | 3.9 | 4 | 15,094 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 834.8 | 104.3 | 197 ms | 9.6 ms | 8.0 | 0 | 25,040 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,267.5 | 79.2 | 289 ms | 12.6 ms | 16.0 | 0 | 38,032 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,018.2 | 63.2 | 484 ms | 15.8 ms | 32.0 | 0 | 60,544 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,939.7 | 46.2 | 805 ms | 21.6 ms | 64.0 | 0 | 88,192 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,864.0 | 30.5 | 1,438 ms | 32.8 ms | 128.0 | 0 | 115,968 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 151.6 | 152.4 | 107 ms | 6.6 ms | 1.0 | 1 | 4,547 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 273.4 | 137.4 | 127 ms | 7.3 ms | 2.0 | 2 | 8,201 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 497.7 | 125.1 | 165 ms | 8.0 ms | 4.0 | 4 | 14,929 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 823.7 | 103.0 | 263 ms | 9.7 ms | 8.0 | 0 | 24,712 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,242.9 | 77.7 | 395 ms | 12.9 ms | 16.0 | 0 | 37,280 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,940.2 | 60.8 | 672 ms | 16.5 ms | 32.0 | 0 | 58,208 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,811.6 | 44.1 | 1,243 ms | 22.7 ms | 64.0 | 0 | 84,352 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 149.4 | 150.6 | 175 ms | 6.6 ms | 1.0 | 1 | 4,481 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 268.6 | 135.5 | 190 ms | 7.4 ms | 2.0 | 2 | 8,055 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 486.6 | 122.7 | 266 ms | 8.2 ms | 4.0 | 4 | 14,596 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,837.0 | 57.5 | 1,266 ms | 17.4 ms | 32.0 | 0 | 55,105 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 145.5 | 147.6 | 291 ms | 6.8 ms | 1.0 | 1 | 4,366 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 260.1 | 132.5 | 311 ms | 7.5 ms | 2.0 | 2 | 7,802 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 473.5 | 118.4 | 478 ms | 8.4 ms | 4.0 | 0 | 14,204 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 759.6 | 94.9 | 782 ms | 10.5 ms | 8.0 | 0 | 22,784 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,104.2 | 69.0 | 1,259 ms | 14.5 ms | 16.0 | 0 | 33,120 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 137.8 | 141.6 | 518 ms | 7.1 ms | 1.0 | 1 | 4,132 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.4 | 129.3 | 561 ms | 7.7 ms | 2.0 | 2 | 7,333 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 441.7 | 110.5 | 950 ms | 9.0 ms | 4.0 | 0 | 13,252 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 687.6 | 86.0 | 1,545 ms | 11.6 ms | 8.0 | 0 | 20,624 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 124.8 | 137.4 | 983 ms | 7.3 ms | 1.0 | 1 | 3,744 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 223.5 | 112.2 | 1,343 ms | 8.9 ms | 2.0 | 0 | 6,706 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 389.6 | 97.6 | 1,882 ms | 10.2 ms | 4.0 | 0 | 11,684 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 507K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.