Speed test matrix
Trinity-Mini-NVFP4 on H200 NVL 141GB
vllm · 0.18.0 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.18.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-arcee-nvfp4-vllm/bin/vllm" serve arcee-ai/Trinity-Mini-NVFP4 --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.873405 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context30K273 ms TTFT
16K context34K478 ms TTFT
32K context33.2K974 ms TTFT
64K context30K2,150 ms TTFT
123K context24.4K5,072 ms TTFT
128K context24K5,376 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 217.949 ms TTFT | 373.371 ms TTFT | 673.9111 ms TTFT | 962.21,397 ms TTFT | 1,652.71,522 ms TTFT | 2,693.3642 ms TTFT | 4,019.3705 ms TTFT | 6,204.81,391 ms TTFT |
| 16K | 215.978 ms TTFT | 370.5106 ms TTFT | 662.7185 ms TTFT | 949.5312 ms TTFT | 1,605.8574 ms TTFT | 2,567.31,051 ms TTFT | 3,753.11,068 ms TTFT | 5,552.71,960 ms TTFT |
| 32K | 212.8105 ms TTFT | 362.7188 ms TTFT | 644.8330 ms TTFT | 918.5593 ms TTFT | 1,521.11,101 ms TTFT | 2,348.01,986 ms TTFT | 3,286.02,006 ms TTFT | 4,561.73,782 ms TTFT |
| 64K | 205.7193 ms TTFT | 347.9341 ms TTFT | 605.2674 ms TTFT | 861.11,186 ms TTFT | 1,359.21,386 ms TTFT | 1,997.74,422 ms TTFT | 2,634.34,482 ms TTFT | Over capacity |
| 123K | 195.6365 ms TTFT | 319.3717 ms TTFT | 546.71,299 ms TTFT | 764.21,683 ms TTFT | 1,141.82,583 ms TTFT | 1,567.59,232 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30K | — | 273 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34K | — | 478 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.2K | — | 974 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30K | — | 2,150 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 24.4K | — | 5,072 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24K | — | 5,376 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 217.9 | 218.4 | 49 ms | 4.6 ms | 1.0 | 1 | 6,536 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 373.3 | 187.5 | 71 ms | 5.3 ms | 2.0 | 2 | 11,200 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 673.9 | 169.2 | 111 ms | 5.9 ms | 4.0 | 4 | 20,217 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 962.2 | 121.8 | 1,397 ms | 8.2 ms | 8.0 | 0 | 28,864 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,652.7 | 104.3 | 1,522 ms | 9.6 ms | 16.0 | 0 | 49,584 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,693.3 | 85.4 | 642 ms | 11.7 ms | 32.0 | 0 | 80,800 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,019.3 | 63.6 | 705 ms | 15.7 ms | 64.0 | 0 | 120,576 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,204.8 | 48.9 | 1,391 ms | 20.4 ms | 128.0 | 0 | 186,112 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 215.9 | 216.7 | 78 ms | 4.6 ms | 1.0 | 1 | 6,479 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 370.5 | 186.2 | 106 ms | 5.4 ms | 2.0 | 2 | 11,114 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 662.7 | 166.9 | 185 ms | 6.0 ms | 3.9 | 4 | 19,880 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 949.5 | 120.5 | 312 ms | 8.3 ms | 8.0 | 8 | 28,485 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,605.8 | 101.4 | 574 ms | 9.9 ms | 16.0 | 0 | 48,177 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,567.3 | 81.3 | 1,051 ms | 12.3 ms | 32.0 | 0 | 77,024 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,753.1 | 59.0 | 1,068 ms | 16.9 ms | 64.0 | 0 | 112,576 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,552.7 | 43.7 | 1,960 ms | 22.9 ms | 128.0 | 0 | 166,528 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 212.8 | 213.8 | 105 ms | 4.7 ms | 1.0 | 1 | 6,383 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 362.7 | 182.8 | 188 ms | 5.5 ms | 2.0 | 2 | 10,880 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 644.8 | 163.1 | 330 ms | 6.1 ms | 4.0 | 4 | 19,345 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 918.5 | 115.9 | 593 ms | 8.6 ms | 8.0 | 0 | 27,560 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,521.1 | 96.2 | 1,101 ms | 10.4 ms | 16.0 | 0 | 45,632 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,348.0 | 74.6 | 1,986 ms | 13.4 ms | 32.0 | 0 | 70,432 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,286.0 | 51.5 | 2,006 ms | 19.4 ms | 64.0 | 0 | 98,560 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,561.7 | 35.8 | 3,782 ms | 27.9 ms | 128.0 | 0 | 136,832 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 205.7 | 207.5 | 193 ms | 4.8 ms | 1.0 | 1 | 6,172 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 347.9 | 176.0 | 341 ms | 5.7 ms | 2.0 | 2 | 10,437 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 605.2 | 154.8 | 674 ms | 6.5 ms | 4.0 | 4 | 18,154 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 861.1 | 108.6 | 1,186 ms | 9.2 ms | 8.0 | 0 | 25,848 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,359.2 | 85.1 | 1,386 ms | 11.7 ms | 16.0 | 0 | 40,768 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,997.7 | 63.8 | 4,422 ms | 15.7 ms | 32.0 | 0 | 59,936 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,634.3 | 41.3 | 4,482 ms | 24.2 ms | 64.0 | 0 | 79,040 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 195.6 | 198.5 | 365 ms | 5.0 ms | 1.0 | 1 | 5,868 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 319.3 | 164.4 | 717 ms | 6.1 ms | 1.9 | 2 | 9,578 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 546.7 | 145.0 | 1,299 ms | 6.9 ms | 3.9 | 4 | 16,399 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 764.2 | 96.2 | 1,683 ms | 10.4 ms | 8.0 | 0 | 22,920 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,141.8 | 71.4 | 2,583 ms | 14.0 ms | 16.0 | 0 | 34,256 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,567.5 | 50.5 | 9,232 ms | 19.8 ms | 32.0 | 0 | 47,008 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.