Speed test matrix
Trinity-Nano-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Nano-Base --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.884631 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context64.7K127 ms TTFT
16K context31.5K516 ms TTFT
32K context76.5K423 ms TTFT
64K context75.5K855 ms TTFT
123K context64K1,938 ms TTFT
128K context53.9K2,394 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 198.153 ms TTFT | 386.373 ms TTFT | 705.270 ms TTFT | 861.0148 ms TTFT | 2,162.2172 ms TTFT | 3,521.4317 ms TTFT | 5,584.2514 ms TTFT | 8,293.8947 ms TTFT |
| 16K | 196.289 ms TTFT | 381.683 ms TTFT | 692.8134 ms TTFT | 851.2218 ms TTFT | 2,068.4185 ms TTFT | 3,312.2524 ms TTFT | 5,100.9835 ms TTFT | 7,307.31,559 ms TTFT |
| 32K | 193.9129 ms TTFT | 373.7153 ms TTFT | 674.2163 ms TTFT | 832.8378 ms TTFT | 1,928.2296 ms TTFT | 2,971.8903 ms TTFT | 4,394.21,486 ms TTFT | 5,904.12,826 ms TTFT |
| 64K | 188.1189 ms TTFT | 360.8239 ms TTFT | 639.5251 ms TTFT | 787.5573 ms TTFT | 1,706.61,032 ms TTFT | 2,476.21,614 ms TTFT | 3,396.53,265 ms TTFT | Over capacity |
| 123K | 178.6356 ms TTFT | 337.4404 ms TTFT | 583.3405 ms TTFT | 717.11,081 ms TTFT | 1,412.81,818 ms TTFT | 1,913.83,249 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 64.7K | — | 127 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.5K | — | 516 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 76.5K | — | 423 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 75.5K | — | 855 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 64K | — | 1,938 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 53.9K | — | 2,394 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 198.1 | 198.6 | 53 ms | 5.0 ms | 1.0 | 1 | 5,942 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 386.3 | 193.8 | 73 ms | 5.2 ms | 2.0 | 2 | 11,587 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 705.2 | 176.9 | 70 ms | 5.7 ms | 4.0 | 4 | 21,152 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 861.0 | 108.0 | 148 ms | 9.3 ms | 8.0 | 0 | 25,824 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,162.2 | 136.7 | 172 ms | 7.3 ms | 16.0 | 16 | 64,863 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,521.4 | 110.8 | 317 ms | 9.0 ms | 32.0 | 0 | 105,632 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,584.2 | 87.9 | 514 ms | 11.4 ms | 64.0 | 0 | 167,488 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 8,293.8 | 65.2 | 947 ms | 15.3 ms | 128.0 | 0 | 248,832 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 196.2 | 197.0 | 89 ms | 5.1 ms | 1.0 | 1 | 5,885 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 381.6 | 191.5 | 83 ms | 5.2 ms | 2.0 | 2 | 11,446 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 692.8 | 174.3 | 134 ms | 5.7 ms | 4.0 | 4 | 20,782 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 851.2 | 106.6 | 218 ms | 9.4 ms | 8.0 | 0 | 25,536 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,068.4 | 131.2 | 185 ms | 7.6 ms | 16.0 | 16 | 62,052 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,312.2 | 104.1 | 524 ms | 9.6 ms | 32.0 | 0 | 99,360 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,100.9 | 80.2 | 835 ms | 12.5 ms | 64.0 | 0 | 153,024 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 7,307.3 | 57.4 | 1,559 ms | 17.4 ms | 128.0 | 0 | 219,264 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 193.9 | 195.1 | 129 ms | 5.1 ms | 1.0 | 1 | 5,817 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 373.7 | 188.1 | 153 ms | 5.3 ms | 2.0 | 2 | 11,211 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 674.2 | 170.0 | 163 ms | 5.9 ms | 4.0 | 4 | 20,223 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 832.8 | 104.4 | 378 ms | 9.6 ms | 8.0 | 0 | 24,984 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,928.2 | 121.4 | 296 ms | 8.2 ms | 15.9 | 16 | 57,817 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,971.8 | 93.1 | 903 ms | 10.7 ms | 32.0 | 0 | 89,152 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,394.2 | 69.0 | 1,486 ms | 14.5 ms | 64.0 | 0 | 131,840 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,904.1 | 46.5 | 2,826 ms | 21.5 ms | 128.0 | 0 | 177,152 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 188.1 | 189.8 | 189 ms | 5.3 ms | 1.0 | 1 | 5,643 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 360.8 | 182.0 | 239 ms | 5.5 ms | 2.0 | 2 | 10,822 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 639.5 | 161.5 | 251 ms | 6.2 ms | 3.9 | 4 | 19,184 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 787.5 | 98.6 | 573 ms | 10.1 ms | 8.0 | 0 | 23,624 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,706.6 | 106.7 | 1,032 ms | 9.4 ms | 16.0 | 0 | 51,200 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,476.2 | 77.6 | 1,614 ms | 12.9 ms | 32.0 | 0 | 74,272 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,396.5 | 53.1 | 3,265 ms | 18.8 ms | 64.0 | 0 | 101,888 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 178.6 | 181.5 | 356 ms | 5.5 ms | 1.0 | 1 | 5,359 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 337.4 | 171.5 | 404 ms | 5.8 ms | 2.0 | 2 | 10,122 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 583.3 | 147.6 | 405 ms | 6.8 ms | 4.0 | 4 | 17,498 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 717.1 | 89.8 | 1,081 ms | 11.1 ms | 8.0 | 0 | 21,513 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,412.8 | 88.0 | 1,818 ms | 11.4 ms | 16.0 | 0 | 42,384 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,913.8 | 59.5 | 3,249 ms | 16.8 ms | 32.0 | 0 | 57,408 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.