Speed test matrix
Trinity-Mini-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Mini-Base --served-model-name trinity --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context44.1K186 ms TTFT
16K context40.1K405 ms TTFT
32K context26.1K1,238 ms TTFT
64K context35.4K1,824 ms TTFT
123K context28.1K4,408 ms TTFT
128K context28K4,609 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 232.953 ms TTFT | 395.876 ms TTFT | 611.1123 ms TTFT | 961.4152 ms TTFT | 1,414.5220 ms TTFT | 2,224.2321 ms TTFT | 3,616.7541 ms TTFT | 5,521.0886 ms TTFT |
| 16K | 231.876 ms TTFT | 394.3103 ms TTFT | 605.3144 ms TTFT | 948.8182 ms TTFT | 1,373.7307 ms TTFT | 2,185.9493 ms TTFT | 3,391.5811 ms TTFT | 4,982.41,434 ms TTFT |
| 32K | 227.5119 ms TTFT | 384.7174 ms TTFT | 589.8173 ms TTFT | 897.3299 ms TTFT | 1,307.1513 ms TTFT | 1,968.9841 ms TTFT | 3,004.51,578 ms TTFT | Over capacity |
| 64K | 220.5187 ms TTFT | 368.3251 ms TTFT | 559.7238 ms TTFT | 838.9614 ms TTFT | 1,217.61,069 ms TTFT | 1,732.61,880 ms TTFT | 2,438.13,656 ms TTFT | Over capacity |
| 123K | 208.4361 ms TTFT | 340.9383 ms TTFT | 514.4423 ms TTFT | 752.11,236 ms TTFT | 1,022.42,068 ms TTFT | 1,375.33,515 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 44.1K | — | 186 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 40.1K | — | 405 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.1K | — | 1,238 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35.4K | — | 1,824 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 28.1K | — | 4,408 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28K | — | 4,609 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 232.9 | 233.5 | 53 ms | 4.3 ms | 1.0 | 1 | 6,986 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 395.8 | 198.8 | 76 ms | 5.0 ms | 2.0 | 2 | 11,874 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 611.1 | 154.6 | 123 ms | 6.5 ms | 3.9 | 4 | 18,329 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 961.4 | 123.4 | 152 ms | 8.1 ms | 8.0 | 8 | 28,842 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,414.5 | 89.1 | 220 ms | 11.2 ms | 16.0 | 0 | 42,432 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,224.2 | 70.4 | 321 ms | 14.2 ms | 32.0 | 0 | 66,720 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,616.7 | 57.0 | 541 ms | 17.5 ms | 64.0 | 0 | 108,480 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,521.0 | 43.5 | 886 ms | 23.0 ms | 128.0 | 0 | 165,633 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 231.8 | 232.7 | 76 ms | 4.3 ms | 1.0 | 1 | 6,954 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 394.3 | 198.0 | 103 ms | 5.0 ms | 1.9 | 2 | 11,827 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 605.3 | 152.5 | 144 ms | 6.6 ms | 4.0 | 4 | 18,157 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 948.8 | 125.8 | 182 ms | 8.0 ms | 8.0 | 8 | 28,473 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,373.7 | 86.9 | 307 ms | 11.5 ms | 16.0 | 0 | 41,200 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,185.9 | 69.2 | 493 ms | 14.4 ms | 32.0 | 0 | 65,568 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,391.5 | 53.6 | 811 ms | 18.7 ms | 64.0 | 0 | 101,760 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,982.4 | 39.4 | 1,434 ms | 25.4 ms | 128.0 | 0 | 149,504 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 227.5 | 228.9 | 119 ms | 4.4 ms | 1.0 | 1 | 6,826 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 384.7 | 193.8 | 174 ms | 5.2 ms | 2.0 | 2 | 11,540 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 589.8 | 149.7 | 173 ms | 6.7 ms | 3.9 | 4 | 17,695 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 897.3 | 113.0 | 299 ms | 8.8 ms | 8.0 | 0 | 26,936 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,307.1 | 82.2 | 513 ms | 12.2 ms | 16.0 | 0 | 39,216 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,968.9 | 62.1 | 841 ms | 16.1 ms | 32.0 | 0 | 59,072 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,004.5 | 47.4 | 1,578 ms | 21.1 ms | 64.0 | 0 | 90,128 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 220.5 | 222.4 | 187 ms | 4.5 ms | 1.0 | 1 | 6,618 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 368.3 | 186.3 | 251 ms | 5.4 ms | 2.0 | 2 | 11,049 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 559.7 | 141.9 | 238 ms | 7.0 ms | 4.0 | 4 | 16,797 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 838.9 | 105.5 | 614 ms | 9.5 ms | 8.0 | 0 | 25,160 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,217.6 | 76.5 | 1,069 ms | 13.1 ms | 16.0 | 0 | 36,528 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,732.6 | 54.6 | 1,880 ms | 18.3 ms | 32.0 | 0 | 51,968 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,438.1 | 38.4 | 3,656 ms | 26.0 ms | 64.0 | 0 | 73,152 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 208.4 | 211.5 | 361 ms | 4.7 ms | 1.0 | 1 | 6,250 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 340.9 | 173.3 | 383 ms | 5.8 ms | 2.0 | 2 | 10,228 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 514.4 | 130.7 | 423 ms | 7.7 ms | 3.9 | 4 | 15,428 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 752.1 | 94.3 | 1,236 ms | 10.6 ms | 8.0 | 0 | 22,577 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,022.4 | 64.0 | 2,068 ms | 15.6 ms | 16.0 | 0 | 30,674 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,375.3 | 43.3 | 3,515 ms | 23.1 ms | 32.0 | 0 | 41,248 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.