Speed test matrix
Trinity-Mini-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-arcee-ai-vllm-73a5831127a9/bin/vllm" serve arcee-ai/Trinity-Mini-Base --served-model-name trinity --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context34.3K239 ms TTFT
16K context33.6K484 ms TTFT
32K context30.4K1,065 ms TTFT
64K context25.1K2,571 ms TTFT
123K context18.7K6,620 ms TTFT
128K context18.5K6,991 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 148.333 ms TTFT | 227.744 ms TTFT | 339.865 ms TTFT | 434.897 ms TTFT | 647.6175 ms TTFT | 946.3279 ms TTFT | 1,414.2509 ms TTFT | Over capacity |
| 16K | 146.361 ms TTFT | 229.373 ms TTFT | 329.4108 ms TTFT | 430.3163 ms TTFT | 619.9278 ms TTFT | 899.2457 ms TTFT | Over capacity | Over capacity |
| 32K | 142.686 ms TTFT | 221.6104 ms TTFT | Looped | 414.5246 ms TTFT | 593.2443 ms TTFT | 850.2799 ms TTFT | Over capacity | Over capacity |
| 64K | 135.3164 ms TTFT | 211.7216 ms TTFT | 308.0296 ms TTFT | 416.7493 ms TTFT | 554.0857 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 123.5321 ms TTFT | 183.0413 ms TTFT | 275.2642 ms TTFT | 369.01,110 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.3K | — | 239 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.6K | — | 484 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.4K | — | 1,065 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.1K | — | 2,571 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 18.7K | — | 6,620 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,991 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 148.3 | 148.7 | 33 ms | 6.7 ms | 1.0 | 1 | 4,450 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 227.7 | 114.1 | 44 ms | 8.8 ms | 2.0 | 0 | 6,830 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 339.8 | 85.5 | 65 ms | 11.7 ms | 4.0 | 0 | 10,192 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 434.8 | 54.9 | 97 ms | 18.2 ms | 8.0 | 0 | 13,040 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 647.6 | 40.9 | 175 ms | 24.5 ms | 16.0 | 0 | 19,424 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 946.3 | 29.8 | 279 ms | 33.6 ms | 32.0 | 0 | 28,384 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,414.2 | 22.2 | 509 ms | 45.0 ms | 64.0 | 0 | 42,368 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 146.3 | 146.8 | 61 ms | 6.8 ms | 1.0 | 1 | 4,390 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 229.3 | 114.6 | 73 ms | 8.7 ms | 2.0 | 0 | 6,880 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 329.4 | 82.7 | 108 ms | 12.1 ms | 4.0 | 0 | 9,880 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 430.3 | 54.3 | 163 ms | 18.4 ms | 8.0 | 0 | 12,904 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 619.9 | 39.1 | 278 ms | 25.6 ms | 16.0 | 0 | 18,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 899.2 | 28.3 | 457 ms | 35.4 ms | 32.0 | 0 | 26,976 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 142.6 | 143.1 | 86 ms | 7.0 ms | 1.0 | 1 | 4,278 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 221.6 | 111.3 | 104 ms | 9.0 ms | 2.0 | 0 | 6,646 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 414.5 | 52.0 | 246 ms | 19.2 ms | 8.0 | 0 | 12,432 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 593.2 | 37.3 | 443 ms | 26.8 ms | 16.0 | 0 | 17,792 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 850.2 | 26.7 | 799 ms | 37.5 ms | 32.0 | 0 | 25,472 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 135.3 | 136.6 | 164 ms | 7.3 ms | 1.0 | 1 | 4,059 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 211.7 | 106.2 | 216 ms | 9.4 ms | 2.0 | 0 | 6,350 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 308.0 | 77.5 | 296 ms | 12.9 ms | 4.0 | 0 | 9,236 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 416.7 | 52.5 | 493 ms | 19.0 ms | 8.0 | 0 | 12,496 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 554.0 | 34.8 | 857 ms | 28.8 ms | 16.0 | 0 | 16,608 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 123.5 | 172.1 | 321 ms | 6.3 ms | 1.0 | 1 | 3,705 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 183.0 | 91.8 | 413 ms | 10.9 ms | 2.0 | 0 | 5,490 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 275.2 | 69.3 | 642 ms | 14.4 ms | 4.0 | 0 | 8,252 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 369.0 | 46.4 | 1,110 ms | 21.6 ms | 8.0 | 0 | 11,064 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.