Speed test matrix
Trinity-Mini on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-arcee-ai-vllm-73a5831127a9/bin/vllm" serve arcee-ai/Trinity-Mini --served-model-name trinity --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context34.9K235 ms TTFT
16K context33.6K484 ms TTFT
32K context30.4K1,065 ms TTFT
64K context25.1K2,573 ms TTFT
123K context18.8K6,597 ms TTFT
128K context18.5K6,979 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 143.637 ms TTFT | 229.751 ms TTFT | 350.366 ms TTFT | 482.799 ms TTFT | 712.5183 ms TTFT | 1,017.5266 ms TTFT | 1,560.9509 ms TTFT | Over capacity |
| 16K | 141.660 ms TTFT | 227.874 ms TTFT | Looped | 473.7147 ms TTFT | 696.7277 ms TTFT | 985.8469 ms TTFT | Over capacity | Over capacity |
| 32K | 138.083 ms TTFT | 221.1102 ms TTFT | 336.3160 ms TTFT | 461.5263 ms TTFT | 657.1441 ms TTFT | 932.7859 ms TTFT | Over capacity | Over capacity |
| 64K | 131.2169 ms TTFT | 208.6218 ms TTFT | 322.8326 ms TTFT | 438.3503 ms TTFT | 615.0915 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 121.0311 ms TTFT | 180.3398 ms TTFT | 288.2619 ms TTFT | 392.21,047 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.9K | — | 235 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.6K | — | 484 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.4K | — | 1,065 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.1K | — | 2,573 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 18.8K | — | 6,597 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,979 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 143.6 | 143.9 | 37 ms | 6.9 ms | 1.0 | 1 | 4,307 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 229.7 | 115.1 | 51 ms | 8.7 ms | 2.0 | 0 | 6,888 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 350.3 | 87.9 | 66 ms | 11.4 ms | 4.0 | 0 | 10,508 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 482.7 | 61.1 | 99 ms | 16.4 ms | 8.0 | 0 | 14,480 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 712.5 | 45.4 | 183 ms | 22.0 ms | 16.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,017.5 | 32.5 | 266 ms | 30.8 ms | 32.0 | 0 | 30,496 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,560.9 | 24.8 | 509 ms | 40.3 ms | 64.0 | 0 | 46,784 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 141.6 | 142.1 | 60 ms | 7.0 ms | 1.0 | 1 | 4,248 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 227.8 | 113.9 | 74 ms | 8.8 ms | 2.0 | 0 | 6,832 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 473.7 | 59.6 | 147 ms | 16.8 ms | 8.0 | 0 | 14,208 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 696.7 | 43.8 | 277 ms | 22.8 ms | 16.0 | 0 | 20,901 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 985.8 | 31.2 | 469 ms | 32.1 ms | 32.0 | 0 | 29,568 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 138.0 | 138.7 | 83 ms | 7.2 ms | 1.0 | 1 | 4,141 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 221.1 | 110.6 | 102 ms | 9.0 ms | 2.0 | 0 | 6,632 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 336.3 | 84.1 | 160 ms | 11.9 ms | 4.0 | 0 | 10,089 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 461.5 | 57.9 | 263 ms | 17.3 ms | 8.0 | 0 | 13,840 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 657.1 | 41.4 | 441 ms | 24.1 ms | 16.0 | 0 | 19,712 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 932.7 | 29.3 | 859 ms | 34.2 ms | 32.0 | 0 | 27,968 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 131.2 | 132.7 | 169 ms | 7.5 ms | 1.0 | 1 | 3,936 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 208.6 | 104.5 | 218 ms | 9.6 ms | 2.0 | 0 | 6,259 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 322.8 | 80.6 | 326 ms | 12.4 ms | 4.0 | 0 | 9,680 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 438.3 | 54.7 | 503 ms | 18.3 ms | 8.0 | 0 | 13,144 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 615.0 | 38.5 | 915 ms | 26.0 ms | 16.0 | 0 | 18,451 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 121.0 | 121.2 | 311 ms | 8.3 ms | 1.0 | 0 | 3,630 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 180.3 | 90.2 | 398 ms | 11.1 ms | 2.0 | 0 | 5,408 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 288.2 | 72.0 | 619 ms | 13.9 ms | 4.0 | 0 | 8,644 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 392.2 | 49.0 | 1,047 ms | 20.4 ms | 8.0 | 0 | 11,760 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.