Speed test matrix
Trinity-Mini-FP8-Block on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-arcee-ai-vllm-73a5831127a9/bin/vllm" serve arcee-ai/Trinity-Mini-FP8-Block --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend triton --moe-backend triton
Prefill
prompt tok/s by context length8K context39.5K207 ms TTFT
16K context38.1K427 ms TTFT
32K context33.9K954 ms TTFT
64K context27.5K2,353 ms TTFT
123K context20.1K6,170 ms TTFT
128K context18.8K6,864 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 154.133 ms TTFT | 267.241 ms TTFT | 467.162 ms TTFT | 667.895 ms TTFT | 1,016.9158 ms TTFT | 1,456.1258 ms TTFT | 2,098.4482 ms TTFT | 2,706.9905 ms TTFT |
| 16K | 151.953 ms TTFT | 264.765 ms TTFT | 458.296 ms TTFT | 652.1147 ms TTFT | Looped | Looped | 1,947.9833 ms TTFT | Over capacity |
| 32K | 147.782 ms TTFT | Looped | 443.3147 ms TTFT | 626.8240 ms TTFT | 907.1432 ms TTFT | 1,264.6801 ms TTFT | Over capacity | Over capacity |
| 64K | 140.2155 ms TTFT | Looped | 413.8288 ms TTFT | 577.8475 ms TTFT | 820.8859 ms TTFT | 1,083.41,617 ms TTFT | Over capacity | Over capacity |
| 123K | 127.2311 ms TTFT | 203.8399 ms TTFT | 361.9587 ms TTFT | 500.6958 ms TTFT | 675.31,780 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.5K | — | 207 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.1K | — | 427 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.9K | — | 954 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27.5K | — | 2,353 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 20.1K | — | 6,170 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.8K | — | 6,864 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 154.1 | 154.5 | 33 ms | 6.5 ms | 1.0 | 1 | 4,623 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 267.2 | 134.6 | 41 ms | 7.4 ms | 2.0 | 2 | 8,014 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 467.1 | 116.9 | 62 ms | 8.6 ms | 4.0 | 0 | 14,012 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 667.8 | 83.7 | 95 ms | 12.0 ms | 8.0 | 0 | 20,035 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,016.9 | 64.0 | 158 ms | 15.6 ms | 16.0 | 0 | 30,496 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,456.1 | 45.9 | 258 ms | 21.8 ms | 32.0 | 0 | 43,680 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,098.4 | 33.0 | 482 ms | 30.3 ms | 64.0 | 0 | 62,951 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,706.9 | 21.3 | 905 ms | 47.0 ms | 128.0 | 0 | 81,208 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 151.9 | 152.4 | 53 ms | 6.6 ms | 1.0 | 1 | 4,555 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 264.7 | 132.8 | 65 ms | 7.5 ms | 2.0 | 2 | 7,942 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 458.2 | 114.6 | 96 ms | 8.7 ms | 4.0 | 0 | 13,744 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 652.1 | 81.8 | 147 ms | 12.2 ms | 8.0 | 0 | 19,564 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 1,947.9 | 30.5 | 833 ms | 32.8 ms | 64.0 | 0 | 58,432 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 147.7 | 148.4 | 82 ms | 6.7 ms | 1.0 | 1 | 4,431 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 443.3 | 110.8 | 147 ms | 9.0 ms | 4.0 | 0 | 13,300 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 626.8 | 78.4 | 240 ms | 12.8 ms | 8.0 | 0 | 18,800 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 907.1 | 56.8 | 432 ms | 17.6 ms | 16.0 | 0 | 27,200 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,264.6 | 39.5 | 801 ms | 25.3 ms | 32.0 | 0 | 37,920 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 140.2 | 141.5 | 155 ms | 7.1 ms | 1.0 | 1 | 4,205 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 413.8 | 103.4 | 288 ms | 9.7 ms | 4.0 | 0 | 12,412 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 577.8 | 72.2 | 475 ms | 13.8 ms | 8.0 | 0 | 17,328 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 820.8 | 51.2 | 859 ms | 19.5 ms | 16.0 | 0 | 24,625 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,083.4 | 33.7 | 1,617 ms | 29.7 ms | 32.0 | 0 | 32,480 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 127.2 | 131.3 | 311 ms | 7.6 ms | 1.0 | 1 | 3,816 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 203.8 | 102.1 | 399 ms | 9.8 ms | 2.0 | 0 | 6,115 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 361.9 | 90.4 | 587 ms | 11.1 ms | 4.0 | 0 | 10,856 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 500.6 | 62.5 | 958 ms | 16.0 ms | 8.0 | 0 | 15,016 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 675.3 | 42.0 | 1,780 ms | 23.8 ms | 16.0 | 0 | 20,256 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.