Speed test matrix
Trinity-Nano-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-arcee-ai-vllm-73a5831127a9/bin/vllm" serve arcee-ai/Trinity-Nano-Base --served-model-name trinity --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context74K111 ms TTFT
16K context32K508 ms TTFT
32K context64K506 ms TTFT
64K context56.6K1,141 ms TTFT
123K context41.3K3,001 ms TTFT
128K context37.5K3,438 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 189.032 ms TTFT | 301.535 ms TTFT | 537.341 ms TTFT | 905.586 ms TTFT | Looped | Looped | 3,299.0480 ms TTFT | Looped |
| 16K | 186.950 ms TTFT | 295.562 ms TTFT | Looped | Looped | Looped | 1,996.8411 ms TTFT | Looped | Looped |
| 32K | 181.673 ms TTFT | 285.673 ms TTFT | 509.480 ms TTFT | Looped | Looped | 1,766.5805 ms TTFT | 2,553.21,479 ms TTFT | Over capacity |
| 64K | 167.7158 ms TTFT | 268.7178 ms TTFT | Looped | 737.4492 ms TTFT | Looped | 1,469.61,550 ms TTFT | Over capacity | Over capacity |
| 123K | Looped | 241.4338 ms TTFT | Looped | 635.9992 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 74K | — | 111 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 64K | — | 506 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 56.6K | — | 1,141 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 41.3K | — | 3,001 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 37.5K | — | 3,438 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 189.0 | 189.3 | 32 ms | 5.3 ms | 1.0 | 1 | 5,671 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 301.5 | 151.2 | 35 ms | 6.6 ms | 2.0 | 2 | 9,045 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 537.3 | 135.8 | 41 ms | 7.4 ms | 4.0 | 4 | 16,124 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 905.5 | 113.4 | 86 ms | 8.8 ms | 8.0 | 0 | 27,160 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 3,299.0 | 51.8 | 480 ms | 19.3 ms | 64.0 | 0 | 98,944 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 186.9 | 187.4 | 50 ms | 5.3 ms | 1.0 | 1 | 5,608 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 295.5 | 148.4 | 62 ms | 6.7 ms | 2.0 | 2 | 8,862 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 1,996.8 | 62.7 | 411 ms | 15.9 ms | 32.0 | 0 | 59,905 | 30.0 s |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 181.6 | 182.5 | 73 ms | 5.5 ms | 1.0 | 1 | 5,449 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 285.6 | 143.9 | 73 ms | 6.9 ms | 2.0 | 2 | 8,570 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 509.4 | 128.7 | 80 ms | 7.8 ms | 4.0 | 4 | 15,280 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,766.5 | 55.7 | 805 ms | 18.0 ms | 32.0 | 0 | 52,992 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,553.2 | 40.3 | 1,479 ms | 24.8 ms | 64.0 | 0 | 76,544 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 167.7 | 168.7 | 158 ms | 5.9 ms | 1.0 | 1 | 5,030 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 268.7 | 135.7 | 178 ms | 7.4 ms | 2.0 | 2 | 8,061 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 737.4 | 92.3 | 492 ms | 10.8 ms | 8.0 | 0 | 22,120 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,469.6 | 46.0 | 1,550 ms | 21.8 ms | 32.0 | 0 | 44,064 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 2 | Valid | 241.4 | 123.6 | 338 ms | 8.1 ms | 2.0 | 2 | 7,241 | 30.0 s |
| Sustained decode | 123K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 8 | Valid | 635.9 | 79.5 | 992 ms | 12.6 ms | 8.0 | 0 | 19,088 | 30.0 s |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.