Speed test matrix
Trinity-Mini-W4A16 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-arcee-ai-vllm-73a5831127a9/bin/vllm" serve arcee-ai/Trinity-Mini-W4A16 --served-model-name trinity --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --attention-backend FLASH_ATTN --linear-backend marlin --moe-backend marlin
Prefill
prompt tok/s by context length8K context34.8K235 ms TTFT
16K context33.7K482 ms TTFT
32K context30.5K1,063 ms TTFT
64K context25.2K2,568 ms TTFT
123K context18.9K6,564 ms TTFT
128K context18.5K6,971 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 191.330 ms TTFT | 343.634 ms TTFT | 596.839 ms TTFT | Looped | 1,285.3149 ms TTFT | 1,824.7243 ms TTFT | 2,494.0460 ms TTFT | 3,105.1861 ms TTFT |
| 16K | 188.249 ms TTFT | 335.256 ms TTFT | 585.761 ms TTFT | 810.7131 ms TTFT | 1,242.1243 ms TTFT | 1,726.5453 ms TTFT | 2,317.7822 ms TTFT | 2,861.71,568 ms TTFT |
| 32K | 181.881 ms TTFT | 322.684 ms TTFT | 551.288 ms TTFT | 766.0236 ms TTFT | Looped | 1,539.7795 ms TTFT | 2,020.81,577 ms TTFT | Over capacity |
| 64K | 170.5148 ms TTFT | 297.2165 ms TTFT | 499.9166 ms TTFT | 697.8458 ms TTFT | 1,001.8836 ms TTFT | 1,275.01,598 ms TTFT | Over capacity | Over capacity |
| 123K | 152.2295 ms TTFT | Looped | 430.3587 ms TTFT | 593.7947 ms TTFT | 798.21,800 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.8K | — | 235 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.7K | — | 482 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.5K | — | 1,063 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.2K | — | 2,568 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 18.9K | — | 6,564 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,971 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 191.3 | 191.6 | 30 ms | 5.2 ms | 1.0 | 1 | 5,738 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 343.6 | 172.4 | 34 ms | 5.8 ms | 2.0 | 2 | 10,306 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 596.8 | 149.9 | 39 ms | 6.7 ms | 4.0 | 4 | 17,900 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,285.3 | 80.8 | 149 ms | 12.4 ms | 16.0 | 0 | 38,558 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,824.7 | 57.3 | 243 ms | 17.5 ms | 32.0 | 0 | 54,720 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,494.0 | 39.1 | 460 ms | 25.5 ms | 64.0 | 0 | 74,815 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,105.1 | 24.3 | 861 ms | 41.1 ms | 128.0 | 0 | 93,152 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 188.2 | 188.6 | 49 ms | 5.3 ms | 1.0 | 1 | 5,645 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 335.2 | 168.5 | 56 ms | 5.9 ms | 2.0 | 2 | 10,055 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 585.7 | 147.1 | 61 ms | 6.8 ms | 4.0 | 4 | 17,570 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 810.7 | 101.7 | 131 ms | 9.8 ms | 8.0 | 0 | 24,322 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,242.1 | 78.0 | 243 ms | 12.8 ms | 16.0 | 0 | 37,264 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,726.5 | 54.0 | 453 ms | 18.5 ms | 32.0 | 0 | 51,776 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,317.7 | 36.2 | 822 ms | 27.6 ms | 64.0 | 0 | 69,504 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,861.7 | 22.2 | 1,568 ms | 45.1 ms | 128.0 | 0 | 85,759 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 181.8 | 182.5 | 81 ms | 5.5 ms | 1.0 | 1 | 5,454 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 322.6 | 162.2 | 84 ms | 6.2 ms | 2.0 | 2 | 9,676 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 551.2 | 139.7 | 88 ms | 7.2 ms | 3.9 | 4 | 16,536 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 766.0 | 96.1 | 236 ms | 10.4 ms | 8.0 | 0 | 22,979 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,539.7 | 48.0 | 795 ms | 20.8 ms | 32.0 | 0 | 46,176 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,020.8 | 31.2 | 1,577 ms | 32.1 ms | 64.0 | 0 | 60,608 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 170.5 | 171.7 | 148 ms | 5.8 ms | 1.0 | 1 | 5,115 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 297.2 | 150.0 | 165 ms | 6.7 ms | 2.0 | 2 | 8,914 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 499.9 | 127.1 | 166 ms | 7.9 ms | 4.0 | 4 | 14,994 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 697.8 | 87.3 | 458 ms | 11.5 ms | 8.0 | 0 | 20,928 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,001.8 | 62.6 | 836 ms | 16.0 ms | 16.0 | 0 | 30,048 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,275.0 | 39.5 | 1,598 ms | 25.3 ms | 32.0 | 0 | 38,240 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 152.2 | 154.3 | 295 ms | 6.5 ms | 1.0 | 1 | 4,566 | 30.0 s |
| Sustained decode | 123K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 4 | Valid | 430.3 | 107.7 | 587 ms | 9.3 ms | 4.0 | 0 | 12,908 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 593.7 | 74.2 | 947 ms | 13.5 ms | 8.0 | 0 | 17,808 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 798.2 | 49.7 | 1,800 ms | 20.1 ms | 16.0 | 0 | 23,936 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.