Speed test matrix
Trinity-Mini-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.21.1rc1.dev305+gc8414a827 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.21.1rc1.dev305+gc8414a827
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c8414a82712b
Run
- Served model
- trinity
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-arcee-ai-vllm-c8414a82712b/bin/vllm" serve arcee-ai/Trinity-Mini-NVFP4 --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend flashinfer_cutlass --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context36.9K222 ms TTFT
16K context41K397 ms TTFT
32K context36.3K892 ms TTFT
64K context29K2,227 ms TTFT
123K context21K5,909 ms TTFT
128K context20.5K6,285 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 172.335 ms TTFT | 314.351 ms TTFT | 549.782 ms TTFT | Looped | 1,232.9278 ms TTFT | 1,742.6485 ms TTFT | 2,415.1725 ms TTFT | 2,969.9995 ms TTFT |
| 16K | 169.754 ms TTFT | Looped | 536.8142 ms TTFT | 771.9258 ms TTFT | Looped | Looped | 2,251.31,012 ms TTFT | 2,725.82,334 ms TTFT |
| 32K | 164.386 ms TTFT | Looped | 515.0276 ms TTFT | Looped | 1,092.51,012 ms TTFT | 1,476.41,989 ms TTFT | 1,961.02,838 ms TTFT | Over capacity |
| 64K | 154.8167 ms TTFT | 272.9297 ms TTFT | 473.7553 ms TTFT | 664.41,106 ms TTFT | 954.02,224 ms TTFT | 1,235.44,499 ms TTFT | Over capacity | Over capacity |
| 123K | 138.9321 ms TTFT | 225.7640 ms TTFT | 400.11,265 ms TTFT | 564.92,403 ms TTFT | 764.84,940 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 36.9K | — | 222 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41K | — | 397 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.3K | — | 892 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29K | — | 2,227 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 21K | — | 5,909 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.5K | — | 6,285 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 172.3 | 172.7 | 35 ms | 5.8 ms | 1.0 | 1 | 5,168 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 314.3 | 157.8 | 51 ms | 6.3 ms | 2.0 | 2 | 9,427 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 549.7 | 138.4 | 82 ms | 7.2 ms | 4.0 | 4 | 16,489 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,232.9 | 77.7 | 278 ms | 12.9 ms | 16.0 | 0 | 36,992 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,742.6 | 55.0 | 485 ms | 18.2 ms | 32.0 | 0 | 52,256 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,415.1 | 38.2 | 725 ms | 26.2 ms | 64.0 | 0 | 72,448 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,969.9 | 23.6 | 995 ms | 42.4 ms | 128.0 | 0 | 89,088 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 169.7 | 170.2 | 54 ms | 5.9 ms | 1.0 | 1 | 5,090 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 536.8 | 135.6 | 142 ms | 7.4 ms | 4.0 | 4 | 16,102 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 771.9 | 96.8 | 258 ms | 10.3 ms | 8.0 | 0 | 23,160 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 2,251.3 | 35.8 | 1,012 ms | 27.9 ms | 64.0 | 0 | 67,520 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,725.8 | 21.5 | 2,334 ms | 46.5 ms | 128.0 | 0 | 81,663 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 164.3 | 165.0 | 86 ms | 6.1 ms | 1.0 | 1 | 4,927 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 515.0 | 132.7 | 276 ms | 7.5 ms | 4.0 | 4 | 15,448 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,092.5 | 68.9 | 1,012 ms | 14.5 ms | 16.0 | 0 | 32,768 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,476.4 | 46.9 | 1,989 ms | 21.3 ms | 32.0 | 0 | 44,288 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,961.0 | 31.2 | 2,838 ms | 32.0 ms | 64.0 | 0 | 58,816 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 154.8 | 156.1 | 167 ms | 6.4 ms | 1.0 | 1 | 4,643 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 272.9 | 138.1 | 297 ms | 7.2 ms | 2.0 | 2 | 8,185 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 473.7 | 118.9 | 553 ms | 8.4 ms | 4.0 | 0 | 14,208 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 664.4 | 83.6 | 1,106 ms | 12.0 ms | 8.0 | 0 | 19,928 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 954.0 | 60.3 | 2,224 ms | 16.6 ms | 16.0 | 0 | 28,624 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,235.4 | 39.4 | 4,499 ms | 25.4 ms | 32.0 | 0 | 37,056 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 138.9 | 141.4 | 321 ms | 7.1 ms | 1.0 | 1 | 4,167 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 225.7 | 113.0 | 640 ms | 8.8 ms | 2.0 | 0 | 6,771 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 400.1 | 100.6 | 1,265 ms | 9.9 ms | 4.0 | 0 | 12,005 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 564.9 | 71.2 | 2,403 ms | 14.0 ms | 8.0 | 0 | 16,944 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 764.8 | 48.7 | 4,940 ms | 20.6 ms | 16.0 | 0 | 22,944 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.