Speed test matrix
Ministral-3-14B-Instruct-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-mistralai-vllm-73a5831127a9/bin/vllm" serve mistralai/Ministral-3-14B-Instruct-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context15.5K527 ms TTFT
16K context16.1K1,010 ms TTFT
32K context13.8K2,339 ms TTFT
64K context10.1K6,373 ms TTFT
128K context6,42420,099 ms TTFT
251K context3,75267,447 ms TTFT
256K context3,69969,766 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 84.9173 ms TTFT | 167.3322 ms TTFT | 260.9603 ms TTFT | 564.21,055 ms TTFT | 964.92,285 ms TTFT | 1,496.54,218 ms TTFT | Over capacity | Over capacity |
| 16K | 78.6181 ms TTFT | 156.2332 ms TTFT | 262.2619 ms TTFT | 514.11,035 ms TTFT | 812.22,462 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 69.0195 ms TTFT | 137.6340 ms TTFT | 268.1645 ms TTFT | 418.41,254 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 55.1220 ms TTFT | 107.6351 ms TTFT | 216.2698 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 39.1276 ms TTFT | 75.6448 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 24.8386 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.5K | — | 527 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16.1K | — | 1,010 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.8K | — | 2,339 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.1K | — | 6,373 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,424.0 | — | 20,099 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,752.0 | — | 67,447 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,699.0 | — | 69,766 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 84.9 | 85.0 | 173 ms | 11.8 ms | 1.0 | 0 | 1,618 | 19.1 s |
| Sustained decode | 8K | 2 | Valid | 167.3 | 84.0 | 322 ms | 11.9 ms | 2.0 | 0 | 5,018 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 260.9 | 82.1 | 603 ms | 12.2 ms | 4.0 | 0 | 7,826 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 564.2 | 74.1 | 1,055 ms | 13.5 ms | 8.0 | 0 | 16,925 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 964.9 | 62.4 | 2,285 ms | 16.0 ms | 16.0 | 0 | 28,936 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,496.5 | 48.5 | 4,218 ms | 20.6 ms | 32.0 | 0 | 44,880 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 78.6 | 78.7 | 181 ms | 12.7 ms | 1.0 | 0 | 2,065 | 26.3 s |
| Sustained decode | 16K | 2 | Valid | 156.2 | 78.4 | 332 ms | 12.8 ms | 2.0 | 0 | 4,684 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 262.2 | 76.4 | 619 ms | 13.1 ms | 4.0 | 0 | 7,865 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 514.1 | 66.3 | 1,035 ms | 15.1 ms | 8.0 | 0 | 15,417 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 812.2 | 52.6 | 2,462 ms | 19.0 ms | 16.0 | 0 | 24,353 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 69.0 | 69.0 | 195 ms | 14.5 ms | 1.0 | 0 | 2,069 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 137.6 | 68.9 | 340 ms | 14.5 ms | 2.0 | 0 | 4,126 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 268.1 | 67.4 | 645 ms | 14.8 ms | 4.0 | 0 | 8,040 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 418.4 | 54.8 | 1,254 ms | 18.3 ms | 8.0 | 0 | 12,551 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.1 | 55.1 | 220 ms | 18.1 ms | 1.0 | 0 | 1,652 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 107.6 | 55.0 | 351 ms | 18.2 ms | 2.0 | 0 | 3,227 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 216.2 | 54.2 | 698 ms | 18.4 ms | 4.0 | 0 | 6,484 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 39.1 | 39.1 | 276 ms | 25.6 ms | 1.0 | 0 | 1,172 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 75.6 | 37.8 | 448 ms | 26.4 ms | 2.0 | 0 | 2,266 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 24.8 | 24.8 | 386 ms | 40.3 ms | 1.0 | 0 | 744 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.