Speed test matrix
Ministral-3-14B-Reasoning-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-mistralai-vllm-73a5831127a9/bin/vllm" serve mistralai/Ministral-3-14B-Reasoning-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context9,021909 ms TTFT
16K context9,9771,630 ms TTFT
32K context9,4153,440 ms TTFT
64K context7,6658,431 ms TTFT
128K context5,36624,061 ms TTFT
251K context3,37175,068 ms TTFT
256K context3,33177,472 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 50.9323 ms TTFT | 102.0610 ms TTFT | 197.61,168 ms TTFT | 373.42,242 ms TTFT | 678.14,520 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 48.7 | 97.6618 ms TTFT | 191.81,194 ms TTFT | 351.42,072 ms TTFT | 599.24,816 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 44.7342 ms TTFT | 89.6644 ms TTFT | 176.81,211 ms TTFT | 306.42,049 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 38.3370 ms TTFT | 76.7680 ms TTFT | 152.81,244 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 29.7429 ms TTFT | 58.3736 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 20.7541 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,021.0 | — | 909 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,977.0 | — | 1,630 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,415.0 | — | 3,440 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,665.0 | — | 8,431 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,366.0 | — | 24,061 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,371.0 | — | 75,068 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,331.0 | — | 77,472 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 50.9 | 50.9 | 323 ms | 19.6 ms | 1.0 | 0 | 1,526 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 102.0 | 51.1 | 610 ms | 19.6 ms | 2.0 | 0 | 3,058 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 197.6 | 50.3 | 1,168 ms | 19.9 ms | 4.0 | 0 | 5,929 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 373.4 | 47.6 | 2,242 ms | 21.0 ms | 8.0 | 0 | 11,198 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 678.1 | 43.4 | 4,520 ms | 23.0 ms | 16.0 | 0 | 20,335 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 48.7 | — | — | — | 1.0 | 0 | 1,379 | 28.3 s |
| Sustained decode | 16K | 2 | Valid | 97.6 | 49.0 | 618 ms | 20.4 ms | 2.0 | 0 | 2,928 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 191.8 | 48.2 | 1,194 ms | 20.8 ms | 4.0 | 0 | 5,752 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 351.4 | 44.2 | 2,072 ms | 22.6 ms | 8.0 | 0 | 10,536 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 599.2 | 38.3 | 4,816 ms | 26.1 ms | 16.0 | 0 | 17,968 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 44.7 | 44.7 | 342 ms | 22.4 ms | 1.0 | 0 | 1,339 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 89.6 | 44.9 | 644 ms | 22.3 ms | 2.0 | 0 | 2,690 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 176.8 | 44.3 | 1,211 ms | 22.6 ms | 4.0 | 0 | 5,305 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 306.4 | 38.6 | 2,049 ms | 25.9 ms | 8.0 | 0 | 9,192 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 38.3 | 38.3 | 370 ms | 26.1 ms | 1.0 | 0 | 1,149 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 76.7 | 38.5 | 680 ms | 26.0 ms | 2.0 | 0 | 2,301 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 152.8 | 38.4 | 1,244 ms | 26.1 ms | 4.0 | 0 | 4,584 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 29.7 | 29.7 | 429 ms | 33.7 ms | 1.0 | 0 | 891 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 58.3 | 29.2 | 736 ms | 34.2 ms | 2.0 | 0 | 1,750 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 20.7 | 20.6 | 541 ms | 48.4 ms | 1.0 | 0 | 619 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.