Speed test matrix
Ministral-3-14B-Base-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-mistralai-vllm-73a5831127a9/bin/vllm" serve mistralai/Ministral-3-14B-Base-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context13.3K618 ms TTFT
16K context12.1K1,340 ms TTFT
32K context10.3K3,142 ms TTFT
64K context7,9548,125 ms TTFT
128K context5,43723,746 ms TTFT
251K context3,38774,712 ms TTFT
256K context3,34477,159 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 51.133 ms TTFT | 102.665 ms TTFT | 201.768 ms TTFT | 380.982 ms TTFT | 682.1120 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 48.940 ms TTFT | 98.352 ms TTFT | 193.074 ms TTFT | 352.9103 ms TTFT | 603.0147 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 44.953 ms TTFT | 90.178 ms TTFT | 177.8122 ms TTFT | 307.8143 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 38.582 ms TTFT | 77.2115 ms TTFT | 153.6162 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.0134 ms TTFT | 58.4195 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 20.7237 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.3K | — | 618 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.1K | — | 1,340 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.3K | — | 3,142 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,954.0 | — | 8,125 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,437.0 | — | 23,746 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,387.0 | — | 74,712 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,344.0 | — | 77,159 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 51.1 | 51.1 | 33 ms | 19.6 ms | 1.0 | 0 | 1,532 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 102.6 | 51.4 | 65 ms | 19.5 ms | 2.0 | 0 | 3,078 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 201.7 | 50.5 | 68 ms | 19.8 ms | 4.0 | 0 | 6,048 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 380.9 | 47.8 | 82 ms | 20.9 ms | 8.0 | 0 | 11,424 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 682.1 | 42.9 | 120 ms | 23.3 ms | 16.0 | 0 | 20,448 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 48.9 | 48.9 | 40 ms | 20.5 ms | 1.0 | 0 | 1,466 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 98.3 | 49.2 | 52 ms | 20.3 ms | 2.0 | 0 | 2,948 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 193.0 | 48.3 | 74 ms | 20.7 ms | 4.0 | 0 | 5,788 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 352.9 | 44.3 | 103 ms | 22.6 ms | 8.0 | 0 | 10,588 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 603.0 | 37.9 | 147 ms | 26.4 ms | 16.0 | 0 | 18,080 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 44.9 | 44.9 | 53 ms | 22.3 ms | 1.0 | 0 | 1,345 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 90.1 | 45.1 | 78 ms | 22.2 ms | 2.0 | 0 | 2,702 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 177.8 | 44.5 | 122 ms | 22.5 ms | 4.0 | 0 | 5,332 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 307.8 | 38.6 | 143 ms | 25.9 ms | 8.0 | 0 | 9,232 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 38.5 | 38.5 | 82 ms | 26.0 ms | 1.0 | 0 | 1,155 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 77.2 | 38.6 | 115 ms | 25.9 ms | 2.0 | 0 | 2,314 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 153.6 | 38.4 | 162 ms | 26.0 ms | 4.0 | 0 | 4,608 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.0 | 30.0 | 134 ms | 33.4 ms | 1.0 | 0 | 898 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 58.4 | 29.2 | 195 ms | 34.2 ms | 2.0 | 0 | 1,752 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 20.7 | 20.7 | 237 ms | 48.3 ms | 1.0 | 0 | 621 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.