Speed test matrix
Ministral-3-8B-Instruct-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.31.1rc1.dev21+g678ef3358 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.31.1rc1.dev21+g678ef3358
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
678ef33584da
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-mistralai-vllm-pr-678ef33584da/bin/vllm" serve mistralai/Ministral-3-8B-Instruct-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral --attention-backend FLASHINFER --linear-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context31.2K263 ms TTFT
16K context26.8K608 ms TTFT
32K context20.2K1,605 ms TTFT
64K context13.5K4,774 ms TTFT
128K context8,12215,897 ms TTFT
251K context4,59355,091 ms TTFT
256K context4,52157,074 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 125.923 ms TTFT | 248.828 ms TTFT | 347.344 ms TTFT | 672.964 ms TTFT | 1,219.390 ms TTFT | 1,918.2136 ms TTFT | Over capacity | Over capacity |
| 16K | 114.931 ms TTFT | 223.440 ms TTFT | 431.564 ms TTFT | 651.583 ms TTFT | 1,077.8129 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 97.944 ms TTFT | 169.958 ms TTFT | 258.781 ms TTFT | 561.7122 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 75.273 ms TTFT | 150.1102 ms TTFT | 293.8141 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 51.0126 ms TTFT | 97.3186 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 31.1229 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.2K | — | 263 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.8K | — | 608 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.2K | — | 1,605 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.5K | — | 4,774 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,122.0 | — | 15,897 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,593.0 | — | 55,091 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,521.0 | — | 57,074 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 125.9 | 127.6 | 23 ms | 7.8 ms | 1.0 | 1 | 3,776 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 248.8 | 127.9 | 28 ms | 7.8 ms | 2.0 | 2 | 7,464 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 347.3 | 119.8 | 44 ms | 8.3 ms | 4.0 | 0 | 10,419 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 672.9 | 103.6 | 64 ms | 9.6 ms | 8.0 | 0 | 20,181 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,219.3 | 84.0 | 90 ms | 11.9 ms | 16.0 | 0 | 36,573 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,918.2 | 61.6 | 136 ms | 16.2 ms | 32.0 | 0 | 57,527 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 114.9 | 115.1 | 31 ms | 8.7 ms | 1.0 | 0 | 2,976 | 25.9 s |
| Sustained decode | 16K | 2 | Valid | 223.4 | 115.1 | 40 ms | 8.7 ms | 2.0 | 0 | 5,440 | 24.3 s |
| Sustained decode | 16K | 4 | Valid | 431.5 | 109.0 | 64 ms | 9.2 ms | 4.0 | 0 | 12,944 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 651.5 | 91.1 | 83 ms | 11.0 ms | 8.0 | 0 | 19,544 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,077.8 | 69.7 | 129 ms | 14.3 ms | 16.0 | 0 | 32,321 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 97.9 | 98.0 | 44 ms | 10.2 ms | 1.0 | 0 | 2,935 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 169.9 | 98.3 | 58 ms | 10.2 ms | 2.0 | 0 | 5,097 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 258.7 | 96.1 | 81 ms | 10.4 ms | 4.0 | 0 | 7,716 | 29.8 s |
| Sustained decode | 32K | 8 | Valid | 561.7 | 73.9 | 122 ms | 13.5 ms | 8.0 | 0 | 16,850 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.2 | 75.3 | 73 ms | 13.3 ms | 1.0 | 0 | 2,257 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 150.1 | 75.1 | 102 ms | 13.3 ms | 2.0 | 0 | 4,502 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 293.8 | 73.7 | 141 ms | 13.6 ms | 4.0 | 0 | 8,813 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 51.0 | 51.0 | 126 ms | 19.6 ms | 1.0 | 0 | 1,528 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 97.3 | 48.7 | 186 ms | 20.5 ms | 2.0 | 0 | 2,920 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 31.1 | 31.1 | 229 ms | 32.1 ms | 1.0 | 0 | 933 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.