Speed test matrix
Ministral-3-14B-Base-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-mistralai-sglang-fa7784d140ee/bin/sglang" serve --model-path mistralai/Ministral-3-14B-Base-2512 --served-model-name mistral --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context13.1K626 ms TTFT
16K context12.1K1,347 ms TTFT
32K context10.3K3,155 ms TTFT
64K context7,9368,142 ms TTFT
128K context5,42623,789 ms TTFT
251K context3,38474,757 ms TTFT
256K context3,33877,288 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 50.128 ms TTFT | 100.243 ms TTFT | 196.749 ms TTFT | 374.552 ms TTFT | 649.464 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 47.928 ms TTFT | 96.045 ms TTFT | 188.051 ms TTFT | 345.956 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 44.331 ms TTFT | 88.648 ms TTFT | 173.756 ms TTFT | 300.764 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 38.436 ms TTFT | 76.855 ms TTFT | 150.665 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.346 ms TTFT | 60.769 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 21.662 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.1K | — | 626 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.1K | — | 1,347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.3K | — | 3,155 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,936.0 | — | 8,142 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,426.0 | — | 23,789 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,384.0 | — | 74,757 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,338.0 | — | 77,288 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 50.1 | 50.1 | 28 ms | 20.0 ms | 1.0 | 0 | 1,501 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 100.2 | 50.1 | 43 ms | 19.9 ms | 2.0 | 0 | 3,004 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 196.7 | 49.3 | 49 ms | 20.3 ms | 4.0 | 0 | 5,900 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 374.5 | 46.9 | 52 ms | 21.3 ms | 8.0 | 0 | 11,233 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 649.4 | 40.7 | 64 ms | 24.6 ms | 16.0 | 0 | 19,472 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 47.9 | 47.9 | 28 ms | 20.9 ms | 1.0 | 0 | 1,437 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 96.0 | 48.0 | 45 ms | 20.8 ms | 2.0 | 0 | 2,880 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 188.0 | 47.1 | 51 ms | 21.2 ms | 4.0 | 0 | 5,640 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 345.9 | 43.3 | 56 ms | 23.1 ms | 8.0 | 0 | 10,376 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 44.3 | 44.3 | 31 ms | 22.6 ms | 1.0 | 0 | 1,327 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 88.6 | 44.3 | 48 ms | 22.6 ms | 2.0 | 0 | 2,656 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 173.7 | 43.5 | 56 ms | 23.0 ms | 4.0 | 0 | 5,210 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 300.7 | 37.6 | 64 ms | 26.6 ms | 8.0 | 0 | 9,016 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 38.4 | 38.4 | 36 ms | 26.0 ms | 1.0 | 0 | 1,152 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 76.8 | 38.4 | 55 ms | 26.0 ms | 2.0 | 0 | 2,304 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 150.6 | 37.7 | 65 ms | 26.5 ms | 4.0 | 0 | 4,516 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.3 | 30.3 | 46 ms | 33.0 ms | 1.0 | 0 | 909 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 60.7 | 30.3 | 69 ms | 33.0 ms | 2.0 | 0 | 1,820 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 21.6 | 21.6 | 62 ms | 46.3 ms | 1.0 | 0 | 648 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.