Speed test matrix
Ministral-3-14B-Instruct-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-mistralai-sglang-fa7784d140ee/bin/sglang" serve --model-path mistralai/Ministral-3-14B-Instruct-2512 --served-model-name mistral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context20.6K398 ms TTFT
16K context18.2K892 ms TTFT
32K context14.4K2,248 ms TTFT
64K context10.2K6,335 ms TTFT
128K context6,40020,168 ms TTFT
251K context3,73267,785 ms TTFT
256K context3,68470,039 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 81.827 ms TTFT | 160.845 ms TTFT | 310.852 ms TTFT | 577.451 ms TTFT | 937.364 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 76.527 ms TTFT | 150.544 ms TTFT | 290.348 ms TTFT | 515.451 ms TTFT | 759.361 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 67.628 ms TTFT | 133.346 ms TTFT | 258.250 ms TTFT | 422.452 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 54.929 ms TTFT | 108.549 ms TTFT | 210.953 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 39.837 ms TTFT | 78.961 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 26.154 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.6K | — | 398 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.2K | — | 892 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.4K | — | 2,248 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.2K | — | 6,335 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,400.0 | — | 20,168 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,732.0 | — | 67,785 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,684.0 | — | 70,039 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 81.8 | 81.9 | 27 ms | 12.2 ms | 1.0 | 0 | 2,454 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 160.8 | 80.7 | 45 ms | 12.4 ms | 2.0 | 0 | 4,824 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 310.8 | 78.2 | 52 ms | 12.8 ms | 4.0 | 0 | 9,320 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 577.4 | 72.6 | 51 ms | 13.8 ms | 8.0 | 0 | 17,327 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 937.3 | 59.1 | 64 ms | 16.9 ms | 16.0 | 0 | 28,112 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 76.5 | 76.6 | 27 ms | 13.1 ms | 1.0 | 0 | 2,295 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 150.5 | 75.4 | 44 ms | 13.3 ms | 2.0 | 0 | 4,512 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 290.3 | 72.9 | 48 ms | 13.7 ms | 4.0 | 0 | 8,706 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 515.4 | 64.6 | 51 ms | 15.5 ms | 8.0 | 0 | 15,456 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 759.3 | 47.7 | 61 ms | 21.0 ms | 16.0 | 0 | 22,768 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 67.6 | 67.7 | 28 ms | 14.8 ms | 1.0 | 0 | 2,027 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 133.3 | 66.8 | 46 ms | 15.0 ms | 2.0 | 0 | 3,996 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 258.2 | 64.8 | 50 ms | 15.4 ms | 4.0 | 0 | 7,746 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 422.4 | 52.9 | 52 ms | 18.9 ms | 8.0 | 0 | 12,672 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 54.9 | 54.9 | 29 ms | 18.2 ms | 1.0 | 0 | 1,646 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 108.5 | 54.3 | 49 ms | 18.4 ms | 2.0 | 0 | 3,254 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 210.9 | 52.8 | 53 ms | 18.9 ms | 4.0 | 0 | 6,324 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 39.8 | 39.8 | 37 ms | 25.1 ms | 1.0 | 0 | 1,194 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 78.9 | 39.5 | 61 ms | 25.3 ms | 2.0 | 0 | 2,366 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 26.1 | 26.1 | 54 ms | 38.4 ms | 1.0 | 0 | 781 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.