Speed test matrix
Ministral-3-8B-Instruct-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-mistralai-sglang-fa7784d140ee/bin/sglang" serve --model-path mistralai/Ministral-3-8B-Instruct-2512 --served-model-name mistral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context29.3K280 ms TTFT
16K context25.5K638 ms TTFT
32K context19.5K1,658 ms TTFT
64K context13.3K4,869 ms TTFT
128K context8,01316,110 ms TTFT
251K context4,55655,523 ms TTFT
256K context4,48857,495 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 121.425 ms TTFT | 235.725 ms TTFT | 447.347 ms TTFT | 809.546 ms TTFT | 1,251.062 ms TTFT | 1,789.173 ms TTFT | Over capacity | Over capacity |
| 16K | 111.725 ms TTFT | 217.440 ms TTFT | 412.642 ms TTFT | 715.949 ms TTFT | 999.854 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 96.225 ms TTFT | 188.542 ms TTFT | 358.945 ms TTFT | 571.452 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 75.126 ms TTFT | 148.044 ms TTFT | 284.749 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 52.132 ms TTFT | 103.151 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 32.846 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.3K | — | 280 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 25.5K | — | 638 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.5K | — | 1,658 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.3K | — | 4,869 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,013.0 | — | 16,110 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,556.0 | — | 55,523 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,488.0 | — | 57,495 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 121.4 | 121.7 | 25 ms | 8.2 ms | 1.0 | 0 | 3,640 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 235.7 | 118.5 | 25 ms | 8.4 ms | 2.0 | 0 | 7,070 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 447.3 | 113.0 | 47 ms | 8.9 ms | 4.0 | 0 | 13,416 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 809.5 | 102.4 | 46 ms | 9.8 ms | 8.0 | 0 | 24,280 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,251.0 | 79.4 | 62 ms | 12.6 ms | 16.0 | 0 | 37,519 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,789.1 | 57.1 | 73 ms | 17.5 ms | 32.0 | 0 | 53,664 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 111.7 | 111.9 | 25 ms | 8.9 ms | 1.0 | 0 | 3,350 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 217.4 | 109.2 | 40 ms | 9.2 ms | 2.0 | 0 | 6,520 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 412.6 | 104.1 | 42 ms | 9.6 ms | 4.0 | 0 | 12,376 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 715.9 | 90.1 | 49 ms | 11.1 ms | 8.0 | 0 | 21,472 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 999.8 | 62.9 | 54 ms | 15.9 ms | 16.0 | 0 | 29,984 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 96.2 | 96.3 | 25 ms | 10.4 ms | 1.0 | 0 | 2,885 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 188.5 | 94.5 | 42 ms | 10.6 ms | 2.0 | 0 | 5,654 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 358.9 | 90.4 | 45 ms | 11.1 ms | 4.0 | 0 | 10,768 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 571.4 | 71.7 | 52 ms | 13.9 ms | 8.0 | 0 | 17,136 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.1 | 75.2 | 26 ms | 13.3 ms | 1.0 | 0 | 2,254 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.0 | 74.1 | 44 ms | 13.5 ms | 2.0 | 0 | 4,437 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 284.7 | 71.5 | 49 ms | 14.0 ms | 4.0 | 0 | 8,540 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 52.1 | 52.2 | 32 ms | 19.2 ms | 1.0 | 0 | 1,564 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 103.1 | 51.6 | 51 ms | 19.4 ms | 2.0 | 0 | 3,094 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 32.8 | 32.8 | 46 ms | 30.4 ms | 1.0 | 0 | 985 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.