Speed test matrix
Ministral-3-14B-Instruct-2512 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-14B-Instruct-2512 --served-model-name mistral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context26.4K310 ms TTFT
16K context24.5K662 ms TTFT
32K context20.1K1,606 ms TTFT
64K context14.5K4,463 ms TTFT
128K context9,17314,049 ms TTFT
251K context5,37047,032 ms TTFT
256K context5,32048,419 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 155.6Requests queued58 ms TTFT | 145.286 ms TTFT | 530.899 ms TTFT | 771.3145 ms TTFT | 1,510.4241 ms TTFT | 2,031.7437 ms TTFT | Over capacity | Over capacity |
| 16K | 147.876 ms TTFT | 92.1105 ms TTFT | 395.6151 ms TTFT | 794.7229 ms TTFT | 1,122.8422 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 134.6100 ms TTFT | 95.3149 ms TTFT | 271.2222 ms TTFT | 446.8399 ms TTFT | 496.0784 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 115.0137 ms TTFT | 59.4250 ms TTFT | 119.3396 ms TTFT | 168.3759 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 89.3226 ms TTFT | 35.0431 ms TTFT | 68.7765 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 62.5401 ms TTFT | 30.9782 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.4K | — | 310 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.5K | — | 662 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.1K | — | 1,606 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.5K | — | 4,463 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,173.0 | — | 14,049 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,370.0 | — | 47,032 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,320.0 | — | 48,419 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 155.6 | 156.8 | 58 ms | 6.4 ms | 1.0 | 1 | 4,669 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 145.2 | 71.1 | 86 ms | 14.1 ms | 2.9 | 0 | 4,356 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 530.8 | 136.4 | 99 ms | 7.3 ms | 4.5 | 4 | 15,922 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 771.3 | 97.2 | 145 ms | 10.3 ms | 12.0 | 0 | 23,136 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,510.4 | 92.0 | 241 ms | 10.9 ms | 16.5 | 0 | 45,312 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,031.7 | 62.3 | 437 ms | 16.1 ms | 37.3 | 0 | 60,944 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 147.8 | 148.8 | 76 ms | 6.7 ms | 1.0 | 1 | 4,435 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 92.1 | 46.4 | 105 ms | 21.6 ms | 34.0 | 0 | 2,762 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 395.6 | 94.8 | 151 ms | 10.5 ms | 5.5 | 0 | 11,868 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 794.7 | 96.6 | 229 ms | 10.3 ms | 8.3 | 0 | 23,841 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,122.8 | 68.0 | 422 ms | 14.7 ms | 17.6 | 0 | 33,680 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 134.6 | 135.7 | 100 ms | 7.4 ms | 1.0 | 1 | 4,037 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 95.3 | 47.7 | 149 ms | 20.9 ms | 18.0 | 0 | 2,860 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 271.2 | 67.8 | 222 ms | 14.8 ms | 5.9 | 0 | 8,136 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 446.8 | 55.6 | 399 ms | 18.0 ms | 11.2 | 0 | 13,400 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 496.0 | 31.3 | 784 ms | 31.9 ms | 24.0 | 0 | 14,879 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 115.0 | 115.1 | 137 ms | 8.7 ms | 1.0 | 0 | 3,451 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 59.4 | 29.7 | 250 ms | 33.7 ms | 24.9 | 0 | 1,782 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 119.3 | 29.9 | 396 ms | 33.4 ms | 22.0 | 0 | 3,576 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 168.3 | 21.2 | 759 ms | 47.2 ms | 30.0 | 0 | 5,040 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 89.3 | 89.4 | 226 ms | 11.2 ms | 1.0 | 0 | 2,680 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 35.0 | 17.5 | 431 ms | 57.0 ms | 16.0 | 0 | 1,048 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 68.7 | 17.3 | 765 ms | 58.0 ms | 19.5 | 0 | 2,060 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 62.5 | 62.5 | 401 ms | 16.0 ms | 1.0 | 0 | 1,875 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 30.9 | 15.5 | 782 ms | 64.4 ms | 20.0 | 0 | 928 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.