Speed test matrix
Ministral-3-14B-Base-2512 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-14B-Base-2512 --served-model-name mistral --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context19.1K430 ms TTFT
16K context18K905 ms TTFT
32K context15.6K2,080 ms TTFT
64K context11.9K5,399 ms TTFT
128K context8,10815,894 ms TTFT
251K context4,97350,784 ms TTFT
256K context4,97451,783 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 107.461 ms TTFT | 207.779 ms TTFT | 394.9112 ms TTFT | Looped | Looped | 1,864.8462 ms TTFT | Over capacity | Over capacity |
| 16K | 103.771 ms TTFT | 197.8105 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 32K | 97.0107 ms TTFT | 180.8147 ms TTFT | Looped | 502.5412 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 86.5140 ms TTFT | 155.7239 ms TTFT | 251.8407 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 71.0230 ms TTFT | 120.1433 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 52.7420 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.1K | — | 430 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18K | — | 905 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.6K | — | 2,080 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.9K | — | 5,399 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,108.0 | — | 15,894 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,973.0 | — | 50,784 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,974.0 | — | 51,783 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 107.4 | 107.6 | 61 ms | 9.3 ms | 1.0 | 0 | 3,223 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 207.7 | 104.0 | 79 ms | 9.6 ms | 2.0 | 0 | 6,230 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 394.9 | 99.0 | 112 ms | 10.1 ms | 4.0 | 0 | 11,848 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 1,864.8 | 58.9 | 462 ms | 17.0 ms | 32.0 | 0 | 55,938 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 103.7 | 103.8 | 71 ms | 9.6 ms | 1.0 | 0 | 3,112 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 197.8 | 99.0 | 105 ms | 10.1 ms | 2.0 | 0 | 5,934 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 97.0 | 97.1 | 107 ms | 10.3 ms | 1.0 | 0 | 2,911 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 180.8 | 90.5 | 147 ms | 11.0 ms | 2.0 | 0 | 5,424 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 502.5 | 63.1 | 412 ms | 15.9 ms | 8.0 | 0 | 15,072 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 86.5 | 86.5 | 140 ms | 11.6 ms | 1.0 | 0 | 2,595 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 155.7 | 78.0 | 239 ms | 12.8 ms | 2.0 | 0 | 4,670 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 251.8 | 63.3 | 407 ms | 15.8 ms | 4.0 | 0 | 7,552 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 71.0 | 71.0 | 230 ms | 14.1 ms | 1.0 | 0 | 2,129 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 120.1 | 60.3 | 433 ms | 16.6 ms | 2.0 | 0 | 3,604 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 52.7 | 52.8 | 420 ms | 19.0 ms | 1.0 | 0 | 1,582 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.