Speed test matrix
Ministral-3-8B-Reasoning-2512 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-8B-Reasoning-2512 --served-model-name mistral --trust-remote-code --tp 1 --reasoning-parser mistral --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context28.7K286 ms TTFT
16K context26.3K617 ms TTFT
32K context22.1K1,465 ms TTFT
64K context16.2K3,994 ms TTFT
128K context10.5K12,293 ms TTFT
251K context6,25940,351 ms TTFT
256K context6,17941,687 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 156.153 ms TTFT | 190.380 ms TTFT | 566.393 ms TTFT | 842.9143 ms TTFT | 1,601.7246 ms TTFT | 2,313.9440 ms TTFT | Over capacity | Over capacity |
| 16K | 149.572 ms TTFT | 132.1106 ms TTFT | 468.8137 ms TTFT | 850.4230 ms TTFT | 1,288.4424 ms TTFT | 1,237.9794 ms TTFT | Over capacity | Over capacity |
| 32K | 137.795 ms TTFT | 77.7156 ms TTFT | 138.6231 ms TTFT | 447.2404 ms TTFT | 544.6790 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 120.0135 ms TTFT | 65.9252 ms TTFT | 130.7407 ms TTFT | 231.0770 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 95.5232 ms TTFT | 41.9437 ms TTFT | 78.0766 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 68.2407 ms TTFT | 34.5814 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 28.7K | — | 286 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.3K | — | 617 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.1K | — | 1,465 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.2K | — | 3,994 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.5K | — | 12,293 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,259.0 | — | 40,351 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,179.0 | — | 41,687 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 156.1 | 156.9 | 53 ms | 6.4 ms | 1.0 | 1 | 4,682 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 190.3 | 91.5 | 80 ms | 10.9 ms | 2.7 | 0 | 5,708 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 566.3 | 141.9 | 93 ms | 7.1 ms | 4.1 | 4 | 16,988 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 842.9 | 105.8 | 143 ms | 9.5 ms | 11.3 | 0 | 25,288 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,601.7 | 100.1 | 246 ms | 10.0 ms | 16.0 | 0 | 48,048 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,313.9 | 70.6 | 440 ms | 14.2 ms | 34.1 | 0 | 69,408 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 149.5 | 150.4 | 72 ms | 6.6 ms | 1.0 | 1 | 4,485 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 132.1 | 64.5 | 106 ms | 15.5 ms | 28.7 | 0 | 3,962 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 468.8 | 115.0 | 137 ms | 8.7 ms | 4.7 | 0 | 14,064 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 850.4 | 107.0 | 230 ms | 9.3 ms | 8.0 | 0 | 25,512 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,288.4 | 79.3 | 424 ms | 12.6 ms | 16.0 | 0 | 38,640 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,237.9 | 39.0 | 794 ms | 25.7 ms | 46.4 | 0 | 37,120 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 137.7 | 138.7 | 95 ms | 7.2 ms | 1.0 | 1 | 4,130 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 77.7 | 39.0 | 156 ms | 25.6 ms | 34.0 | 0 | 2,330 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 138.6 | 34.8 | 231 ms | 28.7 ms | 38.0 | 0 | 4,156 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 447.2 | 56.1 | 404 ms | 17.8 ms | 13.3 | 0 | 13,416 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 544.6 | 34.3 | 790 ms | 29.2 ms | 26.1 | 0 | 16,336 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 120.0 | 120.1 | 135 ms | 8.3 ms | 1.0 | 0 | 3,600 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 65.9 | 32.9 | 252 ms | 30.4 ms | 23.4 | 0 | 1,978 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 130.7 | 32.8 | 407 ms | 30.5 ms | 22.0 | 0 | 3,920 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 231.0 | 28.3 | 770 ms | 35.3 ms | 21.7 | 0 | 6,928 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 95.5 | 95.5 | 232 ms | 10.5 ms | 1.0 | 0 | 2,864 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 41.9 | 20.9 | 437 ms | 47.9 ms | 15.9 | 0 | 1,256 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 78.0 | 19.6 | 766 ms | 51.1 ms | 18.0 | 0 | 2,336 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 68.2 | 68.3 | 407 ms | 14.6 ms | 1.0 | 0 | 2,046 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 34.5 | 17.2 | 814 ms | 58.0 ms | 18.0 | 0 | 1,034 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.