Speed test matrix
Ministral-3-8B-Instruct-2512 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-8B-Instruct-2512 --served-model-name mistral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context37.9K216 ms TTFT
16K context35.4K459 ms TTFT
32K context27.7K1,168 ms TTFT
64K context19.1K3,377 ms TTFT
128K context11.6K11,099 ms TTFT
251K context6,62738,110 ms TTFT
256K context6,54339,369 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 211.055 ms TTFT | 293.973 ms TTFT | 661.492 ms TTFT | 1,157.1141 ms TTFT | 1,392.0247 ms TTFT | 2,523.4454 ms TTFT | 2,269.9861 ms TTFT | Over capacity |
| 16K | 199.081 ms TTFT | 84.8109 ms TTFT | 152.8140 ms TTFT | 819.8226 ms TTFT | 1,485.4423 ms TTFT | 1,641.9807 ms TTFT | Over capacity | Over capacity |
| 32K | 179.098 ms TTFT | 81.0150 ms TTFT | 239.0225 ms TTFT | 539.0406 ms TTFT | 598.3779 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 149.4147 ms TTFT | 73.7251 ms TTFT | 141.9389 ms TTFT | 299.7765 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 113.4225 ms TTFT | 69.5425 ms TTFT | 83.3762 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 77.7405 ms TTFT | 37.7783 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.9K | — | 216 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 35.4K | — | 459 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.7K | — | 1,168 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.1K | — | 3,377 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.6K | — | 11,099 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,627.0 | — | 38,110 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,543.0 | — | 39,369 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 211.0 | 211.7 | 55 ms | 4.7 ms | 1.0 | 1 | 6,329 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 293.9 | 167.0 | 73 ms | 6.3 ms | 2.4 | 2 | 8,818 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 661.4 | 173.3 | 92 ms | 5.8 ms | 5.0 | 4 | 19,841 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,157.1 | 153.5 | 141 ms | 6.6 ms | 9.6 | 8 | 34,710 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,392.0 | 88.2 | 247 ms | 11.3 ms | 23.5 | 0 | 41,760 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,523.4 | 77.3 | 454 ms | 12.9 ms | 36.3 | 0 | 75,712 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,269.9 | 35.8 | 861 ms | 27.9 ms | 96.0 | 0 | 68,096 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 199.0 | 199.9 | 81 ms | 5.0 ms | 1.0 | 1 | 5,969 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.8 | 42.7 | 109 ms | 23.4 ms | 66.0 | 0 | 2,542 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 152.8 | 38.0 | 140 ms | 26.3 ms | 67.9 | 0 | 4,584 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 819.8 | 101.7 | 226 ms | 9.8 ms | 11.3 | 0 | 24,592 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,485.4 | 89.7 | 423 ms | 11.2 ms | 16.0 | 0 | 44,561 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,641.9 | 50.6 | 807 ms | 19.7 ms | 40.0 | 0 | 49,248 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 179.0 | 180.1 | 98 ms | 5.6 ms | 1.0 | 1 | 5,371 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 81.0 | 40.6 | 150 ms | 24.6 ms | 34.0 | 0 | 2,428 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 239.0 | 57.1 | 225 ms | 17.5 ms | 23.1 | 0 | 7,168 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 539.0 | 67.0 | 406 ms | 14.9 ms | 11.7 | 0 | 16,168 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 598.3 | 37.8 | 779 ms | 26.5 ms | 24.0 | 0 | 17,936 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 149.4 | 150.7 | 147 ms | 6.6 ms | 1.0 | 1 | 4,480 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 73.7 | 36.7 | 251 ms | 27.2 ms | 18.4 | 0 | 2,210 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 141.9 | 35.6 | 389 ms | 28.1 ms | 22.0 | 0 | 4,256 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 299.7 | 36.9 | 765 ms | 27.1 ms | 14.0 | 0 | 8,984 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 113.4 | 113.5 | 225 ms | 8.8 ms | 1.0 | 0 | 3,402 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 69.5 | 33.5 | 425 ms | 29.9 ms | 14.1 | 0 | 2,084 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 83.3 | 20.9 | 762 ms | 47.9 ms | 14.9 | 0 | 2,496 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 77.7 | 77.7 | 405 ms | 12.9 ms | 1.0 | 0 | 2,331 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 37.7 | 18.9 | 783 ms | 52.8 ms | 16.0 | 0 | 1,132 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.