Speed test matrix
Mistral-Medium-3.5-128B on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Mistral-Medium-3.5-128B --served-model-name mistral --trust-remote-code --tp 2 --reasoning-parser mistral --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.853102 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context5,0631,624 ms TTFT
16K context4,9513,286 ms TTFT
32K context4,3237,487 ms TTFT
64K context3,36319,188 ms TTFT
128K context2,29856,075 ms TTFT
251K context1,442175,151 ms TTFT
256K context1,424180,931 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 45.5Requests queued72 ms TTFT | 59.2129 ms TTFT | 104.0176 ms TTFT | 176.9193 ms TTFT | 394.2253 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 44.6Requests queued97 ms TTFT | 58.3158 ms TTFT | 99.8190 ms TTFT | 192.9246 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 43.2Requests queued129 ms TTFT | 51.7189 ms TTFT | 101.4264 ms TTFT | 156.5410 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 40.6185 ms TTFT | 38.5260 ms TTFT | 75.7427 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 36.3261 ms TTFT | 33.9456 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 30.1Requests queued432 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,063.0 | — | 1,624 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,951.0 | — | 3,286 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,323.0 | — | 7,487 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,363.0 | — | 19,188 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,298.0 | — | 56,075 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,442.0 | — | 175,151 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,424.0 | — | 180,931 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 45.5 | 45.5 | 72 ms | 22.0 ms | 1.0 | 0 | 1,365 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 59.2 | 29.6 | 129 ms | 33.8 ms | 3.0 | 0 | 1,776 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 104.0 | 26.0 | 176 ms | 38.5 ms | 7.0 | 0 | 3,120 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 176.9 | 22.1 | 193 ms | 45.2 ms | 15.0 | 0 | 5,304 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 394.2 | 23.4 | 253 ms | 42.8 ms | 30.4 | 0 | 11,824 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Requests queued | 44.6 | 44.6 | 97 ms | 22.4 ms | 1.0 | 0 | 1,339 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 58.3 | 29.0 | 158 ms | 34.5 ms | 30.7 | 0 | 1,750 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 99.8 | 24.7 | 190 ms | 40.4 ms | 31.7 | 0 | 2,992 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 192.9 | 23.9 | 246 ms | 41.9 ms | 34.3 | 0 | 5,784 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Requests queued | 43.2 | 43.2 | 129 ms | 23.2 ms | 1.0 | 0 | 1,295 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 51.7 | 25.6 | 189 ms | 39.0 ms | 22.4 | 0 | 1,550 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 101.4 | 25.3 | 264 ms | 39.5 ms | 18.5 | 0 | 3,044 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 156.5 | 19.6 | 410 ms | 51.1 ms | 24.4 | 0 | 4,696 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 40.6 | 40.5 | 185 ms | 24.7 ms | 1.0 | 0 | 1,216 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 38.5 | 19.3 | 260 ms | 51.9 ms | 24.0 | 0 | 1,154 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 75.7 | 18.9 | 427 ms | 52.9 ms | 20.0 | 0 | 2,268 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 36.3 | 36.3 | 261 ms | 27.6 ms | 1.0 | 0 | 1,088 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 33.9 | 16.4 | 456 ms | 60.8 ms | 20.3 | 0 | 1,018 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 30.1 | 30.1 | 432 ms | 33.2 ms | 1.0 | 0 | 904 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.