Speed test matrix
Qwen3.6-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.858466 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context9,760835 ms TTFT
16K context9,7811,649 ms TTFT
32K context9,3753,424 ms TTFT
64K context8,4977,539 ms TTFT
128K context7,24017,677 ms TTFT
251K context5,56245,094 ms TTFT
256K context5,53446,221 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 62.8Requests queued85 ms TTFT | 96.9150 ms TTFT | 219.81,648 ms TTFT | 398.8209 ms TTFT | 708.5355 ms TTFT | 1,189.5Requests queued664 ms TTFT | Over capacity | Over capacity |
| 16K | 62.3107 ms TTFT | 82.7180 ms TTFT | 194.2210 ms TTFT | 360.5340 ms TTFT | 617.4714 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 61.2164 ms TTFT | 75.1267 ms TTFT | 149.0424 ms TTFT | 301.3831 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 59.3314 ms TTFT | 78.4511 ms TTFT | 125.0970 ms TTFT | 229.01,808 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.9521 ms TTFT | 53.91,043 ms TTFT | 85.91,857 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 50.21,023 ms TTFT | 38.61,965 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,760.0 | — | 835 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,781.0 | — | 1,649 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,375.0 | — | 3,424 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,497.0 | — | 7,539 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,240.0 | — | 17,677 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,562.0 | — | 45,094 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,534.0 | — | 46,221 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 62.8 | 62.8 | 85 ms | 15.9 ms | 1.0 | 0 | 1,883 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 96.9 | 48.4 | 150 ms | 20.6 ms | 3.0 | 0 | 2,906 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 219.8 | 53.9 | 1,648 ms | 18.6 ms | 6.2 | 0 | 6,592 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 398.8 | 49.7 | 209 ms | 20.1 ms | 12.2 | 0 | 11,960 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 708.5 | 44.1 | 355 ms | 22.7 ms | 24.4 | 0 | 21,248 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,189.5 | 36.3 | 664 ms | 27.5 ms | 36.8 | 0 | 35,688 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 62.3 | 62.3 | 107 ms | 16.1 ms | 1.0 | 0 | 1,868 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 82.7 | 40.9 | 180 ms | 24.5 ms | 25.3 | 0 | 2,482 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 194.2 | 48.6 | 210 ms | 20.6 ms | 10.0 | 0 | 5,824 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 360.5 | 45.0 | 340 ms | 22.2 ms | 12.9 | 0 | 10,808 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 617.4 | 38.6 | 714 ms | 25.9 ms | 25.7 | 0 | 18,512 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 61.2 | 61.2 | 164 ms | 16.3 ms | 1.0 | 0 | 1,837 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 75.1 | 37.5 | 267 ms | 26.6 ms | 24.4 | 0 | 2,252 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 149.0 | 37.3 | 424 ms | 26.8 ms | 22.0 | 0 | 4,468 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 301.3 | 37.9 | 831 ms | 26.4 ms | 14.0 | 0 | 9,032 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 59.3 | 59.3 | 314 ms | 16.9 ms | 1.0 | 0 | 1,778 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 78.4 | 39.3 | 511 ms | 25.5 ms | 13.8 | 0 | 2,350 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 125.0 | 31.5 | 970 ms | 31.7 ms | 14.0 | 0 | 3,748 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 229.0 | 28.8 | 1,808 ms | 34.7 ms | 14.0 | 0 | 6,864 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.9 | 55.8 | 521 ms | 17.9 ms | 1.0 | 0 | 1,676 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 53.9 | 26.6 | 1,043 ms | 37.6 ms | 14.9 | 0 | 1,616 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 85.9 | 21.8 | 1,857 ms | 45.8 ms | 18.0 | 0 | 2,576 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 50.2 | 50.2 | 1,023 ms | 19.9 ms | 1.0 | 0 | 1,505 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 38.6 | 19.5 | 1,965 ms | 51.2 ms | 16.0 | 0 | 1,158 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.