Speed test matrix
Qwen3.6-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.869044 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context14.2K572 ms TTFT
16K context14.1K1,141 ms TTFT
32K context13.3K2,416 ms TTFT
64K context11.6K5,536 ms TTFT
128K context9,17413,950 ms TTFT
251K context6,51638,490 ms TTFT
256K context6,47439,511 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 169.893 ms TTFT | 262.1123 ms TTFT | 588.5864 ms TTFT | 965.4197 ms TTFT | 1,404.9346 ms TTFT | 2,389.1Requests queued813 ms TTFT | Over capacity | Over capacity |
| 16K | 176.0119 ms TTFT | 287.8163 ms TTFT | 576.8171 ms TTFT | 882.5347 ms TTFT | 1,652.1655 ms TTFT | 2,143.3Requests queued1,232 ms TTFT | Over capacity | Over capacity |
| 32K | 159.1173 ms TTFT | 279.3239 ms TTFT | 506.2327 ms TTFT | 686.8849 ms TTFT | 1,288.71,652 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 152.7313 ms TTFT | 214.9529 ms TTFT | 512.4732 ms TTFT | 520.41,859 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 143.5556 ms TTFT | 179.61,030 ms TTFT | 425.41,888 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 128.61,001 ms TTFT | 217.31,938 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.2K | — | 572 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.1K | — | 1,141 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.3K | — | 2,416 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.6K | — | 5,536 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,174.0 | — | 13,950 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,516.0 | — | 38,490 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,474.0 | — | 39,511 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 169.8 | 173.1 | 93 ms | 5.8 ms | 1.0 | 1 | 5,091 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 262.1 | 149.7 | 123 ms | 6.8 ms | 2.8 | 2 | 7,862 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 588.5 | 153.4 | 864 ms | 6.5 ms | 5.3 | 4 | 17,647 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 965.4 | 124.5 | 197 ms | 8.0 ms | 10.7 | 7 | 28,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,404.9 | 89.8 | 346 ms | 11.1 ms | 23.0 | 0 | 42,138 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 2,389.1 | 77.0 | 813 ms | 13.0 ms | 30.8 | 31 | 71,675 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 176.0 | 180.0 | 119 ms | 5.6 ms | 1.0 | 1 | 5,279 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 287.8 | 156.2 | 163 ms | 6.5 ms | 7.0 | 2 | 8,635 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 576.8 | 149.4 | 171 ms | 6.7 ms | 5.2 | 4 | 17,296 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 882.5 | 110.5 | 347 ms | 9.1 ms | 11.1 | 0 | 26,471 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,652.1 | 104.8 | 655 ms | 9.5 ms | 16.0 | 0 | 49,532 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 2,143.3 | 69.1 | 1,232 ms | 14.5 ms | 30.8 | 31 | 64,283 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 159.1 | 162.5 | 173 ms | 6.2 ms | 1.0 | 1 | 4,772 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 279.3 | 160.9 | 239 ms | 6.4 ms | 7.4 | 2 | 8,376 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 506.2 | 141.5 | 327 ms | 7.1 ms | 5.4 | 4 | 15,186 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 686.8 | 86.7 | 849 ms | 11.5 ms | 12.0 | 0 | 20,593 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,288.7 | 80.8 | 1,652 ms | 12.4 ms | 18.7 | 0 | 38,659 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 152.7 | 161.4 | 313 ms | 6.2 ms | 1.0 | 1 | 4,579 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 214.9 | 103.6 | 529 ms | 9.7 ms | 9.5 | 0 | 6,447 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 512.4 | 139.3 | 732 ms | 7.2 ms | 4.0 | 4 | 15,373 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 520.4 | 68.7 | 1,859 ms | 14.6 ms | 12.0 | 0 | 15,609 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 143.5 | 152.6 | 556 ms | 6.6 ms | 1.0 | 1 | 4,304 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 179.6 | 89.2 | 1,030 ms | 11.2 ms | 8.2 | 0 | 5,388 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 425.4 | 106.3 | 1,888 ms | 9.4 ms | 4.2 | 0 | 12,756 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 128.6 | 141.9 | 1,001 ms | 7.1 ms | 1.0 | 1 | 3,856 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 217.3 | 111.2 | 1,938 ms | 9.0 ms | 2.0 | 0 | 6,519 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.