Speed test matrix
Qwen3.6-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.850275 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-scheduler-strategy extra_buffer --attention-backend fa3 --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.6-27B-DFlash --speculative-num-draft-tokens 16
Prefill
prompt tok/s by context length8K context9,796832 ms TTFT
16K context9,6661,668 ms TTFT
32K context9,3193,445 ms TTFT
64K context8,4547,578 ms TTFT
128K context7,20017,773 ms TTFT
251K context5,53745,292 ms TTFT
256K context5,50846,441 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 126.292 ms TTFT | 258.7130 ms TTFT | 358.81,658 ms TTFT | 655.24,282 ms TTFT | 643.5Requests queued15,868 ms TTFT | 698.2Requests queued29,828 ms TTFT | Over capacity | Over capacity |
| 16K | 141.1115 ms TTFT | 223.8478 ms TTFT | 433.5233 ms TTFT | 631.2348 ms TTFT | 676.0Requests queued27,483 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 125.6170 ms TTFT | 240.0246 ms TTFT | 372.7412 ms TTFT | 581.51,932 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 101.7296 ms TTFT | 221.5558 ms TTFT | 385.2990 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 119.3536 ms TTFT | 216.51,149 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 108.41,006 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,796.0 | — | 832 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,666.0 | — | 1,668 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,319.0 | — | 3,445 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,454.0 | — | 7,578 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,200.0 | — | 17,773 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,537.0 | — | 45,292 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,508.0 | — | 46,441 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 126.2 | 169.9 | 92 ms | 6.3 ms | 1.0 | 1 | 3,786 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 258.7 | 138.5 | 130 ms | 7.3 ms | 2.0 | 2 | 7,761 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 358.8 | 94.1 | 1,658 ms | 10.6 ms | 6.0 | 0 | 10,763 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 655.2 | 82.6 | 4,282 ms | 12.1 ms | 8.0 | 0 | 19,652 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 643.5 | 79.9 | 15,868 ms | 12.5 ms | 8.1 | 9 | 19,292 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 698.2 | 91.1 | 29,828 ms | 11.0 ms | 8.1 | 7 | 20,944 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 141.1 | 158.8 | 115 ms | 6.4 ms | 1.0 | 1 | 4,233 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 223.8 | 111.3 | 478 ms | 9.0 ms | 2.6 | 0 | 6,710 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 433.5 | 115.5 | 233 ms | 8.7 ms | 4.0 | 0 | 12,997 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 631.2 | 86.4 | 348 ms | 11.6 ms | 8.0 | 0 | 18,925 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 676.0 | 84.0 | 27,483 ms | 11.9 ms | 8.1 | 8 | 20,266 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 125.6 | 159.9 | 170 ms | 6.5 ms | 1.0 | 1 | 3,768 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 240.0 | 125.0 | 246 ms | 8.0 ms | 2.3 | 1 | 7,201 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 372.7 | 97.4 | 412 ms | 10.3 ms | 5.0 | 0 | 11,171 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 581.5 | 75.2 | 1,932 ms | 13.3 ms | 8.1 | 0 | 17,441 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 101.7 | 107.4 | 296 ms | 9.3 ms | 1.0 | 0 | 3,048 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 221.5 | 110.1 | 558 ms | 9.1 ms | 2.1 | 0 | 6,643 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 385.2 | 101.2 | 990 ms | 9.9 ms | 4.0 | 0 | 11,551 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 119.3 | 120.8 | 536 ms | 8.3 ms | 1.0 | 1 | 3,558 | 29.8 s |
| Sustained decode | 128K | 2 | Valid | 216.5 | 107.6 | 1,149 ms | 9.3 ms | 2.2 | 0 | 6,494 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 108.4 | 109.8 | 1,006 ms | 9.1 ms | 1.0 | 0 | 3,250 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.