Speed test matrix
Qwen3.6-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.874606 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context9,618847 ms TTFT
16K context9,5311,692 ms TTFT
32K context9,1903,493 ms TTFT
64K context8,2687,748 ms TTFT
128K context7,00818,260 ms TTFT
251K context5,35746,821 ms TTFT
256K context5,32448,043 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 131.798 ms TTFT | 252.0147 ms TTFT | 434.41,639 ms TTFT | 838.9203 ms TTFT | 1,409.0351 ms TTFT | 1,692.5Requests queued11,917 ms TTFT | Over capacity | Over capacity |
| 16K | 131.7116 ms TTFT | 188.5215 ms TTFT | 452.6243 ms TTFT | 815.0348 ms TTFT | 1,333.8663 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 135.7172 ms TTFT | 211.2284 ms TTFT | 441.5417 ms TTFT | 755.0809 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 119.4301 ms TTFT | 220.6550 ms TTFT | 408.31,021 ms TTFT | 661.51,844 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 116.8538 ms TTFT | 161.31,059 ms TTFT | 331.61,909 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 103.81,021 ms TTFT | 161.51,972 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,618.0 | — | 847 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,531.0 | — | 1,692 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,190.0 | — | 3,493 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,268.0 | — | 7,748 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,008.0 | — | 18,260 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,357.0 | — | 46,821 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,324.0 | — | 48,043 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 131.7 | 143.4 | 98 ms | 7.0 ms | 1.0 | 1 | 3,950 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 252.0 | 136.3 | 147 ms | 7.4 ms | 2.0 | 2 | 7,559 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 434.4 | 108.8 | 1,639 ms | 9.2 ms | 6.0 | 0 | 13,032 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 838.9 | 106.2 | 203 ms | 9.4 ms | 8.0 | 0 | 25,157 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,409.0 | 87.4 | 351 ms | 11.4 ms | 16.1 | 0 | 42,265 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,692.5 | 73.9 | 11,917 ms | 13.5 ms | 22.8 | 23 | 50,757 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 131.7 | 138.8 | 116 ms | 7.2 ms | 1.0 | 1 | 3,948 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 188.5 | 94.8 | 215 ms | 10.6 ms | 10.1 | 0 | 5,656 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 452.6 | 111.9 | 243 ms | 8.9 ms | 4.4 | 0 | 13,576 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 815.0 | 102.7 | 348 ms | 9.7 ms | 8.0 | 0 | 24,433 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,333.8 | 82.5 | 663 ms | 12.1 ms | 16.0 | 0 | 39,976 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 135.7 | 132.8 | 172 ms | 7.5 ms | 1.0 | 1 | 4,071 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 211.2 | 103.1 | 284 ms | 9.7 ms | 6.8 | 0 | 6,334 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 441.5 | 111.6 | 417 ms | 9.0 ms | 4.0 | 0 | 13,237 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 755.0 | 96.3 | 809 ms | 10.4 ms | 8.0 | 0 | 22,643 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.4 | 150.3 | 301 ms | 6.9 ms | 1.0 | 1 | 3,581 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 220.6 | 106.5 | 550 ms | 9.4 ms | 3.3 | 0 | 6,615 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 408.3 | 104.1 | 1,021 ms | 9.6 ms | 4.0 | 0 | 12,246 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 661.5 | 83.4 | 1,844 ms | 12.0 ms | 8.1 | 0 | 19,838 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 116.8 | 118.5 | 538 ms | 8.4 ms | 1.0 | 0 | 3,505 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 161.3 | 79.7 | 1,059 ms | 12.6 ms | 6.1 | 0 | 4,836 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 331.6 | 84.0 | 1,909 ms | 11.9 ms | 4.6 | 0 | 9,946 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 103.8 | 104.3 | 1,021 ms | 9.6 ms | 1.0 | 0 | 3,112 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 161.5 | 80.4 | 1,972 ms | 12.4 ms | 3.3 | 0 | 4,842 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.