Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.852730 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --linear-attn-prefill-backend flashinfer --linear-attn-decode-backend flashinfer --mamba-scheduler-strategy extra_buffer --max-running-requests 32 --cuda-graph-max-bs-decode 32 --cuda-graph-backend-prefill tc_piecewise --enable-flashinfer-allreduce-fusion --attention-backend flashinfer --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.5-9B-DFlash --speculative-dflash-block-size 8 --speculative-draft-attention-backend fa4
Prefill
prompt tok/s by context length8K context25.7K317 ms TTFT
16K context26.4K611 ms TTFT
32K context24.4K1,314 ms TTFT
64K context20.5K3,132 ms TTFT
128K context15.5K8,238 ms TTFT
251K context10.5K23,917 ms TTFT
256K context10.5K24,424 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 329.778 ms TTFT | 636.785 ms TTFT | 1,189.593 ms TTFT | 1,919.793 ms TTFT | 2,838.5239 ms TTFT | 3,712.8Requests queued1,551 ms TTFT | Not run | Over capacity |
| 16K | 316.9107 ms TTFT | 560.6117 ms TTFT | 1,097.7112 ms TTFT | 1,892.9146 ms TTFT | 2,640.5396 ms TTFT | 3,378.0Requests queued1,236 ms TTFT | Over capacity | Over capacity |
| 32K | 310.4161 ms TTFT | 427.7210 ms TTFT | 1,072.6168 ms TTFT | 1,667.7571 ms TTFT | 2,463.21,016 ms TTFT | 2,839.8Requests queued3,103 ms TTFT | Over capacity | Over capacity |
| 64K | 326.0290 ms TTFT | 401.5420 ms TTFT | 937.5828 ms TTFT | 1,430.41,328 ms TTFT | 1,801.93,699 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 263.4511 ms TTFT | 420.4981 ms TTFT | 768.81,382 ms TTFT | 1,031.33,629 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 225.41,006 ms TTFT | 194.31,908 ms TTFT | 566.72,827 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.7K | — | 317 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.4K | — | 611 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.4K | — | 1,314 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.5K | — | 3,132 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.5K | — | 8,238 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.5K | — | 23,917 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.5K | — | 24,424 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 329.7 | 332.7 | 78 ms | 3.0 ms | 1.0 | 2 | 9,891 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 636.7 | 327.2 | 85 ms | 3.1 ms | 2.0 | 4 | 19,096 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,189.5 | 301.3 | 93 ms | 3.3 ms | 3.9 | 8 | 35,682 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,919.7 | 240.8 | 93 ms | 4.2 ms | 8.2 | 12 | 57,585 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,838.5 | 186.2 | 239 ms | 5.4 ms | 17.8 | 16 | 85,155 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 3,712.8 | 126.1 | 1,551 ms | 7.9 ms | 29.4 | 59 | 111,352 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 316.9 | 331.6 | 107 ms | 3.0 ms | 1.0 | 2 | 9,505 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 560.6 | 331.8 | 117 ms | 3.0 ms | 7.1 | 4 | 16,815 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,097.7 | 282.8 | 112 ms | 3.5 ms | 4.3 | 8 | 32,928 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,892.9 | 235.8 | 146 ms | 4.2 ms | 8.8 | 10 | 56,776 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,640.5 | 172.1 | 396 ms | 5.8 ms | 16.9 | 16 | 79,220 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 3,378.0 | 114.7 | 1,236 ms | 8.7 ms | 30.1 | 51 | 101,263 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 310.4 | 319.1 | 161 ms | 3.1 ms | 1.0 | 2 | 9,310 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 427.7 | 235.5 | 210 ms | 4.5 ms | 11.7 | 2 | 12,831 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,072.6 | 273.0 | 168 ms | 3.7 ms | 3.9 | 8 | 32,183 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,667.7 | 210.8 | 571 ms | 4.7 ms | 9.0 | 8 | 50,026 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,463.2 | 159.4 | 1,016 ms | 6.3 ms | 16.2 | 16 | 73,940 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 2,839.8 | 97.9 | 3,103 ms | 10.2 ms | 30.0 | 41 | 85,182 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 326.0 | 333.1 | 290 ms | 3.0 ms | 1.0 | 2 | 9,781 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 401.5 | 228.9 | 420 ms | 4.7 ms | 10.9 | 2 | 12,042 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 937.5 | 241.5 | 828 ms | 4.1 ms | 4.0 | 4 | 28,116 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,430.4 | 189.8 | 1,328 ms | 5.3 ms | 7.9 | 8 | 42,912 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,801.9 | 116.7 | 3,699 ms | 8.6 ms | 18.0 | 3 | 54,055 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 263.4 | 280.4 | 511 ms | 3.6 ms | 1.0 | 2 | 7,903 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 420.4 | 217.5 | 981 ms | 4.6 ms | 3.2 | 2 | 12,610 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 768.8 | 201.2 | 1,382 ms | 5.0 ms | 3.9 | 4 | 23,063 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,031.3 | 142.1 | 3,629 ms | 7.0 ms | 8.2 | 8 | 30,622 | 29.7 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 225.4 | 237.1 | 1,006 ms | 4.2 ms | 1.0 | 1 | 6,761 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 194.3 | 99.1 | 1,908 ms | 10.2 ms | 9.6 | 0 | 5,829 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 566.7 | 162.7 | 2,827 ms | 6.1 ms | 3.8 | 4 | 16,997 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.