Speed test matrix
Qwen3.5-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.850929 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --linear-attn-prefill-backend flashinfer --linear-attn-decode-backend flashinfer --mamba-scheduler-strategy extra_buffer --max-running-requests 32 --cuda-graph-max-bs-decode 32 --cuda-graph-backend-prefill tc_piecewise --enable-flashinfer-allreduce-fusion --attention-backend flashinfer --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.5-27B-DFlash --speculative-dflash-block-size 8 --speculative-draft-attention-backend fa4
Prefill
prompt tok/s by context length8K context9,165889 ms TTFT
16K context8,7201,849 ms TTFT
32K context8,1483,940 ms TTFT
64K context6,7179,537 ms TTFT
128K context5,03225,434 ms TTFT
251K context3,43573,022 ms TTFT
256K context3,41474,925 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 180.2102 ms TTFT | 285.6174 ms TTFT | 611.1919 ms TTFT | 947.5200 ms TTFT | 1,550.5Requests queued943 ms TTFT | 1,531.8Requests queued38,412 ms TTFT | Over capacity | Over capacity |
| 16K | 190.8136 ms TTFT | 336.7168 ms TTFT | 586.5184 ms TTFT | 923.2348 ms TTFT | 1,442.7Requests queued783 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 172.8185 ms TTFT | 300.8235 ms TTFT | 553.4321 ms TTFT | 793.4789 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 182.8314 ms TTFT | 302.1420 ms TTFT | 495.2948 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 152.4553 ms TTFT | 233.41,022 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 125.71,040 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,165.0 | — | 889 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,720.0 | — | 1,849 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,148.0 | — | 3,940 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,717.0 | — | 9,537 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,032.0 | — | 25,434 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,435.0 | — | 73,022 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,414.0 | — | 74,925 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 180.2 | 182.6 | 102 ms | 5.5 ms | 1.0 | 1 | 5,405 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 285.6 | 161.0 | 174 ms | 6.3 ms | 2.8 | 2 | 8,564 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 611.1 | 156.4 | 919 ms | 6.4 ms | 5.1 | 4 | 18,325 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 947.5 | 122.7 | 200 ms | 8.1 ms | 10.6 | 4 | 28,424 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 1,550.5 | 107.8 | 943 ms | 9.3 ms | 14.1 | 24 | 46,494 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,531.8 | 105.9 | 38,412 ms | 9.4 ms | 14.5 | 27 | 45,914 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 190.8 | 189.2 | 136 ms | 5.3 ms | 1.0 | 1 | 5,721 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 336.7 | 166.7 | 168 ms | 6.0 ms | 3.2 | 2 | 10,098 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 586.5 | 154.0 | 184 ms | 6.5 ms | 4.8 | 4 | 17,589 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 923.2 | 115.3 | 348 ms | 8.7 ms | 10.8 | 1 | 27,635 | 29.9 s |
| Sustained decode | 16K | 16 | Requests queued | 1,442.7 | 100.5 | 783 ms | 9.9 ms | 14.7 | 27 | 43,257 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 172.8 | 174.0 | 185 ms | 5.7 ms | 1.0 | 1 | 5,183 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 300.8 | 154.7 | 235 ms | 6.5 ms | 2.8 | 2 | 9,018 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 553.4 | 145.5 | 321 ms | 6.9 ms | 4.9 | 4 | 16,592 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 793.4 | 100.4 | 789 ms | 10.0 ms | 11.7 | 0 | 23,780 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 182.8 | 182.8 | 314 ms | 5.5 ms | 1.0 | 1 | 5,480 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 302.1 | 151.6 | 420 ms | 6.6 ms | 2.1 | 2 | 9,062 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 495.2 | 134.7 | 948 ms | 7.4 ms | 5.3 | 3 | 14,856 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 152.4 | 156.2 | 553 ms | 6.4 ms | 1.0 | 1 | 4,571 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 233.4 | 118.3 | 1,022 ms | 8.5 ms | 4.3 | 1 | 6,900 | 29.6 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 125.7 | 137.8 | 1,040 ms | 7.3 ms | 0.9 | 1 | 3,769 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.