Speed test matrix
Qwen3.5-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.858466 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion
Prefill
prompt tok/s by context length8K context9,731838 ms TTFT
16K context9,6411,672 ms TTFT
32K context9,3813,422 ms TTFT
64K context8,5157,523 ms TTFT
128K context7,23817,681 ms TTFT
251K context5,55845,127 ms TTFT
256K context5,53146,243 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 62.8Requests queued87 ms TTFT | 96.9148 ms TTFT | 221.31,653 ms TTFT | 398.5197 ms TTFT | 706.7360 ms TTFT | 1,183.1Requests queued671 ms TTFT | Over capacity | Over capacity |
| 16K | 62.3112 ms TTFT | 82.6171 ms TTFT | 194.1208 ms TTFT | 360.3336 ms TTFT | 616.7642 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 61.2172 ms TTFT | 74.9273 ms TTFT | 149.1413 ms TTFT | 301.4814 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 59.3307 ms TTFT | 78.3524 ms TTFT | 124.8972 ms TTFT | 228.81,826 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.9526 ms TTFT | 54.31,028 ms TTFT | 85.81,824 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 50.21,034 ms TTFT | 38.62,208 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,731.0 | — | 838 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,641.0 | — | 1,672 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,381.0 | — | 3,422 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,515.0 | — | 7,523 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,238.0 | — | 17,681 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,558.0 | — | 45,127 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,531.0 | — | 46,243 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 62.8 | 62.8 | 87 ms | 15.9 ms | 1.0 | 0 | 1,883 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 96.9 | 48.4 | 148 ms | 20.6 ms | 3.0 | 0 | 2,906 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 221.3 | 54.2 | 1,653 ms | 18.4 ms | 6.2 | 0 | 6,636 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 398.5 | 49.6 | 197 ms | 20.1 ms | 12.2 | 0 | 11,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 706.7 | 44.0 | 360 ms | 22.7 ms | 24.4 | 0 | 21,201 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,183.1 | 36.2 | 671 ms | 27.6 ms | 36.8 | 0 | 35,492 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 62.3 | 62.2 | 112 ms | 16.1 ms | 1.0 | 0 | 1,867 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 82.6 | 40.8 | 171 ms | 24.5 ms | 25.3 | 0 | 2,478 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 194.1 | 48.6 | 208 ms | 20.6 ms | 10.0 | 0 | 5,820 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 360.3 | 45.0 | 336 ms | 22.2 ms | 12.9 | 0 | 10,808 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 616.7 | 38.5 | 642 ms | 25.9 ms | 25.7 | 0 | 18,496 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 61.2 | 61.2 | 172 ms | 16.3 ms | 1.0 | 0 | 1,836 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 74.9 | 37.5 | 273 ms | 26.7 ms | 24.4 | 0 | 2,246 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 149.1 | 37.3 | 413 ms | 26.8 ms | 22.0 | 0 | 4,472 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 301.4 | 37.9 | 814 ms | 26.4 ms | 14.0 | 0 | 9,040 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 59.3 | 59.3 | 307 ms | 16.9 ms | 1.0 | 0 | 1,778 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 78.3 | 39.3 | 524 ms | 25.5 ms | 13.7 | 0 | 2,348 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 124.8 | 31.5 | 972 ms | 31.8 ms | 14.0 | 0 | 3,744 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 228.8 | 28.9 | 1,826 ms | 34.6 ms | 14.0 | 0 | 6,856 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.9 | 55.9 | 526 ms | 17.9 ms | 1.0 | 0 | 1,677 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 54.3 | 26.7 | 1,028 ms | 37.4 ms | 14.9 | 0 | 1,626 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 85.8 | 21.8 | 1,824 ms | 45.9 ms | 18.0 | 0 | 2,572 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 50.2 | 50.2 | 1,034 ms | 19.9 ms | 1.0 | 0 | 1,504 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 38.6 | 19.6 | 2,208 ms | 51.1 ms | 16.0 | 0 | 1,156 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.