Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.862486 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion
Prefill
prompt tok/s by context length8K context27.8K293 ms TTFT
16K context28.9K558 ms TTFT
32K context28.4K1,130 ms TTFT
64K context26.2K2,444 ms TTFT
128K context22.4K5,724 ms TTFT
251K context17K14,718 ms TTFT
256K context17K15,019 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 172.782 ms TTFT | 263.9100 ms TTFT | 637.9118 ms TTFT | 1,152.3192 ms TTFT | 1,916.8345 ms TTFT | 2,751.3640 ms TTFT | 4,434.91,282 ms TTFT | 4,455.02,443 ms TTFT |
| 16K | 170.998 ms TTFT | 87.4153 ms TTFT | 196.6210 ms TTFT | 1,048.1313 ms TTFT | 1,554.5643 ms TTFT | 2,915.61,279 ms TTFT | 3,463.32,436 ms TTFT | Over capacity |
| 32K | 167.0161 ms TTFT | 102.1252 ms TTFT | 546.1420 ms TTFT | 755.9788 ms TTFT | 1,591.91,592 ms TTFT | 2,113.23,000 ms TTFT | Over capacity | Over capacity |
| 64K | 158.9284 ms TTFT | 98.4524 ms TTFT | 489.8729 ms TTFT | 546.41,910 ms TTFT | 832.03,867 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 145.7503 ms TTFT | 100.11,114 ms TTFT | 358.51,974 ms TTFT | 516.63,756 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 124.6961 ms TTFT | 83.91,923 ms TTFT | 223.43,818 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.8K | — | 293 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.9K | — | 558 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.4K | — | 1,130 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.2K | — | 2,444 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.4K | — | 5,724 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17K | — | 14,718 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17K | — | 15,019 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 172.7 | 173.3 | 82 ms | 5.8 ms | 1.0 | 1 | 5,182 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 263.9 | 149.2 | 100 ms | 6.8 ms | 2.8 | 2 | 7,917 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 637.9 | 161.6 | 118 ms | 6.2 ms | 5.0 | 4 | 19,134 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,152.3 | 147.3 | 192 ms | 6.8 ms | 9.7 | 8 | 34,570 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,916.8 | 126.7 | 345 ms | 7.9 ms | 21.2 | 16 | 57,503 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,751.3 | 86.7 | 640 ms | 11.5 ms | 48.0 | 0 | 82,528 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,434.9 | 67.7 | 1,282 ms | 14.8 ms | 81.6 | 0 | 133,056 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,455.0 | 35.4 | 2,443 ms | 28.2 ms | 192.0 | 0 | 133,632 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 170.9 | 171.5 | 98 ms | 5.8 ms | 1.0 | 1 | 5,126 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 87.4 | 42.3 | 153 ms | 23.7 ms | 142.8 | 0 | 2,620 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 196.6 | 48.0 | 210 ms | 20.8 ms | 121.2 | 0 | 5,896 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,048.1 | 136.1 | 313 ms | 7.4 ms | 9.7 | 8 | 31,445 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,554.5 | 98.0 | 643 ms | 10.2 ms | 24.0 | 0 | 46,624 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,915.6 | 89.5 | 1,279 ms | 11.2 ms | 33.2 | 0 | 87,456 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,463.3 | 54.3 | 2,436 ms | 18.4 ms | 78.9 | 0 | 103,904 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 167.0 | 167.9 | 161 ms | 6.0 ms | 1.0 | 1 | 5,009 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 102.1 | 51.3 | 252 ms | 19.5 ms | 66.0 | 0 | 3,062 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 546.1 | 137.6 | 420 ms | 7.3 ms | 8.5 | 4 | 16,382 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 755.9 | 95.6 | 788 ms | 10.5 ms | 12.0 | 0 | 22,672 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,591.9 | 98.3 | 1,592 ms | 10.2 ms | 16.8 | 0 | 47,760 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,113.2 | 65.9 | 3,000 ms | 15.2 ms | 34.7 | 0 | 63,392 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 158.9 | 160.6 | 284 ms | 6.2 ms | 1.0 | 1 | 4,768 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 98.4 | 49.4 | 524 ms | 20.3 ms | 33.7 | 0 | 2,952 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 489.8 | 131.0 | 729 ms | 7.7 ms | 4.8 | 4 | 14,695 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 546.4 | 70.3 | 1,910 ms | 14.2 ms | 12.0 | 0 | 16,392 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 832.0 | 54.0 | 3,867 ms | 18.5 ms | 24.5 | 0 | 24,960 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 145.7 | 148.3 | 503 ms | 6.7 ms | 1.0 | 1 | 4,372 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 100.1 | 50.5 | 1,114 ms | 19.8 ms | 18.0 | 0 | 3,002 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 358.5 | 87.8 | 1,974 ms | 11.4 ms | 5.5 | 0 | 10,756 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 516.6 | 65.0 | 3,756 ms | 15.4 ms | 9.5 | 0 | 15,496 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 124.6 | 128.8 | 961 ms | 7.8 ms | 1.0 | 1 | 3,738 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 83.9 | 42.6 | 1,923 ms | 23.5 ms | 10.0 | 0 | 2,516 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 223.4 | 58.6 | 3,818 ms | 17.1 ms | 6.0 | 0 | 6,700 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.