Speed test matrix
Qwen3.5-27B-GPTQ-Int4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.858063 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context6,2361,307 ms TTFT
16K context6,2292,589 ms TTFT
32K context6,0905,272 ms TTFT
64K context5,67511,289 ms TTFT
128K context5,06225,282 ms TTFT
251K context4,17460,089 ms TTFT
256K context4,15761,536 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 93.289 ms TTFT | 145.8131 ms TTFT | 340.21,622 ms TTFT | 601.1204 ms TTFT | 928.0355 ms TTFT | 1,223.8751 ms TTFT | 1,178.5Requests queued82,699 ms TTFT | Over capacity |
| 16K | 91.7113 ms TTFT | 74.5191 ms TTFT | 150.2208 ms TTFT | 464.7348 ms TTFT | 722.5684 ms TTFT | 969.11,267 ms TTFT | Over capacity | Over capacity |
| 32K | 89.7174 ms TTFT | 55.9279 ms TTFT | 118.9462 ms TTFT | 232.8837 ms TTFT | 529.21,600 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 85.6307 ms TTFT | 64.2560 ms TTFT | 109.8973 ms TTFT | 208.11,910 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 78.5528 ms TTFT | 51.41,037 ms TTFT | 93.21,842 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 67.5984 ms TTFT | 42.21,956 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,236.0 | — | 1,307 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,229.0 | — | 2,589 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,090.0 | — | 5,272 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,675.0 | — | 11,289 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,062.0 | — | 25,282 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,174.0 | — | 60,089 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,157.0 | — | 61,536 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 93.2 | 93.2 | 89 ms | 10.7 ms | 1.0 | 0 | 2,794 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 145.8 | 71.9 | 131 ms | 13.9 ms | 2.7 | 0 | 4,374 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 340.2 | 84.8 | 1,622 ms | 11.8 ms | 4.9 | 0 | 10,204 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 601.1 | 74.0 | 204 ms | 13.5 ms | 9.5 | 0 | 18,032 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 928.0 | 57.5 | 355 ms | 17.4 ms | 21.6 | 0 | 27,840 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,223.8 | 38.3 | 751 ms | 26.1 ms | 48.0 | 0 | 36,704 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,178.5 | 33.8 | 82,699 ms | 29.6 ms | 58.9 | 0 | 35,340 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 91.7 | 91.7 | 113 ms | 10.9 ms | 1.0 | 0 | 2,751 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 74.5 | 36.9 | 191 ms | 27.1 ms | 40.4 | 0 | 2,232 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 150.2 | 37.6 | 208 ms | 26.6 ms | 38.0 | 0 | 4,504 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 464.7 | 58.2 | 348 ms | 17.2 ms | 13.2 | 0 | 13,936 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 722.5 | 45.1 | 684 ms | 22.2 ms | 24.9 | 0 | 21,664 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 969.1 | 30.1 | 1,267 ms | 33.2 ms | 49.9 | 0 | 29,056 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 89.7 | 89.7 | 174 ms | 11.1 ms | 1.0 | 0 | 2,691 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 55.9 | 28.0 | 279 ms | 35.7 ms | 50.0 | 0 | 1,676 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 118.9 | 29.3 | 462 ms | 34.1 ms | 42.7 | 0 | 3,564 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 232.8 | 29.1 | 837 ms | 34.3 ms | 43.9 | 0 | 6,984 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 529.2 | 33.5 | 1,600 ms | 29.8 ms | 29.5 | 0 | 15,872 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 85.6 | 85.6 | 307 ms | 11.7 ms | 1.0 | 0 | 2,568 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 64.2 | 32.1 | 560 ms | 31.1 ms | 28.3 | 0 | 1,926 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 109.8 | 27.2 | 973 ms | 36.7 ms | 26.9 | 0 | 3,292 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 208.1 | 26.2 | 1,910 ms | 38.2 ms | 26.2 | 0 | 6,240 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 78.5 | 78.5 | 528 ms | 12.7 ms | 1.0 | 0 | 2,353 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 51.4 | 25.9 | 1,037 ms | 38.7 ms | 16.0 | 0 | 1,542 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 93.2 | 23.7 | 1,842 ms | 42.2 ms | 18.0 | 0 | 2,796 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 67.5 | 67.5 | 984 ms | 14.8 ms | 1.0 | 0 | 2,025 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 42.2 | 21.3 | 1,956 ms | 46.9 ms | 16.6 | 0 | 1,266 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.