Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.852211 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 16
Prefill
prompt tok/s by context length8K context9,751840 ms TTFT
16K context9,6641,673 ms TTFT
32K context9,3553,437 ms TTFT
64K context8,4467,590 ms TTFT
128K context7,21717,738 ms TTFT
251K context5,53345,336 ms TTFT
256K context5,50846,449 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 129.388 ms TTFT | 160.9179 ms TTFT | 355.51,680 ms TTFT | 629.64,510 ms TTFT | 629.8Requests queued30,548 ms TTFT | 654.5Requests queued37,130 ms TTFT | Over capacity | Over capacity |
| 16K | 124.1111 ms TTFT | 214.6169 ms TTFT | 380.0231 ms TTFT | 609.71,096 ms TTFT | 649.1Requests queued12,481 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 126.0176 ms TTFT | 149.4280 ms TTFT | 399.04,087 ms TTFT | 570.3864 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 128.0296 ms TTFT | 216.4557 ms TTFT | 377.1978 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 107.4512 ms TTFT | 200.7991 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 133.1989 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,751.0 | — | 840 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,664.0 | — | 1,673 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,355.0 | — | 3,437 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,446.0 | — | 7,590 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,217.0 | — | 17,738 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,533.0 | — | 45,336 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,508.0 | — | 46,449 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 129.3 | 150.7 | 88 ms | 6.8 ms | 1.0 | 1 | 3,879 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 160.9 | 83.6 | 179 ms | 12.0 ms | 3.0 | 0 | 4,825 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 355.5 | 88.1 | 1,680 ms | 11.3 ms | 5.0 | 0 | 10,660 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 629.6 | 80.3 | 4,510 ms | 12.4 ms | 8.0 | 0 | 18,873 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 629.8 | 78.5 | 30,548 ms | 12.7 ms | 8.1 | 8 | 18,894 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 654.5 | 81.2 | 37,130 ms | 12.3 ms | 8.4 | 6 | 19,626 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 124.1 | 166.2 | 111 ms | 6.4 ms | 1.0 | 1 | 3,721 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 214.6 | 102.6 | 169 ms | 9.7 ms | 5.2 | 1 | 6,435 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 380.0 | 97.2 | 231 ms | 10.3 ms | 5.0 | 0 | 11,397 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 609.7 | 78.0 | 1,096 ms | 12.8 ms | 8.0 | 0 | 18,278 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 649.1 | 72.3 | 12,481 ms | 13.8 ms | 8.2 | 9 | 19,467 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 126.0 | 174.8 | 176 ms | 6.2 ms | 1.0 | 1 | 3,776 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 149.4 | 77.6 | 280 ms | 12.9 ms | 6.1 | 0 | 4,480 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 399.0 | 96.2 | 4,087 ms | 10.4 ms | 4.4 | 0 | 11,960 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 570.3 | 74.0 | 864 ms | 13.5 ms | 8.1 | 0 | 17,096 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 128.0 | 143.7 | 296 ms | 7.0 ms | 1.0 | 1 | 3,837 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 216.4 | 107.1 | 557 ms | 9.3 ms | 2.4 | 0 | 6,488 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 377.1 | 96.2 | 978 ms | 10.4 ms | 4.0 | 0 | 11,312 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 107.4 | 111.2 | 512 ms | 9.0 ms | 1.0 | 0 | 3,221 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 200.7 | 100.7 | 991 ms | 9.9 ms | 2.4 | 0 | 6,019 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 133.1 | 133.2 | 989 ms | 7.5 ms | 1.0 | 1 | 3,992 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.