Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.825970 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context27.6K295 ms TTFT
16K context28.1K573 ms TTFT
32K context27.5K1,165 ms TTFT
64K context24.9K2,569 ms TTFT
128K context20.9K6,113 ms TTFT
251K context15.9K15,822 ms TTFT
256K context15.8K16,195 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 290.783 ms TTFT | 558.394 ms TTFT | 1,070.696 ms TTFT | 1,811.9154 ms TTFT | 3,187.0231 ms TTFT | 4,357.7424 ms TTFT | 4,127.81,301 ms TTFT | 5,688.7Requests queued32,341 ms TTFT |
| 16K | 284.0107 ms TTFT | 188.9176 ms TTFT | 1,025.9131 ms TTFT | 1,657.3254 ms TTFT | 2,921.9461 ms TTFT | 3,449.91,404 ms TTFT | 4,889.92,543 ms TTFT | Over capacity |
| 32K | 275.3168 ms TTFT | 314.9225 ms TTFT | 942.0461 ms TTFT | 1,637.1592 ms TTFT | 2,447.0983 ms TTFT | 2,130.83,294 ms TTFT | Over capacity | Over capacity |
| 64K | 260.9300 ms TTFT | 291.6523 ms TTFT | 833.8835 ms TTFT | 1,344.21,310 ms TTFT | 1,450.63,690 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 239.6537 ms TTFT | 400.21,004 ms TTFT | 699.61,464 ms TTFT | 849.63,935 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 192.41,018 ms TTFT | 342.0Requests queued1,944 ms TTFT | 471.53,790 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.6K | — | 295 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.1K | — | 573 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.5K | — | 1,165 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 24.9K | — | 2,569 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,113 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 15.9K | — | 15,822 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.8K | — | 16,195 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 290.7 | 304.8 | 83 ms | 3.3 ms | 1.0 | 2 | 8,722 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 558.3 | 279.0 | 94 ms | 3.6 ms | 2.0 | 4 | 16,748 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,070.6 | 271.5 | 96 ms | 3.7 ms | 4.2 | 8 | 32,113 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,811.9 | 225.4 | 154 ms | 4.4 ms | 9.0 | 8 | 54,345 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,187.0 | 202.9 | 231 ms | 4.9 ms | 15.9 | 16 | 95,615 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,357.7 | 140.3 | 424 ms | 7.1 ms | 33.7 | 32 | 130,697 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,127.8 | 66.5 | 1,301 ms | 15.0 ms | 96.0 | 0 | 123,802 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 5,688.7 | 58.6 | 32,341 ms | 17.1 ms | 100.8 | 28 | 170,666 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 284.0 | 287.0 | 107 ms | 3.5 ms | 1.0 | 2 | 8,521 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 188.9 | 94.8 | 176 ms | 10.6 ms | 54.4 | 0 | 5,665 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,025.9 | 266.7 | 131 ms | 3.7 ms | 4.0 | 8 | 30,775 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,657.3 | 211.5 | 254 ms | 4.7 ms | 9.6 | 8 | 49,710 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,921.9 | 184.2 | 461 ms | 5.4 ms | 16.0 | 16 | 87,653 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,449.9 | 108.7 | 1,404 ms | 9.2 ms | 38.8 | 0 | 103,476 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,889.9 | 78.7 | 2,543 ms | 12.7 ms | 64.0 | 0 | 146,664 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 275.3 | 278.9 | 168 ms | 3.6 ms | 1.0 | 2 | 8,258 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 314.9 | 204.0 | 225 ms | 5.5 ms | 36.7 | 2 | 9,448 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 942.0 | 236.6 | 461 ms | 4.2 ms | 4.4 | 4 | 28,258 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,637.1 | 209.2 | 592 ms | 4.8 ms | 8.0 | 8 | 49,113 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,447.0 | 157.2 | 983 ms | 6.4 ms | 15.9 | 16 | 73,347 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,130.8 | 71.3 | 3,294 ms | 14.0 ms | 47.8 | 0 | 63,865 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 260.9 | 272.7 | 300 ms | 3.7 ms | 1.0 | 2 | 7,824 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 291.6 | 209.8 | 523 ms | 5.6 ms | 19.0 | 2 | 8,753 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 833.8 | 212.5 | 835 ms | 4.7 ms | 4.6 | 4 | 25,014 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,344.2 | 173.2 | 1,310 ms | 5.8 ms | 7.9 | 8 | 40,338 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,450.6 | 96.0 | 3,690 ms | 10.4 ms | 21.4 | 0 | 43,504 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 239.6 | 247.7 | 537 ms | 4.0 ms | 0.9 | 2 | 7,083 | 29.6 s |
| Sustained decode | 128K | 2 | Valid | 400.2 | 197.9 | 1,004 ms | 5.1 ms | 3.4 | 2 | 12,003 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 699.6 | 182.9 | 1,464 ms | 5.5 ms | 4.1 | 4 | 20,984 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 849.6 | 109.3 | 3,935 ms | 9.2 ms | 9.8 | 0 | 25,481 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 192.4 | 199.3 | 1,018 ms | 5.0 ms | 1.0 | 1 | 5,772 | 30.0 s |
| Sustained decode | 251K | 2 | Requests queued | 342.0 | 184.9 | 1,944 ms | 5.4 ms | 1.9 | 2 | 10,257 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 471.5 | 119.5 | 3,790 ms | 8.4 ms | 4.6 | 2 | 14,097 | 29.9 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.