Speed test matrix
Qwen3.5-2B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.774660 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context61.5K133 ms TTFT
16K context76K212 ms TTFT
32K context77.3K415 ms TTFT
64K context70.2K912 ms TTFT
128K context58.6K2,185 ms TTFT
251K context43.2K5,800 ms TTFT
256K context43.5K5,884 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 531.281 ms TTFT | 1,026.183 ms TTFT | 1,882.477 ms TTFT | 3,398.680 ms TTFT | 5,394.679 ms TTFT | 7,934.077 ms TTFT | 8,994.3733 ms TTFT | 12.1K2,418 ms TTFT |
| 16K | 507.0100 ms TTFT | 604.1109 ms TTFT | 1,839.6100 ms TTFT | 3,348.899 ms TTFT | 5,150.0103 ms TTFT | 7,135.4124 ms TTFT | 7,594.92,485 ms TTFT | 10.4KRequests queued4,812 ms TTFT |
| 32K | 516.0160 ms TTFT | 615.3160 ms TTFT | 1,773.5159 ms TTFT | 2,964.0160 ms TTFT | 4,260.9258 ms TTFT | 5,122.21,844 ms TTFT | 5,201.06,452 ms TTFT | 7,598.7Requests queued12,165 ms TTFT |
| 64K | 530.1305 ms TTFT | 116.9553 ms TTFT | 1,266.6298 ms TTFT | 2,609.4293 ms TTFT | 3,541.4762 ms TTFT | 3,572.97,383 ms TTFT | 3,372.115,312 ms TTFT | Over capacity |
| 128K | 495.2536 ms TTFT | 282.41,020 ms TTFT | 1,333.7970 ms TTFT | 1,947.5972 ms TTFT | 2,358.14,550 ms TTFT | 1,833.214,706 ms TTFT | Over capacity | Over capacity |
| 251K | 419.91,018 ms TTFT | 505.21,522 ms TTFT | 1,027.31,004 ms TTFT | 1,463.54,744 ms TTFT | 1,553.614,925 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 61.5K | — | 133 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 76K | — | 212 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 77.3K | — | 415 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 70.2K | — | 912 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 58.6K | — | 2,185 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 43.2K | — | 5,800 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 43.5K | — | 5,884 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 531.2 | 502.8 | 81 ms | 2.0 ms | 1.0 | 4 | 15,937 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,026.1 | 472.3 | 83 ms | 2.1 ms | 2.0 | 8 | 30,780 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,882.4 | 451.1 | 77 ms | 2.2 ms | 4.1 | 14 | 56,459 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,398.6 | 412.0 | 80 ms | 2.4 ms | 8.0 | 25 | 101,995 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,394.6 | 329.5 | 79 ms | 3.0 ms | 16.2 | 34 | 161,839 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,934.0 | 234.6 | 77 ms | 4.3 ms | 32.4 | 48 | 238,022 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 8,994.3 | 144.3 | 733 ms | 6.9 ms | 71.1 | 64 | 269,838 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.1K | 91.4 | 2,418 ms | 10.9 ms | 127.6 | 138 | 364,106 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 507.0 | 506.7 | 100 ms | 2.0 ms | 0.9 | 4 | 15,210 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 604.1 | 372.1 | 109 ms | 2.8 ms | 43.2 | 4 | 18,120 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,839.6 | 443.7 | 100 ms | 2.3 ms | 4.0 | 12 | 55,189 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,348.8 | 401.9 | 99 ms | 2.5 ms | 7.8 | 25 | 100,464 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 5,150.0 | 301.9 | 103 ms | 3.3 ms | 16.2 | 33 | 154,490 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,135.4 | 217.7 | 124 ms | 4.6 ms | 32.9 | 39 | 214,030 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,594.9 | 123.2 | 2,485 ms | 8.1 ms | 70.9 | 27 | 227,796 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 10.4K | 79.7 | 4,812 ms | 12.5 ms | 127.9 | 118 | 311,985 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 516.0 | 483.3 | 160 ms | 2.1 ms | 0.9 | 4 | 15,480 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 615.3 | 415.3 | 160 ms | 2.4 ms | 30.1 | 4 | 18,458 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,773.5 | 432.7 | 159 ms | 2.3 ms | 3.9 | 12 | 53,201 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,964.0 | 345.7 | 160 ms | 2.9 ms | 7.8 | 21 | 88,476 | 29.9 s |
| Sustained decode | 32K | 16 | Valid | 4,260.9 | 265.0 | 258 ms | 3.8 ms | 16.5 | 32 | 127,818 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,122.2 | 168.8 | 1,844 ms | 5.9 ms | 37.3 | 32 | 153,652 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,201.0 | 86.0 | 6,452 ms | 11.6 ms | 79.9 | 1 | 155,994 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 7,598.7 | 61.0 | 12,165 ms | 16.4 ms | 126.7 | 118 | 227,815 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 530.1 | 486.8 | 305 ms | 2.1 ms | 1.0 | 4 | 15,904 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 116.9 | 60.7 | 553 ms | 16.7 ms | 116.6 | 0 | 3,506 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,266.6 | 380.5 | 298 ms | 2.6 ms | 17.3 | 8 | 37,995 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,609.4 | 336.1 | 293 ms | 3.0 ms | 8.0 | 16 | 78,272 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,541.4 | 223.9 | 762 ms | 4.5 ms | 15.9 | 19 | 106,249 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,572.9 | 127.0 | 7,383 ms | 7.9 ms | 33.8 | 18 | 105,487 | 29.5 s |
| Sustained decode | 64K | 64 | Valid | 3,372.1 | 62.4 | 15,312 ms | 16.0 ms | 78.7 | 4 | 101,120 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 495.2 | 532.3 | 536 ms | 1.9 ms | 0.9 | 4 | 14,670 | 29.6 s |
| Sustained decode | 128K | 2 | Valid | 282.4 | 131.3 | 1,020 ms | 7.6 ms | 43.5 | 2 | 8,418 | 29.8 s |
| Sustained decode | 128K | 4 | Valid | 1,333.7 | 340.1 | 970 ms | 2.9 ms | 3.9 | 8 | 40,011 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,947.5 | 273.3 | 972 ms | 3.7 ms | 7.7 | 14 | 58,392 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,358.1 | 155.9 | 4,550 ms | 6.4 ms | 17.0 | 16 | 70,725 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,833.2 | 64.2 | 14,706 ms | 15.6 ms | 38.2 | 0 | 54,963 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 419.9 | 451.6 | 1,018 ms | 2.2 ms | 0.9 | 3 | 12,596 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 505.2 | 267.1 | 1,522 ms | 3.9 ms | 12.2 | 3 | 14,950 | 29.6 s |
| Sustained decode | 251K | 4 | Valid | 1,027.3 | 273.5 | 1,004 ms | 3.7 ms | 3.9 | 7 | 30,817 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,463.5 | 187.8 | 4,744 ms | 5.3 ms | 7.9 | 8 | 43,895 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,553.6 | 106.1 | 14,925 ms | 9.4 ms | 15.6 | 8 | 45,395 | 29.2 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.