Speed test matrix
Qwen3.5-4B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.806347 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context39.1K208 ms TTFT
16K context43.2K374 ms TTFT
32K context41.2K780 ms TTFT
64K context35.7K1,796 ms TTFT
128K context28.1K4,558 ms TTFT
251K context19.5K12,837 ms TTFT
256K context19.4K13,195 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 395.485 ms TTFT | 746.894 ms TTFT | 1,409.992 ms TTFT | 2,401.795 ms TTFT | 3,860.5111 ms TTFT | 4,702.5455 ms TTFT | 4,672.11,348 ms TTFT | 6,715.5Requests queued16,883 ms TTFT |
| 16K | 399.1111 ms TTFT | 157.1149 ms TTFT | 1,328.2112 ms TTFT | 2,317.3110 ms TTFT | 3,374.6469 ms TTFT | 4,615.3822 ms TTFT | 3,841.92,512 ms TTFT | Over capacity |
| 32K | 390.7163 ms TTFT | 249.1236 ms TTFT | 1,238.4195 ms TTFT | 2,009.3338 ms TTFT | 2,279.61,174 ms TTFT | 2,366.83,196 ms TTFT | Over capacity | Over capacity |
| 64K | 353.6296 ms TTFT | 483.9413 ms TTFT | 1,095.5738 ms TTFT | 1,407.71,392 ms TTFT | 1,774.53,850 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 333.8534 ms TTFT | 501.0986 ms TTFT | 748.31,621 ms TTFT | 1,010.73,894 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 261.8992 ms TTFT | 209.21,926 ms TTFT | 574.72,941 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.1K | — | 208 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 43.2K | — | 374 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 41.2K | — | 780 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35.7K | — | 1,796 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.1K | — | 4,558 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.5K | — | 12,837 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.4K | — | 13,195 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 395.4 | 403.0 | 85 ms | 2.5 ms | 1.0 | 3 | 11,861 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 746.8 | 398.8 | 94 ms | 2.5 ms | 2.1 | 6 | 22,401 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,409.9 | 354.4 | 92 ms | 2.8 ms | 4.3 | 9 | 42,293 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,401.7 | 308.0 | 95 ms | 3.2 ms | 8.2 | 16 | 72,038 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,860.5 | 245.3 | 111 ms | 4.1 ms | 16.4 | 21 | 115,636 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,702.5 | 160.3 | 455 ms | 6.3 ms | 35.0 | 32 | 141,051 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,672.1 | 75.3 | 1,348 ms | 13.3 ms | 95.9 | 0 | 140,176 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 6,715.5 | 62.5 | 16,883 ms | 16.0 ms | 110.4 | 66 | 201,348 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 399.1 | 405.6 | 111 ms | 2.5 ms | 1.0 | 3 | 11,973 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 157.1 | 79.7 | 149 ms | 12.6 ms | 82.1 | 0 | 4,710 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,328.2 | 343.7 | 112 ms | 2.9 ms | 5.9 | 8 | 39,841 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,317.3 | 289.8 | 110 ms | 3.5 ms | 8.0 | 16 | 69,507 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,374.6 | 211.6 | 469 ms | 4.7 ms | 17.7 | 16 | 101,234 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,615.3 | 147.9 | 822 ms | 6.8 ms | 32.4 | 32 | 138,446 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,841.9 | 63.6 | 2,512 ms | 15.7 ms | 96.0 | 0 | 115,165 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 390.7 | 398.6 | 163 ms | 2.5 ms | 1.0 | 3 | 11,720 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 249.1 | 117.1 | 236 ms | 8.5 ms | 61.2 | 0 | 7,472 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,238.4 | 316.8 | 195 ms | 3.2 ms | 4.0 | 8 | 37,145 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,009.3 | 258.8 | 338 ms | 3.9 ms | 7.9 | 16 | 60,279 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,279.6 | 166.2 | 1,174 ms | 6.1 ms | 20.9 | 16 | 68,383 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,366.8 | 79.1 | 3,196 ms | 12.6 ms | 47.8 | 0 | 70,969 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 353.6 | 360.6 | 296 ms | 2.8 ms | 1.0 | 2 | 10,607 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 483.9 | 251.6 | 413 ms | 4.4 ms | 13.2 | 2 | 14,515 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,095.5 | 291.4 | 738 ms | 3.4 ms | 3.9 | 8 | 32,860 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,407.7 | 193.0 | 1,392 ms | 5.2 ms | 9.2 | 8 | 42,221 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,774.5 | 114.1 | 3,850 ms | 8.8 ms | 19.3 | 2 | 53,144 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 333.8 | 344.0 | 534 ms | 2.9 ms | 0.9 | 2 | 10,013 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 501.0 | 283.1 | 986 ms | 3.5 ms | 2.7 | 4 | 15,029 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 748.3 | 205.9 | 1,621 ms | 4.9 ms | 4.8 | 4 | 22,446 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,010.7 | 130.8 | 3,894 ms | 7.6 ms | 8.5 | 6 | 29,701 | 29.4 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 261.8 | 287.7 | 992 ms | 3.5 ms | 0.9 | 2 | 7,853 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 209.2 | 106.9 | 1,926 ms | 9.4 ms | 7.4 | 0 | 6,274 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 574.7 | 159.0 | 2,941 ms | 6.3 ms | 3.9 | 4 | 17,237 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.