Speed test matrix
Qwen3.5-2B-Base on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861540 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context62.2K131 ms TTFT
16K context79.8K202 ms TTFT
32K context78.4K410 ms TTFT
64K context73.4K873 ms TTFT
128K context63.4K2,019 ms TTFT
251K context47.5K5,280 ms TTFT
256K context48K5,329 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 448.076 ms TTFT | 874.884 ms TTFT | 1,669.692 ms TTFT | 3,038.4126 ms TTFT | 5,216.9356 ms TTFT | 8,267.4586 ms TTFT | 10.2K1,114 ms TTFT | 12.4K2,658 ms TTFT |
| 16K | 439.297 ms TTFT | 847.3112 ms TTFT | 1,610.1152 ms TTFT | 2,859.8166 ms TTFT | 4,868.0194 ms TTFT | 7,120.8914 ms TTFT | 9,726.21,802 ms TTFT | 8,257.25,062 ms TTFT |
| 32K | 426.4153 ms TTFT | 504.7219 ms TTFT | 1,499.8241 ms TTFT | 2,534.9239 ms TTFT | 4,054.2434 ms TTFT | 4,252.22,205 ms TTFT | 4,433.06,550 ms TTFT | 6,716.312,677 ms TTFT |
| 64K | 404.5264 ms TTFT | 133.9555 ms TTFT | 1,329.8410 ms TTFT | 2,143.9279 ms TTFT | 2,497.02,126 ms TTFT | 2,926.97,621 ms TTFT | 4,414.815,218 ms TTFT | Over capacity |
| 128K | 364.6506 ms TTFT | 380.3971 ms TTFT | 1,071.2549 ms TTFT | 1,463.72,357 ms TTFT | 1,859.17,441 ms TTFT | 1,445.715,753 ms TTFT | Over capacity | Over capacity |
| 251K | 312.51,027 ms TTFT | 99.62,031 ms TTFT | 707.33,286 ms TTFT | 935.77,511 ms TTFT | 590.415,743 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 62.2K | — | 131 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 79.8K | — | 202 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 78.4K | — | 410 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 73.4K | — | 873 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 63.4K | — | 2,019 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 47.5K | — | 5,280 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 48K | — | 5,329 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 448.0 | 451.5 | 76 ms | 2.2 ms | 1.0 | 3 | 13,439 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 874.8 | 440.8 | 84 ms | 2.3 ms | 2.0 | 6 | 26,243 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,669.6 | 421.1 | 92 ms | 2.4 ms | 4.0 | 12 | 50,089 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,038.4 | 384.1 | 126 ms | 2.6 ms | 8.0 | 24 | 91,152 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,216.9 | 343.1 | 356 ms | 2.9 ms | 17.3 | 32 | 156,511 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,267.4 | 263.2 | 586 ms | 3.8 ms | 31.7 | 64 | 248,019 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.2K | 172.7 | 1,114 ms | 5.8 ms | 79.8 | 64 | 306,939 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.4K | 98.1 | 2,658 ms | 10.2 ms | 173.8 | 0 | 371,269 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 439.2 | 443.6 | 97 ms | 2.3 ms | 1.0 | 3 | 13,174 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 847.3 | 431.6 | 112 ms | 2.3 ms | 5.2 | 6 | 25,419 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,610.1 | 408.6 | 152 ms | 2.4 ms | 4.0 | 12 | 48,303 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,859.8 | 361.5 | 166 ms | 2.8 ms | 8.1 | 16 | 85,791 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,868.0 | 311.3 | 194 ms | 3.2 ms | 15.7 | 32 | 146,042 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,120.8 | 225.0 | 914 ms | 4.4 ms | 32.6 | 32 | 213,625 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 9,726.2 | 160.1 | 1,802 ms | 6.2 ms | 63.2 | 64 | 291,782 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,257.2 | 69.0 | 5,062 ms | 14.5 ms | 191.5 | 0 | 247,882 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 426.4 | 433.1 | 153 ms | 2.3 ms | 1.0 | 3 | 12,793 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 504.7 | 290.6 | 219 ms | 4.2 ms | 62.8 | 2 | 15,140 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,499.8 | 385.0 | 241 ms | 2.6 ms | 4.0 | 12 | 44,993 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,534.9 | 326.2 | 239 ms | 3.1 ms | 8.4 | 16 | 76,048 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,054.2 | 257.0 | 434 ms | 3.9 ms | 15.6 | 32 | 121,619 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,252.2 | 151.7 | 2,205 ms | 6.6 ms | 41.2 | 32 | 127,575 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,433.0 | 76.1 | 6,550 ms | 13.1 ms | 95.8 | 0 | 132,992 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,716.3 | 59.1 | 12,677 ms | 16.9 ms | 153.6 | 0 | 201,439 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 404.5 | 415.3 | 264 ms | 2.4 ms | 1.0 | 3 | 12,135 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 133.9 | 66.1 | 555 ms | 15.1 ms | 117.2 | 0 | 4,018 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,329.8 | 339.7 | 410 ms | 2.9 ms | 3.9 | 8 | 39,913 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,143.9 | 273.8 | 279 ms | 3.7 ms | 7.9 | 16 | 64,318 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,497.0 | 172.9 | 2,126 ms | 5.8 ms | 18.9 | 16 | 74,912 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,926.9 | 99.0 | 7,621 ms | 10.1 ms | 41.1 | 0 | 87,809 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,414.8 | 85.3 | 15,218 ms | 11.7 ms | 64.0 | 0 | 132,414 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 364.6 | 384.4 | 506 ms | 2.6 ms | 1.0 | 3 | 10,937 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 380.3 | 243.8 | 971 ms | 4.9 ms | 28.7 | 2 | 11,409 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,071.2 | 279.6 | 549 ms | 3.6 ms | 3.9 | 8 | 32,133 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,463.7 | 201.8 | 2,357 ms | 5.0 ms | 9.0 | 8 | 43,908 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,859.1 | 117.8 | 7,441 ms | 8.5 ms | 16.4 | 9 | 55,538 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 1,445.7 | 57.7 | 15,753 ms | 17.3 ms | 41.0 | 0 | 43,360 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 312.5 | 335.7 | 1,027 ms | 3.0 ms | 0.9 | 2 | 9,375 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 99.6 | 49.8 | 2,031 ms | 20.1 ms | 33.6 | 0 | 2,988 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 707.3 | 193.8 | 3,286 ms | 5.2 ms | 4.3 | 4 | 21,219 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 935.7 | 128.3 | 7,511 ms | 7.8 ms | 7.9 | 7 | 28,519 | 30.5 s |
| Sustained decode | 251K | 16 | Valid | 590.4 | 48.8 | 15,743 ms | 20.5 ms | 23.0 | 0 | 17,712 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.