Speed test matrix
Qwen3.5-2B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861540 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion
Prefill
prompt tok/s by context length8K context62.3K131 ms TTFT
16K context77.6K208 ms TTFT
32K context80.6K399 ms TTFT
64K context74.5K860 ms TTFT
128K context63.9K2,004 ms TTFT
251K context47.8K5,245 ms TTFT
256K context47.9K5,342 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 447.178 ms TTFT | 875.469 ms TTFT | 1,673.978 ms TTFT | 3,033.7129 ms TTFT | 5,212.3298 ms TTFT | 8,260.8623 ms TTFT | 10.1K1,284 ms TTFT | 12.3K2,637 ms TTFT |
| 16K | 438.997 ms TTFT | 845.5105 ms TTFT | 1,609.7144 ms TTFT | 2,863.2169 ms TTFT | 4,890.3127 ms TTFT | 7,138.81,076 ms TTFT | 9,709.81,663 ms TTFT | 8,259.75,173 ms TTFT |
| 32K | 427.6149 ms TTFT | 518.8243 ms TTFT | 1,501.1220 ms TTFT | 2,534.8250 ms TTFT | 4,054.6341 ms TTFT | 4,188.52,503 ms TTFT | 4,428.76,499 ms TTFT | 6,680.312,727 ms TTFT |
| 64K | 404.6270 ms TTFT | 128.9537 ms TTFT | 1,330.9480 ms TTFT | 2,140.2261 ms TTFT | 2,476.32,260 ms TTFT | 2,949.78,385 ms TTFT | 4,400.914,365 ms TTFT | Over capacity |
| 128K | 364.8489 ms TTFT | 405.7993 ms TTFT | 1,075.9490 ms TTFT | 1,472.02,306 ms TTFT | 1,855.67,329 ms TTFT | 1,454.914,654 ms TTFT | Over capacity | Over capacity |
| 251K | 313.11,001 ms TTFT | 103.31,951 ms TTFT | 727.02,837 ms TTFT | 959.37,369 ms TTFT | 587.614,595 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 62.3K | — | 131 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 77.6K | — | 208 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 80.6K | — | 399 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 74.5K | — | 860 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 63.9K | — | 2,004 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 47.8K | — | 5,245 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 47.9K | — | 5,342 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 447.1 | 450.8 | 78 ms | 2.2 ms | 1.0 | 3 | 13,413 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 875.4 | 440.3 | 69 ms | 2.3 ms | 2.0 | 6 | 26,261 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,673.9 | 421.0 | 78 ms | 2.4 ms | 4.0 | 12 | 50,215 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,033.7 | 384.3 | 129 ms | 2.6 ms | 8.0 | 24 | 91,012 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,212.3 | 340.3 | 298 ms | 2.9 ms | 17.3 | 32 | 156,360 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,260.8 | 262.2 | 623 ms | 3.8 ms | 31.7 | 64 | 247,825 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.1K | 174.5 | 1,284 ms | 5.8 ms | 79.3 | 64 | 304,104 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.3K | 97.5 | 2,637 ms | 10.3 ms | 174.7 | 0 | 368,864 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 438.9 | 443.1 | 97 ms | 2.3 ms | 1.0 | 3 | 13,168 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 845.5 | 432.3 | 105 ms | 2.3 ms | 6.2 | 6 | 25,365 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,609.7 | 408.1 | 144 ms | 2.5 ms | 4.0 | 12 | 48,289 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,863.2 | 361.9 | 169 ms | 2.8 ms | 8.1 | 16 | 85,896 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,890.3 | 309.8 | 127 ms | 3.2 ms | 15.8 | 32 | 146,707 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,138.8 | 225.0 | 1,076 ms | 4.4 ms | 32.5 | 32 | 214,169 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 9,709.8 | 160.4 | 1,663 ms | 6.2 ms | 62.8 | 64 | 291,292 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,259.7 | 69.1 | 5,173 ms | 14.5 ms | 190.6 | 0 | 247,808 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 427.6 | 434.1 | 149 ms | 2.3 ms | 1.0 | 3 | 12,826 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 518.8 | 291.9 | 243 ms | 4.2 ms | 59.7 | 2 | 15,563 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,501.1 | 384.6 | 220 ms | 2.6 ms | 4.0 | 12 | 45,033 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,534.8 | 327.1 | 250 ms | 3.1 ms | 8.4 | 16 | 76,041 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,054.6 | 256.8 | 341 ms | 3.9 ms | 15.7 | 32 | 121,638 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,188.5 | 152.7 | 2,503 ms | 6.6 ms | 41.3 | 32 | 125,648 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,428.7 | 76.0 | 6,499 ms | 13.2 ms | 95.9 | 0 | 132,864 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,680.3 | 58.9 | 12,727 ms | 17.0 ms | 154.0 | 0 | 200,544 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 404.6 | 415.6 | 270 ms | 2.4 ms | 1.0 | 3 | 12,137 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 128.9 | 63.9 | 537 ms | 15.6 ms | 118.4 | 0 | 3,868 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,330.9 | 340.6 | 480 ms | 2.9 ms | 3.9 | 8 | 39,925 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,140.2 | 273.5 | 261 ms | 3.7 ms | 7.9 | 16 | 64,202 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,476.3 | 172.6 | 2,260 ms | 5.8 ms | 19.1 | 16 | 74,287 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,949.7 | 99.7 | 8,385 ms | 10.0 ms | 40.8 | 0 | 88,496 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,400.9 | 84.8 | 14,365 ms | 11.8 ms | 64.0 | 0 | 132,032 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 364.8 | 383.7 | 489 ms | 2.6 ms | 1.0 | 3 | 10,944 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 405.7 | 246.9 | 993 ms | 4.8 ms | 24.3 | 2 | 12,157 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,075.9 | 280.0 | 490 ms | 3.6 ms | 3.9 | 8 | 32,276 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,472.0 | 201.7 | 2,306 ms | 5.0 ms | 8.9 | 8 | 44,160 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,855.6 | 118.3 | 7,329 ms | 8.4 ms | 16.9 | 6 | 55,156 | 29.7 s |
| Sustained decode | 128K | 32 | Valid | 1,454.9 | 58.2 | 14,654 ms | 17.2 ms | 38.0 | 0 | 43,648 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 313.1 | 335.6 | 1,001 ms | 3.0 ms | 0.9 | 2 | 9,392 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 103.3 | 51.5 | 1,951 ms | 19.4 ms | 32.5 | 0 | 3,100 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 727.0 | 198.3 | 2,837 ms | 5.0 ms | 4.3 | 4 | 21,809 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 959.3 | 126.9 | 7,369 ms | 7.9 ms | 7.9 | 8 | 29,197 | 30.4 s |
| Sustained decode | 251K | 16 | Valid | 587.6 | 47.0 | 14,595 ms | 21.3 ms | 24.0 | 0 | 17,616 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.