Speed test matrix
Qwen3.5-0.8B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861366 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion
Prefill
prompt tok/s by context length8K context70.1K116 ms TTFT
16K context98.7K163 ms TTFT
32K context94.1K341 ms TTFT
64K context90K712 ms TTFT
128K context76.6K1,671 ms TTFT
251K context54.9K4,568 ms TTFT
256K context56K4,564 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 573.677 ms TTFT | 1,105.650 ms TTFT | 2,133.492 ms TTFT | 4,037.2111 ms TTFT | 6,861.4138 ms TTFT | 10.2K313 ms TTFT | 13.8K1,109 ms TTFT | 16K2,495 ms TTFT |
| 16K | 560.598 ms TTFT | 443.2119 ms TTFT | 2,036.1130 ms TTFT | 3,744.9126 ms TTFT | 6,077.4131 ms TTFT | 7,809.1827 ms TTFT | 8,494.61,784 ms TTFT | 8,846.45,079 ms TTFT |
| 32K | 540.6147 ms TTFT | 686.2247 ms TTFT | 1,872.3231 ms TTFT | 3,259.4130 ms TTFT | 4,862.8242 ms TTFT | 5,897.32,283 ms TTFT | 6,446.56,188 ms TTFT | 8,140.812,809 ms TTFT |
| 64K | 502.0290 ms TTFT | 492.9515 ms TTFT | 1,605.4250 ms TTFT | 2,606.6257 ms TTFT | 3,486.21,993 ms TTFT | 3,972.74,220 ms TTFT | 2,748.315,234 ms TTFT | Over capacity |
| 128K | 445.2522 ms TTFT | 88.61,008 ms TTFT | 1,094.11,425 ms TTFT | 1,677.22,383 ms TTFT | 2,080.83,911 ms TTFT | 1,445.814,771 ms TTFT | Over capacity | Over capacity |
| 251K | 361.61,007 ms TTFT | 96.21,914 ms TTFT | 777.63,333 ms TTFT | 1,097.64,614 ms TTFT | 671.416,221 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 70.1K | — | 116 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 98.7K | — | 163 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 94.1K | — | 341 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 90K | — | 712 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 76.6K | — | 1,671 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 54.9K | — | 4,568 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 56K | — | 4,564 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 573.6 | 579.1 | 77 ms | 1.7 ms | 1.0 | 4 | 17,208 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,105.6 | 574.3 | 50 ms | 1.7 ms | 2.0 | 8 | 33,167 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 2,133.4 | 539.5 | 92 ms | 1.9 ms | 4.0 | 16 | 64,001 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 4,037.2 | 511.9 | 111 ms | 2.0 ms | 8.0 | 32 | 121,116 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 6,861.4 | 437.6 | 138 ms | 2.3 ms | 16.3 | 48 | 205,846 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 10.2K | 322.8 | 313 ms | 3.1 ms | 31.8 | 64 | 305,459 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 13.8K | 217.4 | 1,109 ms | 4.6 ms | 64.2 | 64 | 415,017 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 16K | 127.2 | 2,495 ms | 7.9 ms | 127.1 | 128 | 479,581 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 560.5 | 568.0 | 98 ms | 1.8 ms | 1.0 | 4 | 16,814 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 443.2 | 351.1 | 119 ms | 4.4 ms | 98.5 | 2 | 13,296 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 2,036.1 | 518.1 | 130 ms | 1.9 ms | 3.9 | 16 | 61,080 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,744.9 | 473.5 | 126 ms | 2.1 ms | 8.0 | 24 | 112,348 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 6,077.4 | 387.1 | 131 ms | 2.6 ms | 15.8 | 48 | 182,318 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,809.1 | 231.3 | 827 ms | 4.3 ms | 35.5 | 42 | 234,243 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,494.6 | 149.1 | 1,784 ms | 6.7 ms | 69.2 | 64 | 254,833 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,846.4 | 74.2 | 5,079 ms | 13.5 ms | 192.0 | 0 | 265,376 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 540.6 | 551.5 | 147 ms | 1.8 ms | 1.0 | 4 | 16,217 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 686.2 | 534.9 | 247 ms | 1.9 ms | 55.8 | 4 | 20,585 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,872.3 | 477.0 | 231 ms | 2.1 ms | 3.9 | 12 | 56,169 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,259.4 | 416.2 | 130 ms | 2.4 ms | 7.9 | 24 | 97,782 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,862.8 | 311.4 | 242 ms | 3.2 ms | 16.1 | 32 | 145,886 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,897.3 | 196.6 | 2,283 ms | 5.1 ms | 33.8 | 32 | 176,903 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,446.5 | 104.4 | 6,188 ms | 9.6 ms | 72.9 | 0 | 193,408 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,140.8 | 76.8 | 12,809 ms | 13.0 ms | 128.0 | 0 | 244,224 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 502.0 | 522.7 | 290 ms | 1.9 ms | 1.0 | 4 | 15,060 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 492.9 | 321.9 | 515 ms | 4.2 ms | 58.7 | 2 | 14,783 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,605.4 | 413.4 | 250 ms | 2.4 ms | 3.9 | 12 | 48,160 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,606.6 | 333.1 | 257 ms | 3.0 ms | 7.8 | 16 | 78,196 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,486.2 | 224.7 | 1,993 ms | 4.5 ms | 15.8 | 17 | 104,600 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,972.7 | 135.5 | 4,220 ms | 7.4 ms | 31.9 | 32 | 119,166 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,748.3 | 52.9 | 15,234 ms | 18.9 ms | 95.2 | 0 | 82,432 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 445.2 | 472.4 | 522 ms | 2.1 ms | 0.9 | 3 | 13,355 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 88.6 | 44.4 | 1,008 ms | 22.5 ms | 74.4 | 0 | 2,658 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,094.1 | 339.5 | 1,425 ms | 2.9 ms | 7.4 | 8 | 32,825 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,677.2 | 229.8 | 2,383 ms | 4.4 ms | 8.9 | 8 | 50,312 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,080.8 | 137.4 | 3,911 ms | 7.3 ms | 15.7 | 16 | 62,424 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,445.8 | 55.2 | 14,771 ms | 18.1 ms | 46.7 | 0 | 43,376 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 361.6 | 402.2 | 1,007 ms | 2.5 ms | 0.9 | 3 | 10,848 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 96.2 | 48.4 | 1,914 ms | 20.7 ms | 35.2 | 0 | 2,884 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 777.6 | 209.3 | 3,333 ms | 4.8 ms | 4.3 | 4 | 23,325 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,097.6 | 149.6 | 4,614 ms | 6.7 ms | 7.7 | 8 | 32,927 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 671.4 | 53.3 | 16,221 ms | 18.8 ms | 23.4 | 0 | 20,144 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.