Speed test matrix
Qwen3.5-0.8B-Base on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861366 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context77.8K105 ms TTFT
16K context104.3K155 ms TTFT
32K context103.1K312 ms TTFT
64K context91.8K698 ms TTFT
128K context78.6K1,627 ms TTFT
251K context55.4K4,525 ms TTFT
256K context56.5K4,528 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 573.675 ms TTFT | 1,098.585 ms TTFT | 2,135.790 ms TTFT | 4,048.4108 ms TTFT | 6,830.1136 ms TTFT | 10.2K391 ms TTFT | 13.9K1,168 ms TTFT | 15.7K2,446 ms TTFT |
| 16K | 560.594 ms TTFT | 445.1114 ms TTFT | 2,041.2135 ms TTFT | 3,745.7146 ms TTFT | 6,030.1158 ms TTFT | 7,763.1650 ms TTFT | 8,521.81,811 ms TTFT | 8,842.75,179 ms TTFT |
| 32K | 539.2152 ms TTFT | 688.7257 ms TTFT | 1,871.3228 ms TTFT | 3,258.9129 ms TTFT | 4,849.6286 ms TTFT | 5,861.82,408 ms TTFT | 6,432.86,658 ms TTFT | 8,147.113,149 ms TTFT |
| 64K | 503.3289 ms TTFT | 483.0411 ms TTFT | 1,606.3262 ms TTFT | 2,596.9404 ms TTFT | 3,481.42,095 ms TTFT | 4,017.14,150 ms TTFT | 2,758.815,464 ms TTFT | Over capacity |
| 128K | 448.0524 ms TTFT | 89.2981 ms TTFT | 1,094.81,416 ms TTFT | 1,659.32,383 ms TTFT | 2,085.13,993 ms TTFT | 1,449.914,920 ms TTFT | Over capacity | Over capacity |
| 251K | 361.8985 ms TTFT | 97.11,936 ms TTFT | 784.53,433 ms TTFT | 1,099.34,640 ms TTFT | 677.815,694 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 77.8K | — | 105 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 104.3K | — | 155 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 103.1K | — | 312 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 91.8K | — | 698 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 78.6K | — | 1,627 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 55.4K | — | 4,525 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 56.5K | — | 4,528 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 573.6 | 579.2 | 75 ms | 1.7 ms | 1.0 | 4 | 17,207 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,098.5 | 572.8 | 85 ms | 1.7 ms | 1.9 | 8 | 32,954 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 2,135.7 | 540.4 | 90 ms | 1.9 ms | 3.8 | 16 | 64,069 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 4,048.4 | 513.2 | 108 ms | 1.9 ms | 8.0 | 32 | 121,453 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 6,830.1 | 435.7 | 136 ms | 2.3 ms | 16.2 | 48 | 204,906 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 10.2K | 324.7 | 391 ms | 3.1 ms | 31.9 | 64 | 305,882 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 13.9K | 217.0 | 1,168 ms | 4.6 ms | 64.0 | 64 | 416,853 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.7K | 125.3 | 2,446 ms | 8.0 ms | 127.0 | 128 | 479,055 | 30.6 s |
| Sustained decode | 16K | 1 | Valid | 560.5 | 568.1 | 94 ms | 1.8 ms | 1.0 | 4 | 16,815 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 445.1 | 351.8 | 114 ms | 4.4 ms | 99.5 | 2 | 13,354 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 2,041.2 | 519.8 | 135 ms | 1.9 ms | 3.9 | 16 | 61,235 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,745.7 | 473.8 | 146 ms | 2.1 ms | 7.9 | 24 | 112,371 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 6,030.1 | 385.4 | 158 ms | 2.6 ms | 15.7 | 48 | 180,907 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,763.1 | 230.3 | 650 ms | 4.3 ms | 35.4 | 41 | 232,903 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,521.8 | 150.9 | 1,811 ms | 6.6 ms | 68.2 | 64 | 255,655 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,842.7 | 74.2 | 5,179 ms | 13.5 ms | 192.0 | 0 | 265,278 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 539.2 | 550.6 | 152 ms | 1.8 ms | 1.0 | 4 | 16,175 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 688.7 | 534.0 | 257 ms | 1.9 ms | 54.2 | 4 | 20,662 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,871.3 | 479.8 | 228 ms | 2.1 ms | 4.0 | 12 | 56,138 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,258.9 | 416.3 | 129 ms | 2.4 ms | 7.9 | 24 | 97,765 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,849.6 | 310.6 | 286 ms | 3.2 ms | 16.1 | 32 | 145,485 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,861.8 | 196.6 | 2,408 ms | 5.1 ms | 33.7 | 32 | 175,850 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,432.8 | 104.9 | 6,658 ms | 9.5 ms | 72.6 | 0 | 192,959 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,147.1 | 77.4 | 13,149 ms | 12.9 ms | 128.0 | 0 | 244,546 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 503.3 | 523.7 | 289 ms | 1.9 ms | 1.0 | 4 | 15,099 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 483.0 | 320.9 | 411 ms | 4.3 ms | 61.4 | 2 | 14,491 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,606.3 | 413.1 | 262 ms | 2.4 ms | 3.9 | 12 | 48,189 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,596.9 | 333.2 | 404 ms | 3.0 ms | 7.9 | 16 | 77,906 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,481.4 | 226.4 | 2,095 ms | 4.4 ms | 15.9 | 16 | 104,445 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,017.1 | 135.3 | 4,150 ms | 7.4 ms | 31.5 | 32 | 120,521 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,758.8 | 53.1 | 15,464 ms | 18.8 ms | 94.5 | 0 | 82,752 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 448.0 | 473.0 | 524 ms | 2.1 ms | 1.0 | 3 | 13,440 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 89.2 | 44.7 | 981 ms | 22.4 ms | 72.6 | 0 | 2,674 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,094.8 | 329.9 | 1,416 ms | 3.0 ms | 7.2 | 8 | 32,844 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,659.3 | 228.8 | 2,383 ms | 4.4 ms | 8.9 | 8 | 49,775 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,085.1 | 138.0 | 3,993 ms | 7.2 ms | 15.8 | 16 | 62,545 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,449.9 | 55.5 | 14,920 ms | 18.0 ms | 46.3 | 0 | 43,488 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 361.8 | 401.9 | 985 ms | 2.5 ms | 0.9 | 3 | 10,853 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 97.1 | 48.8 | 1,936 ms | 20.5 ms | 34.7 | 0 | 2,912 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 784.5 | 212.2 | 3,433 ms | 4.7 ms | 4.3 | 4 | 23,534 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,099.3 | 150.0 | 4,640 ms | 6.7 ms | 7.6 | 8 | 32,975 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 677.8 | 53.2 | 15,694 ms | 18.8 ms | 23.4 | 0 | 20,334 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.