Speed test matrix
Qwen3.5-0.8B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.768973 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context76.1K107 ms TTFT
16K context95.6K169 ms TTFT
32K context99K324 ms TTFT
64K context91.8K698 ms TTFT
128K context72.9K1,756 ms TTFT
251K context50.8K4,938 ms TTFT
256K context51.4K4,978 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 628.277 ms TTFT | 1,209.574 ms TTFT | 2,213.877 ms TTFT | 3,769.078 ms TTFT | 5,996.275 ms TTFT | 8,039.088 ms TTFT | 8,561.6742 ms TTFT | 12.3K181 ms TTFT |
| 16K | 622.7100 ms TTFT | 571.9102 ms TTFT | 2,151.896 ms TTFT | 3,690.097 ms TTFT | 5,308.9103 ms TTFT | 7,538.3143 ms TTFT | 8,973.61,346 ms TTFT | 10.3KRequests queued4,866 ms TTFT |
| 32K | 582.7161 ms TTFT | 652.5167 ms TTFT | 2,115.3153 ms TTFT | 3,179.2214 ms TTFT | 4,497.4281 ms TTFT | 4,920.71,843 ms TTFT | 4,743.36,210 ms TTFT | 7,287.612,127 ms TTFT |
| 64K | 535.1304 ms TTFT | 133.2554 ms TTFT | 1,628.7293 ms TTFT | 2,568.4962 ms TTFT | 3,524.3721 ms TTFT | 3,520.37,553 ms TTFT | 3,024.215,221 ms TTFT | Over capacity |
| 128K | 490.8504 ms TTFT | 142.91,008 ms TTFT | 1,392.3Requests queued543 ms TTFT | 1,934.8973 ms TTFT | 1,945.47,565 ms TTFT | 1,584.714,891 ms TTFT | Over capacity | Over capacity |
| 251K | 395.9981 ms TTFT | 166.31,980 ms TTFT | 942.3Requests queued1,023 ms TTFT | 1,076.06,762 ms TTFT | 775.615,825 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 76.1K | — | 107 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 95.6K | — | 169 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 99K | — | 324 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 91.8K | — | 698 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 72.9K | — | 1,756 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 50.8K | — | 4,938 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 51.4K | — | 4,978 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 628.2 | 610.8 | 77 ms | 1.6 ms | 1.0 | 5 | 18,846 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,209.5 | 583.8 | 74 ms | 1.7 ms | 2.0 | 8 | 36,282 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 2,213.8 | 560.7 | 77 ms | 1.8 ms | 3.9 | 16 | 66,414 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,769.0 | 478.7 | 78 ms | 2.1 ms | 7.9 | 27 | 113,063 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,996.2 | 358.2 | 75 ms | 2.8 ms | 16.2 | 42 | 179,893 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,039.0 | 234.3 | 88 ms | 4.3 ms | 34.0 | 54 | 241,162 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 8,561.6 | 138.1 | 742 ms | 7.2 ms | 77.6 | 64 | 256,863 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.3K | 93.5 | 181 ms | 10.7 ms | 127.5 | 141 | 369,125 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 622.7 | 616.5 | 100 ms | 1.6 ms | 1.0 | 5 | 18,680 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 571.9 | 479.7 | 102 ms | 2.1 ms | 49.0 | 4 | 17,157 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 2,151.8 | 535.2 | 96 ms | 1.9 ms | 4.0 | 16 | 64,552 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,690.0 | 444.8 | 97 ms | 2.2 ms | 8.0 | 26 | 110,677 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 5,308.9 | 334.3 | 103 ms | 3.0 ms | 16.3 | 33 | 159,267 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,538.3 | 232.8 | 143 ms | 4.3 ms | 32.1 | 34 | 226,136 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,973.6 | 141.4 | 1,346 ms | 7.1 ms | 64.5 | 64 | 269,176 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 10.3K | 79.0 | 4,866 ms | 12.7 ms | 127.7 | 96 | 308,589 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 582.7 | 602.3 | 161 ms | 1.7 ms | 1.0 | 4 | 17,481 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 652.5 | 535.5 | 167 ms | 1.9 ms | 35.1 | 4 | 19,575 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 2,115.3 | 511.7 | 153 ms | 2.0 ms | 3.9 | 16 | 63,454 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,179.2 | 400.9 | 214 ms | 2.5 ms | 7.9 | 24 | 95,367 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,497.4 | 282.0 | 281 ms | 3.5 ms | 16.7 | 32 | 134,930 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,920.7 | 168.7 | 1,843 ms | 5.9 ms | 37.5 | 32 | 147,601 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,743.3 | 76.8 | 6,210 ms | 13.0 ms | 88.5 | 3 | 142,405 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,287.6 | 58.0 | 12,127 ms | 17.2 ms | 127.1 | 82 | 218,612 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 535.1 | 536.3 | 304 ms | 1.9 ms | 0.9 | 4 | 16,052 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 133.2 | 66.8 | 554 ms | 15.0 ms | 85.5 | 0 | 3,994 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,628.7 | 437.4 | 293 ms | 2.3 ms | 12.0 | 11 | 48,859 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,568.4 | 336.3 | 962 ms | 3.0 ms | 8.0 | 16 | 77,052 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,524.3 | 225.0 | 721 ms | 4.4 ms | 15.9 | 19 | 105,152 | 29.8 s |
| Sustained decode | 64K | 32 | Valid | 3,520.3 | 125.1 | 7,553 ms | 8.0 ms | 33.0 | 14 | 105,620 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,024.2 | 61.0 | 15,221 ms | 16.4 ms | 76.2 | 1 | 90,680 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 490.8 | 511.4 | 504 ms | 2.0 ms | 0.9 | 4 | 14,723 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 142.9 | 68.4 | 1,008 ms | 14.6 ms | 57.1 | 0 | 4,286 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 1,392.3 | 358.2 | 543 ms | 2.8 ms | 3.8 | 10 | 41,420 | 29.7 s |
| Sustained decode | 128K | 8 | Valid | 1,934.8 | 252.7 | 973 ms | 4.0 ms | 8.0 | 12 | 57,595 | 29.8 s |
| Sustained decode | 128K | 16 | Valid | 1,945.4 | 127.2 | 7,565 ms | 7.9 ms | 16.3 | 14 | 58,473 | 30.1 s |
| Sustained decode | 128K | 32 | Valid | 1,584.7 | 60.7 | 14,891 ms | 16.5 ms | 44.9 | 0 | 47,510 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 395.9 | 436.8 | 981 ms | 2.3 ms | 0.9 | 3 | 11,877 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 166.3 | 80.6 | 1,980 ms | 12.6 ms | 29.1 | 0 | 4,988 | 30.0 s |
| Sustained decode | 251K | 4 | Requests queued | 942.3 | 249.4 | 1,023 ms | 4.0 ms | 3.6 | 7 | 28,267 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,076.0 | 176.2 | 6,762 ms | 5.7 ms | 8.3 | 8 | 32,280 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 775.6 | 58.6 | 15,825 ms | 17.1 ms | 21.8 | 0 | 23,249 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.