Speed test matrix
Qwen3.5-35B-A3B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.841682 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context33.6K243 ms TTFT
16K context36.3K444 ms TTFT
32K context35.9K894 ms TTFT
64K context30.4K2,110 ms TTFT
128K context23.9K5,363 ms TTFT
251K context16.4K15,310 ms TTFT
256K context16.2K15,741 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 332.181 ms TTFT | 576.793 ms TTFT | 914.8107 ms TTFT | 1,505.5150 ms TTFT | 2,230.5246 ms TTFT | 3,354.2661 ms TTFT | 3,747.4Requests queued16,292 ms TTFT | Over capacity |
| 16K | 330.1109 ms TTFT | 248.7138 ms TTFT | 876.7158 ms TTFT | 1,466.9237 ms TTFT | 2,058.1405 ms TTFT | 3,232.21,255 ms TTFT | 3,427.8Requests queued14,752 ms TTFT | Over capacity |
| 32K | 333.4161 ms TTFT | 240.5262 ms TTFT | 866.5333 ms TTFT | 1,398.9539 ms TTFT | 1,839.31,551 ms TTFT | 2,946.22,969 ms TTFT | Over capacity | Over capacity |
| 64K | 316.6277 ms TTFT | 528.3503 ms TTFT | 787.6829 ms TTFT | 1,292.91,329 ms TTFT | 1,404.53,540 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 304.3526 ms TTFT | 484.7949 ms TTFT | 761.11,362 ms TTFT | 1,096.32,270 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 264.2983 ms TTFT | 217.31,910 ms TTFT | 618.42,793 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.6K | — | 243 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36.3K | — | 444 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 35.9K | — | 894 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.4K | — | 2,110 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.9K | — | 5,363 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.4K | — | 15,310 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.2K | — | 15,741 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 332.1 | 334.4 | 81 ms | 3.0 ms | 1.0 | 2 | 9,963 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 576.7 | 309.8 | 93 ms | 3.2 ms | 2.2 | 4 | 17,297 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 914.8 | 228.5 | 107 ms | 4.4 ms | 4.3 | 4 | 27,442 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,505.5 | 191.6 | 150 ms | 5.2 ms | 8.0 | 8 | 45,159 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,230.5 | 139.4 | 246 ms | 7.2 ms | 17.5 | 16 | 66,911 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,354.2 | 102.9 | 661 ms | 9.7 ms | 32.0 | 25 | 100,629 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 3,747.4 | 91.5 | 16,292 ms | 10.9 ms | 40.7 | 45 | 112,368 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 330.1 | 336.1 | 109 ms | 3.0 ms | 1.0 | 2 | 9,902 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 248.7 | 251.3 | 138 ms | 5.4 ms | 22.9 | 2 | 7,460 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 876.7 | 223.1 | 158 ms | 4.5 ms | 4.8 | 4 | 26,294 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,466.9 | 186.2 | 237 ms | 5.4 ms | 8.0 | 8 | 44,002 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,058.1 | 131.2 | 405 ms | 7.6 ms | 18.3 | 16 | 61,740 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,232.2 | 99.8 | 1,255 ms | 10.0 ms | 31.9 | 25 | 96,959 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 3,427.8 | 83.3 | 14,752 ms | 12.0 ms | 40.8 | 41 | 102,882 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 333.4 | 336.6 | 161 ms | 3.0 ms | 1.0 | 2 | 10,001 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 240.5 | 119.5 | 262 ms | 8.4 ms | 21.0 | 1 | 7,189 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 866.5 | 219.6 | 333 ms | 4.6 ms | 4.3 | 4 | 25,986 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,398.9 | 179.3 | 539 ms | 5.6 ms | 7.9 | 8 | 41,948 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,839.3 | 116.8 | 1,551 ms | 8.6 ms | 19.5 | 0 | 55,165 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,946.2 | 97.0 | 2,969 ms | 10.3 ms | 32.0 | 0 | 88,336 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 316.6 | 326.1 | 277 ms | 3.1 ms | 1.0 | 2 | 9,498 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 528.3 | 284.4 | 503 ms | 3.5 ms | 3.8 | 4 | 15,845 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 787.6 | 206.6 | 829 ms | 4.9 ms | 4.8 | 4 | 23,610 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,292.9 | 169.5 | 1,329 ms | 5.9 ms | 8.1 | 8 | 38,782 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,404.5 | 92.1 | 3,540 ms | 10.9 ms | 21.7 | 0 | 42,138 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 304.3 | 308.6 | 526 ms | 3.2 ms | 1.0 | 2 | 9,128 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 484.7 | 238.9 | 949 ms | 4.3 ms | 3.5 | 2 | 14,538 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 761.1 | 200.1 | 1,362 ms | 5.0 ms | 3.9 | 4 | 22,825 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,096.3 | 151.1 | 2,270 ms | 6.6 ms | 8.2 | 8 | 32,888 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 264.2 | 283.5 | 983 ms | 3.5 ms | 0.9 | 2 | 7,927 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 217.3 | 109.1 | 1,910 ms | 9.2 ms | 8.9 | 0 | 6,518 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 618.4 | 174.2 | 2,793 ms | 5.7 ms | 3.9 | 4 | 18,544 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.