Speed test matrix
Qwen3.5-35B-A3B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.804911 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --attention-backend flashinfer --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context38.8K210 ms TTFT
16K context39.9K404 ms TTFT
32K context33.7K953 ms TTFT
64K context25.1K2,549 ms TTFT
128K context16.5K7,742 ms TTFT
251K context9,84625,472 ms TTFT
256K context9,69026,396 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 376.684 ms TTFT | 655.982 ms TTFT | 1,105.793 ms TTFT | 1,703.1159 ms TTFT | 2,778.6237 ms TTFT | 3,547.8696 ms TTFT | 5,490.61,266 ms TTFT | 6,559.1Requests queued2,512 ms TTFT |
| 16K | 370.4102 ms TTFT | 131.6199 ms TTFT | 818.2162 ms TTFT | 1,631.4268 ms TTFT | 2,531.1416 ms TTFT | 2,778.21,262 ms TTFT | 4,173.82,474 ms TTFT | Over capacity |
| 32K | 361.2172 ms TTFT | 142.9282 ms TTFT | 989.0307 ms TTFT | 1,366.8554 ms TTFT | 1,942.51,569 ms TTFT | 1,986.33,257 ms TTFT | Over capacity | Over capacity |
| 64K | 336.8285 ms TTFT | 252.5562 ms TTFT | 839.2809 ms TTFT | 1,238.61,652 ms TTFT | 1,382.83,947 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 283.7534 ms TTFT | 426.4839 ms TTFT | 689.61,545 ms TTFT | 883.54,597 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 229.61,005 ms TTFT | 312.82,213 ms TTFT | 459.34,285 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 38.8K | — | 210 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.9K | — | 404 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.7K | — | 953 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.1K | — | 2,549 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.5K | — | 7,742 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,846.0 | — | 25,472 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,690.0 | — | 26,396 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 376.6 | 380.8 | 84 ms | 2.6 ms | 1.0 | 3 | 11,299 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 655.9 | 329.1 | 82 ms | 3.0 ms | 1.9 | 4 | 19,675 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,105.7 | 280.8 | 93 ms | 3.6 ms | 3.9 | 8 | 33,172 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,703.1 | 213.3 | 159 ms | 4.7 ms | 8.8 | 8 | 51,087 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,778.6 | 174.4 | 237 ms | 5.7 ms | 15.9 | 16 | 83,354 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,547.8 | 111.2 | 696 ms | 9.0 ms | 39.7 | 0 | 106,426 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,490.6 | 87.9 | 1,266 ms | 11.4 ms | 64.0 | 0 | 164,591 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 6,559.1 | 54.0 | 2,512 ms | 18.5 ms | 126.8 | 63 | 196,784 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 370.4 | 376.9 | 102 ms | 2.7 ms | 1.0 | 3 | 11,111 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 131.6 | 66.5 | 199 ms | 15.0 ms | 66.4 | 0 | 3,946 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 818.2 | 213.9 | 162 ms | 5.0 ms | 23.4 | 4 | 24,545 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,631.4 | 202.1 | 268 ms | 4.9 ms | 8.3 | 8 | 48,943 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,531.1 | 160.2 | 416 ms | 6.2 ms | 16.4 | 16 | 75,920 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,778.2 | 88.4 | 1,262 ms | 11.3 ms | 45.2 | 0 | 83,340 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,173.8 | 66.4 | 2,474 ms | 15.1 ms | 76.0 | 0 | 125,226 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 361.2 | 366.2 | 172 ms | 2.7 ms | 1.0 | 2 | 10,834 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 142.9 | 70.4 | 282 ms | 14.2 ms | 59.6 | 0 | 4,287 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 989.0 | 246.2 | 307 ms | 4.1 ms | 3.9 | 7 | 29,581 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 1,366.8 | 179.4 | 554 ms | 5.6 ms | 9.4 | 8 | 41,000 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,942.5 | 127.1 | 1,569 ms | 7.9 ms | 18.9 | 14 | 58,374 | 30.1 s |
| Sustained decode | 32K | 32 | Valid | 1,986.3 | 66.2 | 3,257 ms | 15.1 ms | 46.0 | 0 | 59,582 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 336.8 | 341.2 | 285 ms | 2.9 ms | 1.0 | 2 | 10,103 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 252.5 | 119.2 | 562 ms | 8.4 ms | 30.5 | 1 | 7,506 | 29.7 s |
| Sustained decode | 64K | 4 | Valid | 839.2 | 216.6 | 809 ms | 4.6 ms | 4.4 | 4 | 25,171 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,238.6 | 162.6 | 1,652 ms | 6.2 ms | 7.9 | 8 | 37,155 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,382.8 | 92.3 | 3,947 ms | 10.8 ms | 22.1 | 0 | 41,467 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 283.7 | 294.5 | 534 ms | 3.4 ms | 1.0 | 2 | 8,511 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 426.4 | 214.8 | 839 ms | 4.7 ms | 4.0 | 2 | 12,790 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 689.6 | 179.5 | 1,545 ms | 5.6 ms | 3.9 | 4 | 20,683 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 883.5 | 113.7 | 4,597 ms | 8.8 ms | 9.2 | 2 | 26,174 | 29.6 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 229.6 | 240.1 | 1,005 ms | 4.2 ms | 1.0 | 1 | 6,886 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 312.8 | 176.5 | 2,213 ms | 5.7 ms | 3.2 | 2 | 9,383 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 459.3 | 119.7 | 4,285 ms | 8.4 ms | 4.7 | 1 | 13,365 | 29.1 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.