Speed test matrix
Qwen3.5-35B-A3B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.851790 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-ssm-dtype bfloat16 --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context39.6K206 ms TTFT
16K context40.4K399 ms TTFT
32K context34.6K927 ms TTFT
64K context26.2K2,449 ms TTFT
128K context17.5K7,314 ms TTFT
251K context10.6K23,621 ms TTFT
256K context10.5K24,454 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 251.876 ms TTFT | 440.970 ms TTFT | 816.7104 ms TTFT | 1,298.6185 ms TTFT | 1,913.2367 ms TTFT | 3,282.1708 ms TTFT | 5,279.21,269 ms TTFT | 6,344.82,604 ms TTFT |
| 16K | 246.798 ms TTFT | 98.1140 ms TTFT | 607.6219 ms TTFT | 1,174.1339 ms TTFT | 1,780.2678 ms TTFT | 3,107.01,318 ms TTFT | 4,587.42,695 ms TTFT | Over capacity |
| 32K | 240.9160 ms TTFT | 185.6250 ms TTFT | 770.6395 ms TTFT | 1,157.1665 ms TTFT | 1,554.81,664 ms TTFT | 2,746.63,458 ms TTFT | Over capacity | Over capacity |
| 64K | 230.1293 ms TTFT | 328.6523 ms TTFT | 648.5714 ms TTFT | 967.01,818 ms TTFT | 1,275.53,621 ms TTFT | 1,975.97,164 ms TTFT | Over capacity | Over capacity |
| 128K | 209.7522 ms TTFT | 151.2976 ms TTFT | 608.41,425 ms TTFT | 787.53,598 ms TTFT | 1,256.37,237 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 179.91,016 ms TTFT | 255.01,937 ms TTFT | 430.34,910 ms TTFT | 687.17,248 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.6K | — | 206 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 40.4K | — | 399 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 34.6K | — | 927 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.2K | — | 2,449 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.5K | — | 7,314 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.6K | — | 23,621 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.5K | — | 24,454 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 251.8 | 253.6 | 76 ms | 3.9 ms | 1.0 | 2 | 7,554 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 440.9 | 216.8 | 70 ms | 4.7 ms | 2.2 | 2 | 13,228 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 816.7 | 203.9 | 104 ms | 4.9 ms | 4.0 | 4 | 24,499 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,298.6 | 162.6 | 185 ms | 6.2 ms | 8.1 | 8 | 38,954 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,913.2 | 119.4 | 367 ms | 8.4 ms | 20.2 | 0 | 57,392 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,282.1 | 103.6 | 708 ms | 9.6 ms | 32.0 | 0 | 98,464 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,279.2 | 81.3 | 1,269 ms | 12.3 ms | 65.1 | 0 | 158,378 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,344.8 | 50.1 | 2,604 ms | 19.9 ms | 172.8 | 0 | 190,368 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 246.7 | 248.5 | 98 ms | 4.0 ms | 1.0 | 2 | 7,401 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 98.1 | 49.3 | 140 ms | 20.3 ms | 130.0 | 0 | 2,942 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 607.6 | 168.3 | 219 ms | 6.2 ms | 32.2 | 4 | 18,229 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,174.1 | 152.4 | 339 ms | 6.6 ms | 10.2 | 8 | 35,219 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,780.2 | 111.6 | 678 ms | 9.0 ms | 21.6 | 0 | 53,408 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,107.0 | 98.6 | 1,318 ms | 10.1 ms | 32.0 | 0 | 93,216 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,587.4 | 71.5 | 2,695 ms | 14.0 ms | 68.8 | 0 | 137,600 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 240.9 | 242.2 | 160 ms | 4.1 ms | 1.0 | 1 | 7,226 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 185.6 | 88.1 | 250 ms | 11.3 ms | 50.7 | 0 | 5,568 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 770.6 | 192.5 | 395 ms | 5.2 ms | 4.0 | 4 | 23,117 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,157.1 | 148.1 | 665 ms | 6.8 ms | 8.6 | 8 | 34,712 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,554.8 | 99.1 | 1,664 ms | 10.1 ms | 22.8 | 0 | 46,640 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,746.6 | 86.5 | 3,458 ms | 11.6 ms | 32.4 | 0 | 82,400 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 230.1 | 232.5 | 293 ms | 4.3 ms | 1.0 | 1 | 6,904 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 328.6 | 179.0 | 523 ms | 5.8 ms | 12.2 | 2 | 9,857 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 648.5 | 170.3 | 714 ms | 5.9 ms | 4.8 | 4 | 19,456 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 967.0 | 123.6 | 1,818 ms | 8.1 ms | 9.8 | 7 | 29,052 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,275.5 | 83.7 | 3,621 ms | 11.9 ms | 23.0 | 0 | 38,256 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,975.9 | 64.9 | 7,164 ms | 15.4 ms | 38.1 | 0 | 59,279 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 209.7 | 213.5 | 522 ms | 4.7 ms | 1.0 | 1 | 6,288 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 151.2 | 73.8 | 976 ms | 13.6 ms | 28.4 | 0 | 4,536 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 608.4 | 158.0 | 1,425 ms | 6.3 ms | 4.1 | 4 | 18,251 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 787.5 | 102.2 | 3,598 ms | 9.8 ms | 10.7 | 0 | 23,624 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,256.3 | 87.3 | 7,237 ms | 11.5 ms | 16.0 | 0 | 37,680 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 179.9 | 186.1 | 1,016 ms | 5.4 ms | 1.0 | 1 | 5,396 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 255.0 | 121.3 | 1,937 ms | 8.2 ms | 6.6 | 1 | 7,649 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 430.3 | 114.1 | 4,910 ms | 8.8 ms | 4.8 | 0 | 12,908 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 687.1 | 95.2 | 7,248 ms | 10.5 ms | 8.0 | 0 | 20,608 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.