Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.860929 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context41.2K198 ms TTFT
16K context47.7K338 ms TTFT
32K context43.7K734 ms TTFT
64K context38.2K1,678 ms TTFT
128K context28.9K4,432 ms TTFT
251K context19.3K13,015 ms TTFT
256K context19.1K13,401 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 250.884 ms TTFT | 452.3105 ms TTFT | 810.3123 ms TTFT | 1,259.7204 ms TTFT | 1,723.7350 ms TTFT | 3,058.3662 ms TTFT | 4,420.91,296 ms TTFT | 6,165.82,395 ms TTFT |
| 16K | 247.4105 ms TTFT | 115.8150 ms TTFT | 726.9215 ms TTFT | 1,156.5346 ms TTFT | 1,441.9700 ms TTFT | 2,706.31,344 ms TTFT | 3,360.52,491 ms TTFT | Over capacity |
| 32K | 242.0168 ms TTFT | 95.7260 ms TTFT | 589.4457 ms TTFT | 991.4684 ms TTFT | 1,174.61,687 ms TTFT | 1,903.33,305 ms TTFT | Over capacity | Over capacity |
| 64K | 231.5298 ms TTFT | 109.4520 ms TTFT | 627.7958 ms TTFT | 878.42,068 ms TTFT | 1,410.13,735 ms TTFT | 1,638.37,196 ms TTFT | Over capacity | Over capacity |
| 128K | 214.4529 ms TTFT | 77.0963 ms TTFT | 314.91,923 ms TTFT | 762.43,862 ms TTFT | 1,049.37,438 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 185.7990 ms TTFT | 81.81,944 ms TTFT | 357.93,848 ms TTFT | 585.67,297 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.2K | — | 198 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 47.7K | — | 338 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.7K | — | 734 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 38.2K | — | 1,678 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.9K | — | 4,432 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.3K | — | 13,015 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.1K | — | 13,401 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 250.8 | 252.4 | 84 ms | 4.0 ms | 1.0 | 2 | 7,524 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 452.3 | 220.8 | 105 ms | 4.6 ms | 2.1 | 2 | 13,570 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 810.3 | 203.5 | 123 ms | 4.9 ms | 4.0 | 4 | 24,306 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,259.7 | 158.2 | 204 ms | 6.3 ms | 8.1 | 8 | 37,785 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,723.7 | 107.2 | 350 ms | 9.3 ms | 21.3 | 0 | 51,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,058.3 | 95.6 | 662 ms | 10.5 ms | 32.0 | 0 | 91,744 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,420.9 | 67.9 | 1,296 ms | 14.7 ms | 71.5 | 0 | 132,640 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,165.8 | 45.5 | 2,395 ms | 22.0 ms | 128.0 | 0 | 184,960 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 247.4 | 249.2 | 105 ms | 4.0 ms | 1.0 | 2 | 7,423 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 115.8 | 56.1 | 150 ms | 17.8 ms | 88.0 | 0 | 3,474 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 726.9 | 179.5 | 215 ms | 5.6 ms | 4.7 | 4 | 21,808 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,156.5 | 148.5 | 346 ms | 6.7 ms | 9.2 | 8 | 34,695 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,441.9 | 90.5 | 700 ms | 11.0 ms | 23.5 | 0 | 43,248 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,706.3 | 83.6 | 1,344 ms | 12.0 ms | 35.7 | 0 | 81,184 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,360.5 | 52.7 | 2,491 ms | 19.0 ms | 83.7 | 0 | 100,800 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 242.0 | 243.7 | 168 ms | 4.1 ms | 1.0 | 1 | 7,260 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 95.7 | 48.0 | 260 ms | 20.8 ms | 66.0 | 0 | 2,868 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 589.4 | 160.3 | 457 ms | 6.4 ms | 14.6 | 4 | 17,680 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 991.4 | 133.7 | 684 ms | 7.5 ms | 10.7 | 8 | 29,754 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,174.6 | 74.9 | 1,687 ms | 13.4 ms | 24.0 | 0 | 35,232 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,903.3 | 60.6 | 3,305 ms | 16.5 ms | 45.3 | 0 | 57,088 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 231.5 | 233.9 | 298 ms | 4.3 ms | 1.0 | 1 | 6,945 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 109.4 | 54.8 | 520 ms | 18.2 ms | 33.6 | 0 | 3,282 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 627.7 | 163.5 | 958 ms | 6.1 ms | 4.5 | 4 | 18,829 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 878.4 | 111.3 | 2,068 ms | 9.0 ms | 10.5 | 0 | 26,352 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,410.1 | 92.8 | 3,735 ms | 10.8 ms | 16.0 | 0 | 42,305 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,638.3 | 53.6 | 7,196 ms | 18.7 ms | 36.4 | 0 | 49,152 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 214.4 | 218.2 | 529 ms | 4.6 ms | 1.0 | 1 | 6,431 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 77.0 | 38.8 | 963 ms | 25.8 ms | 34.0 | 0 | 2,310 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 314.9 | 75.0 | 1,923 ms | 13.3 ms | 18.7 | 0 | 9,448 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 762.4 | 97.3 | 3,862 ms | 10.3 ms | 8.9 | 0 | 22,872 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,049.3 | 68.2 | 7,438 ms | 14.7 ms | 16.2 | 0 | 31,472 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 185.7 | 192.1 | 990 ms | 5.2 ms | 1.0 | 1 | 5,571 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 81.8 | 41.4 | 1,944 ms | 24.2 ms | 17.9 | 0 | 2,455 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 357.9 | 92.4 | 3,848 ms | 10.8 ms | 5.4 | 0 | 10,737 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 585.6 | 77.5 | 7,297 ms | 12.9 ms | 8.0 | 0 | 17,568 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.