Speed test matrix
Qwen3.6-35B-A3B-FP8-dynamic on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861046 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context43.6K187 ms TTFT
16K context46.7K345 ms TTFT
32K context44.5K721 ms TTFT
64K context37.3K1,719 ms TTFT
128K context28.3K4,528 ms TTFT
251K context19K13,228 ms TTFT
256K context19K13,460 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 247.985 ms TTFT | 434.7114 ms TTFT | 788.8123 ms TTFT | 1,223.7211 ms TTFT | 1,686.4362 ms TTFT | 3,086.3658 ms TTFT | 4,386.91,269 ms TTFT | 6,119.12,482 ms TTFT |
| 16K | 244.9105 ms TTFT | 116.9128 ms TTFT | 707.4206 ms TTFT | 1,131.4333 ms TTFT | 1,422.3660 ms TTFT | 2,683.31,267 ms TTFT | 3,308.82,456 ms TTFT | Over capacity |
| 32K | 238.5162 ms TTFT | 95.6253 ms TTFT | 569.4430 ms TTFT | 964.7825 ms TTFT | 1,158.81,578 ms TTFT | 1,876.63,237 ms TTFT | Over capacity | Over capacity |
| 64K | 229.3305 ms TTFT | 108.6549 ms TTFT | 613.8757 ms TTFT | 849.71,879 ms TTFT | 1,408.83,776 ms TTFT | 1,639.67,183 ms TTFT | Over capacity | Over capacity |
| 128K | 210.2560 ms TTFT | 77.4996 ms TTFT | 314.81,847 ms TTFT | 756.93,742 ms TTFT | 1,044.17,669 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 182.31,133 ms TTFT | 82.51,984 ms TTFT | 357.03,971 ms TTFT | 585.47,935 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.6K | — | 187 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.7K | — | 345 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.5K | — | 721 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 37.3K | — | 1,719 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.3K | — | 4,528 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19K | — | 13,228 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19K | — | 13,460 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 247.9 | 249.6 | 85 ms | 4.0 ms | 1.0 | 2 | 7,435 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 434.7 | 213.8 | 114 ms | 4.7 ms | 2.1 | 2 | 13,040 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 788.8 | 196.3 | 123 ms | 5.1 ms | 4.0 | 4 | 23,664 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,223.7 | 155.0 | 211 ms | 6.5 ms | 8.4 | 8 | 36,706 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,686.4 | 105.1 | 362 ms | 9.5 ms | 21.9 | 0 | 50,592 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,086.3 | 95.7 | 658 ms | 10.4 ms | 32.0 | 0 | 92,576 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,386.9 | 67.4 | 1,269 ms | 14.8 ms | 71.5 | 0 | 131,616 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,119.1 | 45.8 | 2,482 ms | 21.8 ms | 128.0 | 0 | 183,583 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 244.9 | 245.9 | 105 ms | 4.1 ms | 1.0 | 2 | 7,327 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 116.9 | 56.7 | 128 ms | 17.6 ms | 86.3 | 0 | 3,508 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 707.4 | 175.2 | 206 ms | 5.8 ms | 4.7 | 4 | 21,220 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,131.4 | 146.2 | 333 ms | 6.9 ms | 9.5 | 8 | 33,943 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,422.3 | 89.3 | 660 ms | 11.2 ms | 23.7 | 0 | 42,672 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,683.3 | 82.8 | 1,267 ms | 12.1 ms | 36.0 | 0 | 80,480 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,308.8 | 52.0 | 2,456 ms | 19.2 ms | 84.3 | 0 | 99,263 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 238.5 | 240.0 | 162 ms | 4.2 ms | 1.0 | 1 | 7,153 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 95.6 | 48.0 | 253 ms | 20.8 ms | 66.0 | 0 | 2,866 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 569.4 | 156.4 | 430 ms | 6.6 ms | 15.4 | 4 | 17,080 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 964.7 | 122.2 | 825 ms | 8.2 ms | 11.0 | 7 | 28,778 | 29.8 s |
| Sustained decode | 32K | 16 | Valid | 1,158.8 | 73.9 | 1,578 ms | 13.5 ms | 23.0 | 0 | 34,752 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,876.6 | 59.7 | 3,237 ms | 16.8 ms | 46.2 | 0 | 56,288 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 229.3 | 231.7 | 305 ms | 4.3 ms | 1.0 | 1 | 6,879 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 108.6 | 54.5 | 549 ms | 18.4 ms | 33.7 | 0 | 3,256 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 613.8 | 160.0 | 757 ms | 6.3 ms | 4.5 | 4 | 18,412 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 849.7 | 107.8 | 1,879 ms | 9.3 ms | 10.8 | 0 | 25,488 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,408.8 | 92.5 | 3,776 ms | 10.8 ms | 16.0 | 0 | 42,256 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,639.6 | 53.6 | 7,183 ms | 18.6 ms | 36.8 | 0 | 49,184 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 210.2 | 214.4 | 560 ms | 4.7 ms | 1.0 | 1 | 6,307 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 77.4 | 38.9 | 996 ms | 25.7 ms | 34.0 | 0 | 2,320 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 314.8 | 75.0 | 1,847 ms | 13.3 ms | 18.5 | 0 | 9,444 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 756.9 | 96.5 | 3,742 ms | 10.4 ms | 9.0 | 0 | 22,704 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,044.1 | 68.1 | 7,669 ms | 14.7 ms | 16.2 | 0 | 31,312 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 182.3 | 189.1 | 1,133 ms | 5.3 ms | 1.0 | 1 | 5,469 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 82.5 | 41.7 | 1,984 ms | 24.0 ms | 17.9 | 0 | 2,474 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 357.0 | 92.3 | 3,971 ms | 10.8 ms | 5.4 | 0 | 10,708 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 585.4 | 78.8 | 7,935 ms | 12.7 ms | 8.0 | 0 | 17,560 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.