Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.837062 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context42.4K192 ms TTFT
16K context46.5K347 ms TTFT
32K context43.7K734 ms TTFT
64K context36.4K1,760 ms TTFT
128K context27.1K4,716 ms TTFT
251K context17.9K14,021 ms TTFT
256K context17.7K14,411 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 394.9100 ms TTFT | 665.198 ms TTFT | 1,130.896 ms TTFT | 1,732.7151 ms TTFT | 2,798.9245 ms TTFT | 3,602.8689 ms TTFT | Not run | Not run |
| 16K | 380.7110 ms TTFT | 652.8122 ms TTFT | 1,090.2116 ms TTFT | 1,674.4272 ms TTFT | 2,654.0390 ms TTFT | 3,174.01,251 ms TTFT | Not run | Over capacity |
| 32K | 380.0174 ms TTFT | 632.1206 ms TTFT | 1,050.7234 ms TTFT | 1,538.5607 ms TTFT | 2,319.61,176 ms TTFT | 2,461.03,072 ms TTFT | Not run | Over capacity |
| 64K | 365.9307 ms TTFT | 477.8546 ms TTFT | 991.2669 ms TTFT | 1,353.81,643 ms TTFT | 1,773.03,669 ms TTFT | 2,679.47,472 ms TTFT | Over capacity | Over capacity |
| 128K | 338.3537 ms TTFT | 495.3998 ms TTFT | 864.71,208 ms TTFT | 1,243.12,765 ms TTFT | 1,299.07,201 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 295.91,015 ms TTFT | 430.11,917 ms TTFT | 691.4Requests queued2,788 ms TTFT | 932.67,226 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.4K | — | 192 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.5K | — | 347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.7K | — | 734 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.4K | — | 1,760 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 27.1K | — | 4,716 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.9K | — | 14,021 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.7K | — | 14,411 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 394.9 | 399.2 | 100 ms | 2.5 ms | 1.0 | 3 | 11,846 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 665.1 | 338.3 | 98 ms | 3.0 ms | 2.0 | 4 | 19,953 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,130.8 | 286.2 | 96 ms | 3.5 ms | 3.9 | 8 | 33,923 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,732.7 | 218.0 | 151 ms | 4.6 ms | 8.8 | 8 | 51,970 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,798.9 | 174.6 | 245 ms | 5.7 ms | 16.0 | 16 | 83,939 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,602.8 | 112.6 | 689 ms | 8.9 ms | 39.3 | 0 | 108,100 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 380.7 | 384.8 | 110 ms | 2.6 ms | 1.0 | 3 | 11,421 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 652.8 | 329.1 | 122 ms | 3.0 ms | 2.0 | 4 | 19,580 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,090.2 | 276.7 | 116 ms | 3.6 ms | 4.0 | 8 | 32,705 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,674.4 | 208.9 | 272 ms | 4.8 ms | 9.0 | 8 | 50,225 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,654.0 | 165.9 | 390 ms | 6.0 ms | 16.1 | 16 | 79,613 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,174.0 | 99.7 | 1,251 ms | 10.0 ms | 42.7 | 0 | 95,207 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 380.0 | 386.7 | 174 ms | 2.6 ms | 1.0 | 3 | 11,400 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 632.1 | 318.4 | 206 ms | 3.1 ms | 2.5 | 4 | 18,962 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,050.7 | 269.5 | 234 ms | 3.7 ms | 4.1 | 8 | 31,518 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,538.5 | 198.0 | 607 ms | 5.1 ms | 9.4 | 8 | 46,147 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,319.6 | 151.1 | 1,176 ms | 6.6 ms | 17.1 | 16 | 69,584 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,461.0 | 81.6 | 3,072 ms | 12.3 ms | 48.0 | 0 | 73,796 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 365.9 | 383.6 | 307 ms | 2.6 ms | 1.0 | 3 | 10,976 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 477.8 | 245.8 | 546 ms | 4.4 ms | 12.2 | 2 | 14,333 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 991.2 | 261.7 | 669 ms | 3.8 ms | 3.8 | 8 | 29,733 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,353.8 | 180.2 | 1,643 ms | 5.6 ms | 9.8 | 8 | 40,600 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,773.0 | 115.7 | 3,669 ms | 8.6 ms | 19.8 | 1 | 52,725 | 29.7 s |
| Sustained decode | 64K | 32 | Valid | 2,679.4 | 95.5 | 7,472 ms | 10.5 ms | 33.0 | 0 | 80,343 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 338.3 | 348.8 | 537 ms | 2.9 ms | 0.9 | 2 | 10,147 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 495.3 | 239.0 | 998 ms | 4.3 ms | 5.7 | 2 | 14,858 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 864.7 | 225.4 | 1,208 ms | 4.4 ms | 3.9 | 4 | 25,935 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,243.1 | 169.7 | 2,765 ms | 5.9 ms | 7.8 | 8 | 37,290 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,299.0 | 88.1 | 7,201 ms | 11.3 ms | 20.7 | 0 | 38,965 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 295.9 | 319.3 | 1,015 ms | 3.1 ms | 0.9 | 2 | 8,877 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 430.1 | 220.7 | 1,917 ms | 4.6 ms | 4.3 | 2 | 12,901 | 30.0 s |
| Sustained decode | 251K | 4 | Requests queued | 691.4 | 190.1 | 2,788 ms | 5.3 ms | 3.8 | 4 | 20,738 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 932.6 | 128.3 | 7,226 ms | 7.8 ms | 8.3 | 4 | 27,366 | 29.3 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.