Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.860929 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context42.3K193 ms TTFT
16K context47.5K339 ms TTFT
32K context44.7K718 ms TTFT
64K context38.4K1,668 ms TTFT
128K context29K4,414 ms TTFT
251K context19.2K13,047 ms TTFT
256K context19.1K13,397 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 250.984 ms TTFT | 453.1101 ms TTFT | 811.0144 ms TTFT | 1,272.2214 ms TTFT | 1,761.4379 ms TTFT | 3,123.9695 ms TTFT | 4,506.31,320 ms TTFT | 6,167.12,477 ms TTFT |
| 16K | 247.6103 ms TTFT | 113.8167 ms TTFT | 725.8227 ms TTFT | 1,151.7339 ms TTFT | 1,459.8694 ms TTFT | 2,734.71,309 ms TTFT | 3,396.32,632 ms TTFT | Over capacity |
| 32K | 242.4163 ms TTFT | 95.9295 ms TTFT | 604.3441 ms TTFT | 1,007.0706 ms TTFT | 1,184.61,708 ms TTFT | 1,952.53,415 ms TTFT | Over capacity | Over capacity |
| 64K | 231.5303 ms TTFT | 120.3591 ms TTFT | 640.2821 ms TTFT | 877.71,965 ms TTFT | 1,423.83,839 ms TTFT | 1,698.67,612 ms TTFT | Over capacity | Over capacity |
| 128K | 214.0515 ms TTFT | 77.21,043 ms TTFT | 342.91,994 ms TTFT | 791.64,027 ms TTFT | 1,071.07,926 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 186.0994 ms TTFT | 93.72,074 ms TTFT | 385.64,000 ms TTFT | 585.77,880 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.3K | — | 193 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 47.5K | — | 339 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.7K | — | 718 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 38.4K | — | 1,668 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29K | — | 4,414 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.2K | — | 13,047 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.1K | — | 13,397 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 250.9 | 252.5 | 84 ms | 4.0 ms | 1.0 | 2 | 7,528 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 453.1 | 221.1 | 101 ms | 4.5 ms | 2.1 | 2 | 13,592 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 811.0 | 204.1 | 144 ms | 4.9 ms | 4.0 | 4 | 24,328 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,272.2 | 159.3 | 214 ms | 6.3 ms | 8.1 | 8 | 38,165 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,761.4 | 109.6 | 379 ms | 9.1 ms | 21.1 | 0 | 52,832 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,123.9 | 98.5 | 695 ms | 10.2 ms | 32.0 | 0 | 93,728 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,506.3 | 69.1 | 1,320 ms | 14.5 ms | 70.4 | 0 | 135,168 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,167.1 | 45.6 | 2,477 ms | 21.9 ms | 128.0 | 0 | 184,960 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 247.6 | 249.5 | 103 ms | 4.0 ms | 1.0 | 2 | 7,428 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 113.8 | 55.3 | 167 ms | 18.1 ms | 90.3 | 0 | 3,414 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 725.8 | 179.0 | 227 ms | 5.7 ms | 4.7 | 4 | 21,788 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,151.7 | 148.1 | 339 ms | 6.8 ms | 9.2 | 8 | 34,551 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,459.8 | 91.6 | 694 ms | 10.9 ms | 23.4 | 0 | 43,792 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,734.7 | 84.3 | 1,309 ms | 11.9 ms | 35.2 | 0 | 82,047 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,396.3 | 53.3 | 2,632 ms | 18.8 ms | 83.0 | 0 | 101,888 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 242.4 | 243.9 | 163 ms | 4.1 ms | 1.0 | 1 | 7,273 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 95.9 | 48.1 | 295 ms | 20.8 ms | 66.0 | 0 | 2,876 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 604.3 | 161.8 | 441 ms | 6.4 ms | 12.7 | 4 | 18,128 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,007.0 | 135.6 | 706 ms | 7.4 ms | 10.6 | 8 | 30,210 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,184.6 | 75.5 | 1,708 ms | 13.2 ms | 24.0 | 0 | 35,536 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,952.5 | 62.0 | 3,415 ms | 16.1 ms | 44.5 | 0 | 58,560 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 231.5 | 233.9 | 303 ms | 4.3 ms | 1.0 | 1 | 6,944 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 120.3 | 59.7 | 591 ms | 16.8 ms | 32.2 | 0 | 3,608 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 640.2 | 165.5 | 821 ms | 6.1 ms | 4.4 | 4 | 19,205 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 877.7 | 111.2 | 1,965 ms | 9.0 ms | 10.5 | 0 | 26,328 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,423.8 | 93.9 | 3,839 ms | 10.7 ms | 16.0 | 0 | 42,704 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,698.6 | 55.2 | 7,612 ms | 18.1 ms | 35.4 | 0 | 50,944 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 214.0 | 217.9 | 515 ms | 4.6 ms | 1.0 | 1 | 6,420 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 77.2 | 38.9 | 1,043 ms | 25.7 ms | 34.0 | 0 | 2,316 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 342.9 | 81.3 | 1,994 ms | 12.3 ms | 15.9 | 0 | 10,288 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 791.6 | 100.8 | 4,027 ms | 9.9 ms | 8.5 | 0 | 23,744 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,071.0 | 72.4 | 7,926 ms | 13.8 ms | 16.0 | 0 | 32,128 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 186.0 | 192.3 | 994 ms | 5.2 ms | 1.0 | 1 | 5,576 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 93.7 | 46.8 | 2,074 ms | 21.4 ms | 16.6 | 0 | 2,811 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 385.6 | 98.8 | 4,000 ms | 10.1 ms | 5.0 | 0 | 11,564 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 585.7 | 81.2 | 7,880 ms | 12.3 ms | 8.0 | 0 | 17,568 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.