Speed test matrix
Qwen3.5-35B-A3B-Base on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861674 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context34K240 ms TTFT
16K context36.5K442 ms TTFT
32K context35.6K902 ms TTFT
64K context31K2,064 ms TTFT
128K context24.5K5,227 ms TTFT
251K context17.2K14,611 ms TTFT
256K context17.2K14,900 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 214.384 ms TTFT | 320.5105 ms TTFT | 605.6138 ms TTFT | 910.5201 ms TTFT | 1,549.9350 ms TTFT | 2,293.6662 ms TTFT | 3,351.21,255 ms TTFT | Over capacity |
| 16K | 211.5104 ms TTFT | 169.0161 ms TTFT | 621.5209 ms TTFT | 863.3346 ms TTFT | 1,452.6691 ms TTFT | 1,995.11,306 ms TTFT | 2,962.42,553 ms TTFT | Over capacity |
| 32K | 207.0159 ms TTFT | 132.1267 ms TTFT | 549.8427 ms TTFT | 753.0830 ms TTFT | 1,287.21,728 ms TTFT | 1,528.63,153 ms TTFT | Over capacity | Over capacity |
| 64K | 199.8286 ms TTFT | 90.5514 ms TTFT | 370.31,013 ms TTFT | 828.81,812 ms TTFT | 1,147.53,711 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 185.4524 ms TTFT | 136.61,001 ms TTFT | 453.01,901 ms TTFT | 712.63,693 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 162.51,018 ms TTFT | 229.41,949 ms TTFT | 389.13,761 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34K | — | 240 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36.5K | — | 442 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 35.6K | — | 902 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 31K | — | 2,064 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.5K | — | 5,227 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.2K | — | 14,611 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.2K | — | 14,900 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 214.3 | 214.9 | 84 ms | 4.7 ms | 1.0 | 1 | 6,428 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 320.5 | 165.3 | 105 ms | 6.1 ms | 2.3 | 2 | 9,613 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 605.6 | 154.7 | 138 ms | 6.5 ms | 4.8 | 4 | 18,166 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 910.5 | 112.6 | 201 ms | 8.9 ms | 10.7 | 0 | 27,312 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,549.9 | 97.0 | 350 ms | 10.3 ms | 16.0 | 0 | 46,496 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,293.6 | 70.3 | 662 ms | 14.2 ms | 34.7 | 0 | 68,800 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,351.2 | 47.2 | 1,255 ms | 21.2 ms | 64.0 | 0 | 100,544 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 211.5 | 212.4 | 104 ms | 4.7 ms | 1.0 | 1 | 6,345 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 169.0 | 79.5 | 161 ms | 12.6 ms | 31.3 | 0 | 5,068 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 621.5 | 155.6 | 209 ms | 6.4 ms | 4.1 | 4 | 18,644 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 863.3 | 106.9 | 346 ms | 9.4 ms | 10.8 | 0 | 25,896 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,452.6 | 90.4 | 691 ms | 11.1 ms | 16.0 | 0 | 43,568 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,995.1 | 61.8 | 1,306 ms | 16.2 ms | 37.9 | 0 | 59,856 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,962.4 | 41.3 | 2,553 ms | 24.2 ms | 64.0 | 0 | 88,833 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 207.0 | 208.1 | 159 ms | 4.8 ms | 1.0 | 1 | 6,209 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 132.1 | 62.7 | 267 ms | 16.0 ms | 34.9 | 0 | 3,964 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 549.8 | 142.8 | 427 ms | 7.0 ms | 4.6 | 4 | 16,492 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 753.0 | 94.7 | 830 ms | 10.6 ms | 11.9 | 0 | 22,584 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,287.2 | 79.3 | 1,728 ms | 12.6 ms | 17.3 | 0 | 38,608 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,528.6 | 48.9 | 3,153 ms | 20.4 ms | 43.3 | 0 | 45,856 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 199.8 | 202.0 | 286 ms | 5.0 ms | 1.0 | 1 | 5,994 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 90.5 | 45.4 | 514 ms | 22.0 ms | 34.0 | 0 | 2,716 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 370.3 | 87.7 | 1,013 ms | 11.4 ms | 16.2 | 0 | 11,108 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 828.8 | 102.9 | 1,812 ms | 9.7 ms | 8.6 | 0 | 24,864 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,147.5 | 71.8 | 3,711 ms | 13.9 ms | 16.1 | 0 | 34,416 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 185.4 | 188.8 | 524 ms | 5.3 ms | 1.0 | 1 | 5,563 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 136.6 | 66.3 | 1,001 ms | 15.1 ms | 14.9 | 0 | 4,098 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 453.0 | 112.9 | 1,901 ms | 8.9 ms | 4.6 | 0 | 13,588 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 712.6 | 93.4 | 3,693 ms | 10.7 ms | 8.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 162.5 | 168.4 | 1,018 ms | 5.9 ms | 1.0 | 1 | 4,875 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 229.4 | 108.4 | 1,949 ms | 9.2 ms | 3.5 | 0 | 6,880 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 389.1 | 101.9 | 3,761 ms | 9.8 ms | 4.0 | 0 | 11,672 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.