Speed test matrix
Qwen3.5-35B-A3B-GPTQ-Int4 on H200 NVL 141GB
sglang · 0.5.6.post2 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post2
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/Chevron7Locked/sglang
- Build
- cuda-13.2
- Commit
ce79bc7e3964
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang-pr-22121/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.869706 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context30.9K264 ms TTFT
16K context31.7K509 ms TTFT
32K context31.3K1,027 ms TTFT
64K context28.2K2,272 ms TTFT
128K context22.5K5,681 ms TTFT
251K context16.1K15,583 ms TTFT
256K context15.9K16,041 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 220.8254 ms TTFT | 411.5351 ms TTFT | 740.7Requests queued551 ms TTFT | 1,336.0247 ms TTFT | 2,101.5358 ms TTFT | 3,287.6716 ms TTFT | 4,748.81,257 ms TTFT | 6,237.82,464 ms TTFT |
| 16K | 218.1288 ms TTFT | 403.9389 ms TTFT | 724.5607 ms TTFT | 1,249.9696 ms TTFT | 1,925.4693 ms TTFT | 2,993.01,317 ms TTFT | 4,154.32,503 ms TTFT | 5,171.84,926 ms TTFT |
| 32K | 213.3382 ms TTFT | 390.8487 ms TTFT | 686.3689 ms TTFT | 1,147.4786 ms TTFT | 1,778.11,596 ms TTFT | 2,550.23,511 ms TTFT | 3,284.26,095 ms TTFT | Over capacity |
| 64K | 204.3587 ms TTFT | 366.1664 ms TTFT | 617.0967 ms TTFT | 1,035.21,714 ms TTFT | 1,462.03,454 ms TTFT | 1,910.06,947 ms TTFT | Over capacity | Over capacity |
| 128K | 189.7778 ms TTFT | 326.6881 ms TTFT | 521.21,782 ms TTFT | 826.73,386 ms TTFT | 1,055.06,665 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 165.81,401 ms TTFT | 269.81,777 ms TTFT | 428.53,441 ms TTFT | 575.76,703 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.9K | — | 264 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.7K | — | 509 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.3K | — | 1,027 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.2K | — | 2,272 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.5K | — | 5,681 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.1K | — | 15,583 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.9K | — | 16,041 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 220.8 | 222.8 | 254 ms | 4.5 ms | 1.0 | 1 | 6,624 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 411.5 | 207.8 | 351 ms | 4.8 ms | 2.0 | 2 | 12,344 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 740.7 | 187.9 | 551 ms | 5.3 ms | 3.9 | 4 | 22,220 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,336.0 | 169.2 | 247 ms | 5.9 ms | 8.0 | 8 | 40,081 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,101.5 | 136.7 | 358 ms | 7.3 ms | 16.0 | 16 | 63,040 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,287.6 | 103.6 | 716 ms | 9.7 ms | 32.0 | 0 | 98,624 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,748.8 | 74.8 | 1,257 ms | 13.4 ms | 64.0 | 0 | 142,464 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,237.8 | 49.2 | 2,464 ms | 20.3 ms | 128.0 | 0 | 187,136 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 218.1 | 220.3 | 288 ms | 4.5 ms | 1.0 | 1 | 6,544 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 403.9 | 204.2 | 389 ms | 4.9 ms | 1.9 | 2 | 12,116 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 724.5 | 183.8 | 607 ms | 5.4 ms | 4.0 | 4 | 21,732 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,249.9 | 159.0 | 696 ms | 6.3 ms | 7.8 | 8 | 37,496 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,925.4 | 123.3 | 693 ms | 8.1 ms | 15.9 | 16 | 57,760 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,993.0 | 94.1 | 1,317 ms | 10.6 ms | 32.0 | 0 | 89,792 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,154.3 | 65.3 | 2,503 ms | 15.3 ms | 64.0 | 0 | 124,608 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,171.8 | 40.7 | 4,926 ms | 24.6 ms | 128.0 | 0 | 155,136 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 213.3 | 216.1 | 382 ms | 4.6 ms | 1.0 | 1 | 6,397 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 390.8 | 198.2 | 487 ms | 5.0 ms | 2.0 | 2 | 11,724 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 686.3 | 174.6 | 689 ms | 5.7 ms | 3.9 | 4 | 20,588 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,147.4 | 147.5 | 786 ms | 6.8 ms | 7.9 | 8 | 34,424 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,778.1 | 111.2 | 1,596 ms | 9.0 ms | 16.0 | 0 | 53,344 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,550.2 | 80.0 | 3,511 ms | 12.5 ms | 32.0 | 0 | 76,512 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,284.2 | 51.3 | 6,095 ms | 19.5 ms | 64.0 | 0 | 98,496 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 204.3 | 208.1 | 587 ms | 4.8 ms | 1.0 | 1 | 6,129 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 366.1 | 186.9 | 664 ms | 5.4 ms | 2.0 | 2 | 10,982 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 617.0 | 158.2 | 967 ms | 6.3 ms | 3.9 | 4 | 18,508 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,035.2 | 128.5 | 1,714 ms | 7.8 ms | 7.9 | 8 | 29,336 | 28.3 s |
| Sustained decode | 64K | 16 | Valid | 1,462.0 | 93.4 | 3,454 ms | 10.7 ms | 16.0 | 0 | 43,856 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,910.0 | 62.3 | 6,947 ms | 16.1 ms | 32.0 | 0 | 57,280 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 189.7 | 194.8 | 778 ms | 5.1 ms | 1.0 | 1 | 5,691 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 326.6 | 167.3 | 881 ms | 6.0 ms | 2.0 | 2 | 9,801 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 521.2 | 137.1 | 1,782 ms | 7.3 ms | 3.8 | 4 | 15,634 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 826.7 | 103.6 | 3,386 ms | 9.7 ms | 8.0 | 0 | 24,800 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,055.0 | 67.0 | 6,665 ms | 14.9 ms | 16.0 | 0 | 31,648 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 165.8 | 173.8 | 1,401 ms | 5.8 ms | 1.0 | 1 | 4,974 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 269.8 | 142.2 | 1,777 ms | 7.0 ms | 1.9 | 2 | 8,091 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 428.5 | 107.2 | 3,441 ms | 9.3 ms | 4.0 | 0 | 12,852 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 575.7 | 73.7 | 6,703 ms | 13.6 ms | 8.0 | 0 | 17,264 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.