Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on H200 NVL 141GB
sglang · 0.5.6.post2 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post2
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/Chevron7Locked/sglang
- Build
- cuda-13.2
- Commit
ce79bc7e3964
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang-pr-22121/bin/sglang" serve --model-path Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.875135 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context12.7K644 ms TTFT
16K context13.2K1,220 ms TTFT
32K context12.9K2,491 ms TTFT
64K context11.5K5,581 ms TTFT
128K context9,22813,869 ms TTFT
251K context6,62437,863 ms TTFT
256K context6,55139,044 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 139.1573 ms TTFT | 248.5580 ms TTFT | 423.21,359 ms TTFT | 679.02,393 ms TTFT | 1,061.71,157 ms TTFT | 1,588.9767 ms TTFT | 2,006.6Requests queued1,283 ms TTFT | Over capacity |
| 16K | 137.2630 ms TTFT | 245.8640 ms TTFT | 414.01,478 ms TTFT | 661.22,300 ms TTFT | 1,018.42,127 ms TTFT | 1,494.21,626 ms TTFT | Over capacity | Over capacity |
| 32K | 134.6748 ms TTFT | 245.61,068 ms TTFT | 401.31,754 ms TTFT | 629.22,039 ms TTFT | 943.21,701 ms TTFT | 1,331.03,027 ms TTFT | Over capacity | Over capacity |
| 64K | 130.6950 ms TTFT | 235.81,321 ms TTFT | 376.82,238 ms TTFT | 575.61,836 ms TTFT | 826.13,588 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 122.71,237 ms TTFT | 217.81,632 ms TTFT | 338.52,645 ms TTFT | 492.93,417 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 116.92,097 ms TTFT | 189.72,782 ms TTFT | 277.24,048 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.7K | — | 644 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.2K | — | 1,220 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.9K | — | 2,491 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.5K | — | 5,581 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,228.0 | — | 13,869 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,624.0 | — | 37,863 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,551.0 | — | 39,044 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 139.1 | 142.0 | 573 ms | 7.0 ms | 1.0 | 1 | 4,174 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 248.5 | 125.3 | 580 ms | 8.0 ms | 1.9 | 2 | 7,333 | 29.5 s |
| Sustained decode | 8K | 4 | Valid | 423.2 | 103.7 | 1,359 ms | 9.6 ms | 4.0 | 0 | 12,692 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 679.0 | 82.8 | 2,393 ms | 12.1 ms | 8.0 | 0 | 20,368 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,061.7 | 67.3 | 1,157 ms | 14.9 ms | 16.0 | 0 | 31,840 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,588.9 | 50.4 | 767 ms | 19.9 ms | 32.0 | 0 | 47,648 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 2,006.6 | 36.1 | 1,283 ms | 27.7 ms | 58.0 | 0 | 60,204 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 137.2 | 140.3 | 630 ms | 7.1 ms | 1.0 | 1 | 4,115 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 245.8 | 124.7 | 640 ms | 8.0 ms | 2.0 | 2 | 7,335 | 29.8 s |
| Sustained decode | 16K | 4 | Valid | 414.0 | 101.4 | 1,478 ms | 9.9 ms | 4.0 | 0 | 12,416 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 661.2 | 81.1 | 2,300 ms | 12.3 ms | 8.0 | 0 | 19,832 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,018.4 | 64.1 | 2,127 ms | 15.6 ms | 16.0 | 0 | 30,544 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,494.2 | 47.4 | 1,626 ms | 21.1 ms | 32.0 | 0 | 44,832 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 134.6 | 138.3 | 748 ms | 7.2 ms | 1.0 | 1 | 4,037 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 245.6 | 121.7 | 1,068 ms | 8.2 ms | 2.0 | 0 | 7,366 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 401.3 | 98.4 | 1,754 ms | 10.2 ms | 4.0 | 0 | 12,036 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 629.2 | 79.0 | 2,039 ms | 12.7 ms | 8.0 | 0 | 18,872 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 943.2 | 59.5 | 1,701 ms | 16.8 ms | 16.0 | 0 | 28,288 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,331.0 | 42.3 | 3,027 ms | 23.7 ms | 32.0 | 0 | 39,904 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 130.6 | 134.9 | 950 ms | 7.4 ms | 1.0 | 1 | 3,919 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 235.8 | 116.9 | 1,321 ms | 8.6 ms | 2.0 | 0 | 7,072 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 376.8 | 92.5 | 2,238 ms | 10.8 ms | 4.0 | 0 | 11,304 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 575.6 | 71.3 | 1,836 ms | 14.0 ms | 8.0 | 0 | 17,264 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 826.1 | 51.0 | 3,588 ms | 19.6 ms | 16.0 | 0 | 24,784 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 122.7 | 126.1 | 1,237 ms | 7.9 ms | 1.0 | 1 | 3,679 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 217.8 | 107.7 | 1,632 ms | 9.3 ms | 2.0 | 0 | 6,534 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 338.5 | 82.5 | 2,645 ms | 12.1 ms | 4.0 | 0 | 10,156 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 492.9 | 61.7 | 3,417 ms | 16.2 ms | 8.0 | 0 | 14,784 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 116.9 | 116.9 | 2,097 ms | 8.6 ms | 1.0 | 0 | 3,506 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 189.7 | 93.4 | 2,782 ms | 10.7 ms | 2.0 | 0 | 5,690 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 277.2 | 66.2 | 4,048 ms | 15.1 ms | 4.0 | 0 | 8,316 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.