Speed test matrix
Qwen3.5-122B-A10B-FP8 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen --trust-remote-code --tp 2 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.803667 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --attention-backend flashinfer --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context27.7K294 ms TTFT
16K context28.4K568 ms TTFT
32K context24.7K1,301 ms TTFT
64K context19.1K3,355 ms TTFT
128K context13K9,857 ms TTFT
251K context7,99531,369 ms TTFT
256K context7,86932,506 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 332.990 ms TTFT | 546.594 ms TTFT | 854.6110 ms TTFT | 1,389.8160 ms TTFT | 2,025.8253 ms TTFT | 2,764.5675 ms TTFT | 4,794.24,182 ms TTFT | 4,752.0Requests queued20,815 ms TTFT |
| 16K | 320.4113 ms TTFT | 254.8194 ms TTFT | 841.5170 ms TTFT | 1,347.0259 ms TTFT | 1,908.0663 ms TTFT | 2,803.01,288 ms TTFT | 4,457.82,443 ms TTFT | Over capacity |
| 32K | 315.1186 ms TTFT | 337.2256 ms TTFT | 803.8418 ms TTFT | 1,287.4641 ms TTFT | 1,746.31,787 ms TTFT | 2,873.73,297 ms TTFT | Over capacity | Over capacity |
| 64K | 288.1305 ms TTFT | 453.5528 ms TTFT | 796.0768 ms TTFT | 1,147.71,646 ms TTFT | 1,438.03,618 ms TTFT | 2,417.17,219 ms TTFT | Over capacity | Over capacity |
| 128K | 269.0525 ms TTFT | 306.5981 ms TTFT | 648.31,623 ms TTFT | 938.93,749 ms TTFT | 1,226.87,233 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 237.7999 ms TTFT | 344.81,576 ms TTFT | 533.52,899 ms TTFT | 647.57,498 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.7K | — | 294 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.4K | — | 568 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.7K | — | 1,301 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.1K | — | 3,355 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13K | — | 9,857 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,995.0 | — | 31,369 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,869.0 | — | 32,506 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 332.9 | 336.2 | 90 ms | 3.0 ms | 1.0 | 2 | 9,985 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 546.5 | 278.9 | 94 ms | 3.6 ms | 2.0 | 4 | 16,382 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 854.6 | 215.0 | 110 ms | 4.7 ms | 4.4 | 4 | 25,637 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,389.8 | 174.2 | 160 ms | 5.7 ms | 8.0 | 8 | 41,677 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,025.8 | 128.1 | 253 ms | 7.8 ms | 19.0 | 16 | 60,759 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,764.5 | 87.4 | 675 ms | 11.4 ms | 48.0 | 0 | 82,840 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,794.2 | 74.0 | 4,182 ms | 13.5 ms | 64.0 | 0 | 143,842 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 4,752.0 | 68.3 | 20,815 ms | 14.6 ms | 70.2 | 71 | 142,551 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 320.4 | 323.3 | 113 ms | 3.1 ms | 1.0 | 2 | 9,601 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 254.8 | 125.1 | 194 ms | 8.0 ms | 21.5 | 1 | 7,642 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 841.5 | 214.5 | 170 ms | 4.7 ms | 4.4 | 4 | 25,240 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,347.0 | 167.5 | 259 ms | 6.0 ms | 8.0 | 8 | 40,408 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,908.0 | 120.4 | 663 ms | 8.3 ms | 19.7 | 6 | 57,236 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,803.0 | 89.2 | 1,288 ms | 11.2 ms | 38.0 | 0 | 84,088 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,457.8 | 69.1 | 2,443 ms | 14.5 ms | 64.0 | 0 | 133,602 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 315.1 | 317.9 | 186 ms | 3.1 ms | 0.9 | 2 | 9,447 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 337.2 | 205.1 | 256 ms | 5.4 ms | 25.3 | 2 | 10,112 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 803.8 | 202.7 | 418 ms | 4.9 ms | 4.5 | 4 | 24,114 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,287.4 | 162.5 | 641 ms | 6.2 ms | 8.2 | 8 | 38,613 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,746.3 | 110.9 | 1,787 ms | 9.0 ms | 20.7 | 0 | 52,390 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,873.7 | 93.8 | 3,297 ms | 10.7 ms | 32.0 | 0 | 86,165 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 288.1 | 296.1 | 305 ms | 3.4 ms | 1.0 | 2 | 8,633 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 453.5 | 220.3 | 528 ms | 4.6 ms | 5.3 | 2 | 13,603 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 796.0 | 202.1 | 768 ms | 4.9 ms | 4.0 | 4 | 23,872 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,147.7 | 149.2 | 1,646 ms | 6.7 ms | 8.1 | 8 | 34,426 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,438.0 | 94.7 | 3,618 ms | 10.6 ms | 22.8 | 0 | 43,116 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,417.1 | 80.6 | 7,219 ms | 12.4 ms | 32.4 | 0 | 72,459 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 269.0 | 278.4 | 525 ms | 3.6 ms | 0.9 | 2 | 8,069 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 306.5 | 180.8 | 981 ms | 5.9 ms | 15.8 | 2 | 9,195 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 648.3 | 171.7 | 1,623 ms | 5.8 ms | 4.8 | 4 | 19,444 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 938.9 | 121.1 | 3,749 ms | 8.3 ms | 9.5 | 3 | 28,067 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 1,226.8 | 84.6 | 7,233 ms | 11.8 ms | 18.9 | 0 | 36,774 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 237.7 | 246.9 | 999 ms | 4.1 ms | 1.0 | 1 | 7,129 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 344.8 | 180.9 | 1,576 ms | 5.6 ms | 4.8 | 2 | 10,345 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 533.5 | 154.2 | 2,899 ms | 6.5 ms | 4.4 | 4 | 16,002 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 647.5 | 89.9 | 7,498 ms | 11.1 ms | 11.3 | 0 | 19,427 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.