Speed test matrix
Qwen3.5-4B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.856315 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --linear-attn-prefill-backend flashinfer --linear-attn-decode-backend flashinfer --mamba-scheduler-strategy extra_buffer --max-running-requests 32 --cuda-graph-max-bs-decode 32 --cuda-graph-backend-prefill tc_piecewise --enable-flashinfer-allreduce-fusion --attention-backend flashinfer --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.5-4B-DFlash --speculative-dflash-block-size 8 --speculative-draft-attention-backend fa4
Prefill
prompt tok/s by context length8K context37.1K219 ms TTFT
16K context38.2K422 ms TTFT
32K context34.9K919 ms TTFT
64K context27.6K2,325 ms TTFT
128K context19.5K6,579 ms TTFT
251K context12.3K20,439 ms TTFT
256K context12.2K21,030 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 454.286 ms TTFT | 842.383 ms TTFT | 1,508.884 ms TTFT | 2,422.296 ms TTFT | 3,607.1229 ms TTFT | 4,657.1Requests queued651 ms TTFT | Not run | Over capacity |
| 16K | 451.9104 ms TTFT | 797.4119 ms TTFT | 1,481.1108 ms TTFT | 2,247.9146 ms TTFT | 3,111.4413 ms TTFT | 3,997.3Requests queued1,271 ms TTFT | Not run | Over capacity |
| 32K | 442.5165 ms TTFT | 657.4170 ms TTFT | 1,357.0208 ms TTFT | 2,103.9218 ms TTFT | 2,386.91,008 ms TTFT | 3,284.5Requests queued3,152 ms TTFT | Over capacity | Over capacity |
| 64K | 404.2298 ms TTFT | 551.4455 ms TTFT | 1,151.4413 ms TTFT | 1,626.61,422 ms TTFT | 2,129.93,130 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 358.5548 ms TTFT | 258.1807 ms TTFT | 846.61,776 ms TTFT | 1,205.62,533 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 292.5998 ms TTFT | 327.91,542 ms TTFT | 610.53,449 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.1K | — | 219 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.2K | — | 422 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 34.9K | — | 919 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27.6K | — | 2,325 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.5K | — | 6,579 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.3K | — | 20,439 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.2K | — | 21,030 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 454.2 | 455.5 | 86 ms | 2.2 ms | 1.0 | 3 | 13,625 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 842.3 | 412.9 | 83 ms | 2.4 ms | 2.0 | 6 | 25,270 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,508.8 | 383.0 | 84 ms | 2.6 ms | 4.0 | 12 | 45,260 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,422.2 | 309.4 | 96 ms | 3.2 ms | 8.6 | 16 | 72,668 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,607.1 | 222.2 | 229 ms | 4.5 ms | 16.8 | 16 | 108,193 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 4,657.1 | 149.5 | 651 ms | 6.7 ms | 31.2 | 32 | 139,717 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 451.9 | 452.8 | 104 ms | 2.2 ms | 1.0 | 3 | 13,556 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 797.4 | 407.4 | 119 ms | 2.5 ms | 2.4 | 6 | 23,920 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,481.1 | 377.1 | 108 ms | 2.7 ms | 3.9 | 12 | 44,433 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,247.9 | 295.0 | 146 ms | 3.4 ms | 8.9 | 16 | 67,432 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,111.4 | 199.0 | 413 ms | 5.0 ms | 18.2 | 16 | 93,331 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 3,997.3 | 136.9 | 1,271 ms | 7.3 ms | 29.7 | 65 | 119,904 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 442.5 | 449.9 | 165 ms | 2.2 ms | 1.0 | 3 | 13,274 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 657.4 | 392.7 | 170 ms | 2.5 ms | 6.0 | 4 | 19,718 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,357.0 | 344.7 | 208 ms | 2.9 ms | 3.9 | 8 | 40,701 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,103.9 | 273.7 | 218 ms | 3.7 ms | 8.0 | 16 | 63,064 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,386.9 | 164.9 | 1,008 ms | 6.1 ms | 19.9 | 16 | 71,584 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 3,284.5 | 112.6 | 3,152 ms | 8.9 ms | 29.7 | 43 | 98,524 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 404.2 | 411.7 | 298 ms | 2.4 ms | 1.0 | 3 | 12,125 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 551.4 | 369.6 | 455 ms | 2.7 ms | 7.5 | 4 | 16,538 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,151.4 | 304.9 | 413 ms | 3.3 ms | 4.2 | 8 | 34,539 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,626.6 | 211.8 | 1,422 ms | 4.7 ms | 8.5 | 8 | 48,790 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,129.9 | 163.4 | 3,130 ms | 6.2 ms | 15.9 | 16 | 63,877 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 358.5 | 373.5 | 548 ms | 2.7 ms | 0.9 | 2 | 10,753 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 258.1 | 225.7 | 807 ms | 5.5 ms | 15.4 | 2 | 7,741 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 846.6 | 223.6 | 1,776 ms | 4.5 ms | 4.6 | 4 | 25,401 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,205.6 | 163.2 | 2,533 ms | 6.1 ms | 7.9 | 8 | 36,171 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 292.5 | 319.7 | 998 ms | 3.1 ms | 0.9 | 2 | 8,774 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 327.9 | 198.7 | 1,542 ms | 5.3 ms | 6.3 | 2 | 9,834 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 610.5 | 168.7 | 3,449 ms | 5.9 ms | 4.2 | 4 | 18,307 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.