Speed test matrix
Qwen3.5-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.845720 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --attention-backend flashinfer --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context13.3K611 ms TTFT
16K context12.8K1,264 ms TTFT
32K context10.7K2,996 ms TTFT
64K context8,4787,556 ms TTFT
128K context5,94721,520 ms TTFT
251K context3,75466,799 ms TTFT
256K context3,70968,960 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 184.196 ms TTFT | 360.9123 ms TTFT | 632.9853 ms TTFT | 1,023.7165 ms TTFT | 1,327.1343 ms TTFT | 2,263.4Requests queued2,417 ms TTFT | Over capacity | Over capacity |
| 16K | 180.0117 ms TTFT | 296.7171 ms TTFT | 579.0178 ms TTFT | 903.7340 ms TTFT | 1,566.3643 ms TTFT | 1,887.3Requests queued1,234 ms TTFT | Over capacity | Over capacity |
| 32K | 170.3186 ms TTFT | 283.9221 ms TTFT | 506.6404 ms TTFT | 702.7827 ms TTFT | 1,110.01,984 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 159.7315 ms TTFT | 179.8586 ms TTFT | 463.7985 ms TTFT | 676.71,844 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 139.8545 ms TTFT | 215.71,040 ms TTFT | 364.41,896 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 116.51,050 ms TTFT | 171.61,944 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.3K | — | 611 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.8K | — | 1,264 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.7K | — | 2,996 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,478.0 | — | 7,556 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,947.0 | — | 21,520 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,754.0 | — | 66,799 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,709.0 | — | 68,960 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 184.1 | 185.0 | 96 ms | 5.4 ms | 1.0 | 1 | 5,520 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 360.9 | 182.9 | 123 ms | 5.5 ms | 2.0 | 2 | 10,825 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 632.9 | 159.7 | 853 ms | 6.3 ms | 4.4 | 4 | 18,978 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,023.7 | 134.1 | 165 ms | 7.5 ms | 10.0 | 8 | 30,708 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,327.1 | 84.2 | 343 ms | 11.9 ms | 24.0 | 0 | 39,797 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 2,263.4 | 75.7 | 2,417 ms | 13.2 ms | 29.6 | 30 | 67,856 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 180.0 | 180.2 | 117 ms | 5.6 ms | 1.0 | 1 | 5,398 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 296.7 | 159.7 | 171 ms | 6.3 ms | 8.1 | 2 | 8,900 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 579.0 | 150.3 | 178 ms | 6.7 ms | 5.1 | 4 | 17,371 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 903.7 | 113.3 | 340 ms | 8.8 ms | 10.9 | 0 | 27,108 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,566.3 | 98.4 | 643 ms | 10.2 ms | 16.0 | 0 | 46,972 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 1,887.3 | 64.1 | 1,234 ms | 15.6 ms | 29.6 | 30 | 56,635 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 170.3 | 170.7 | 186 ms | 5.9 ms | 1.0 | 1 | 5,107 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 283.9 | 147.5 | 221 ms | 6.8 ms | 6.5 | 2 | 8,515 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 506.6 | 140.9 | 404 ms | 7.2 ms | 5.5 | 4 | 15,190 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 702.7 | 89.1 | 827 ms | 11.2 ms | 12.0 | 0 | 21,068 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,110.0 | 69.5 | 1,984 ms | 14.4 ms | 18.9 | 0 | 33,269 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 159.7 | 161.1 | 315 ms | 6.2 ms | 1.0 | 1 | 4,790 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 179.8 | 87.5 | 586 ms | 11.4 ms | 12.6 | 0 | 5,393 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 463.7 | 114.3 | 985 ms | 8.8 ms | 4.2 | 0 | 13,908 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 676.7 | 87.6 | 1,844 ms | 11.4 ms | 8.0 | 0 | 20,298 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 139.8 | 144.0 | 545 ms | 6.9 ms | 1.0 | 1 | 4,192 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 215.7 | 103.9 | 1,040 ms | 9.6 ms | 3.9 | 0 | 6,469 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 364.4 | 92.3 | 1,896 ms | 10.8 ms | 4.0 | 0 | 10,926 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 116.5 | 116.0 | 1,050 ms | 8.6 ms | 1.0 | 0 | 3,493 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 171.6 | 84.2 | 1,944 ms | 11.9 ms | 2.6 | 0 | 5,146 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.