Speed test matrix
Qwen3.5-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.854925 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context13.7K593 ms TTFT
16K context13.1K1,235 ms TTFT
32K context11.3K2,849 ms TTFT
64K context8,7907,288 ms TTFT
128K context6,13820,851 ms TTFT
251K context3,92763,861 ms TTFT
256K context3,88565,845 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 88.2Requests queued90 ms TTFT | 136.3141 ms TTFT | 324.71,629 ms TTFT | 610.3228 ms TTFT | 1,064.8349 ms TTFT | 1,582.8662 ms TTFT | Over capacity | Over capacity |
| 16K | 87.4107 ms TTFT | 98.5173 ms TTFT | 292.3216 ms TTFT | 574.0353 ms TTFT | 936.0659 ms TTFT | 1,314.41,260 ms TTFT | Over capacity | Over capacity |
| 32K | 85.2177 ms TTFT | 81.4288 ms TTFT | 185.0415 ms TTFT | 486.5788 ms TTFT | 725.61,616 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 81.7299 ms TTFT | 104.0603 ms TTFT | 240.6960 ms TTFT | 410.21,949 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 75.4528 ms TTFT | 101.71,046 ms TTFT | 211.11,935 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 65.91,013 ms TTFT | 92.71,988 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.7K | — | 593 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.1K | — | 1,235 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.3K | — | 2,849 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,790.0 | — | 7,288 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,138.0 | — | 20,851 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,927.0 | — | 63,861 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,885.0 | — | 65,845 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 88.2 | 88.2 | 90 ms | 11.3 ms | 1.0 | 0 | 2,645 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 136.3 | 67.5 | 141 ms | 14.8 ms | 2.8 | 0 | 4,090 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 324.7 | 81.1 | 1,629 ms | 12.3 ms | 5.1 | 0 | 9,740 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 610.3 | 75.9 | 228 ms | 13.2 ms | 9.5 | 0 | 18,304 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,064.8 | 66.2 | 349 ms | 15.1 ms | 20.3 | 0 | 31,936 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,582.8 | 49.6 | 662 ms | 20.2 ms | 47.5 | 0 | 47,488 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 87.4 | 87.3 | 107 ms | 11.4 ms | 1.0 | 0 | 2,621 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 98.5 | 49.4 | 173 ms | 20.2 ms | 34.0 | 0 | 2,956 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 292.3 | 69.7 | 216 ms | 14.3 ms | 11.2 | 0 | 8,768 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 574.0 | 71.5 | 353 ms | 14.0 ms | 10.4 | 0 | 17,216 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 936.0 | 58.3 | 659 ms | 17.2 ms | 21.9 | 0 | 28,080 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,314.4 | 41.3 | 1,260 ms | 24.2 ms | 48.0 | 0 | 39,424 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 85.2 | 85.3 | 177 ms | 11.7 ms | 1.0 | 0 | 2,557 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 81.4 | 40.5 | 288 ms | 24.7 ms | 38.3 | 0 | 2,442 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 185.0 | 45.5 | 415 ms | 22.0 ms | 32.7 | 0 | 5,548 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 486.5 | 60.9 | 788 ms | 16.4 ms | 12.7 | 0 | 14,592 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 725.6 | 46.0 | 1,616 ms | 21.8 ms | 24.0 | 0 | 21,760 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 81.7 | 81.7 | 299 ms | 12.2 ms | 1.0 | 0 | 2,451 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 104.0 | 51.3 | 603 ms | 19.5 ms | 18.0 | 0 | 3,120 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 240.6 | 58.5 | 960 ms | 17.1 ms | 10.1 | 0 | 7,216 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 410.2 | 52.4 | 1,949 ms | 19.1 ms | 12.0 | 0 | 12,305 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 75.4 | 75.4 | 528 ms | 13.3 ms | 1.0 | 0 | 2,263 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 101.7 | 51.2 | 1,046 ms | 19.5 ms | 10.0 | 0 | 3,050 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 211.1 | 52.2 | 1,935 ms | 19.1 ms | 6.4 | 0 | 6,332 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 65.9 | 65.9 | 1,013 ms | 15.2 ms | 1.0 | 0 | 1,977 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 92.7 | 46.5 | 1,988 ms | 21.5 ms | 6.0 | 0 | 2,780 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.