Speed test matrix
Qwen3.6-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.858781 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.4K566 ms TTFT
16K context14.5K1,114 ms TTFT
32K context13.7K2,345 ms TTFT
64K context11.9K5,368 ms TTFT
128K context9,52013,443 ms TTFT
251K context6,81036,826 ms TTFT
256K context6,75437,870 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 88.6Requests queued84 ms TTFT | 137.2138 ms TTFT | 320.91,638 ms TTFT | 582.3201 ms TTFT | 990.1354 ms TTFT | 1,412.8661 ms TTFT | Over capacity | Over capacity |
| 16K | 87.5106 ms TTFT | 90.9166 ms TTFT | 239.6219 ms TTFT | 477.2342 ms TTFT | 796.7702 ms TTFT | 1,120.71,251 ms TTFT | Over capacity | Over capacity |
| 32K | 85.6158 ms TTFT | 66.9279 ms TTFT | 138.9450 ms TTFT | 354.0801 ms TTFT | 587.31,610 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 82.1284 ms TTFT | 72.0567 ms TTFT | 139.4957 ms TTFT | 273.51,918 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 76.1579 ms TTFT | 72.5967 ms TTFT | 103.61,912 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 66.4994 ms TTFT | 48.31,941 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.4K | — | 566 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.5K | — | 1,114 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.7K | — | 2,345 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.9K | — | 5,368 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,520.0 | — | 13,443 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,810.0 | — | 36,826 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,754.0 | — | 37,870 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 88.6 | 88.7 | 84 ms | 11.3 ms | 1.0 | 0 | 2,658 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 137.2 | 67.8 | 138 ms | 14.7 ms | 2.7 | 0 | 4,116 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 320.9 | 80.1 | 1,638 ms | 12.5 ms | 5.1 | 0 | 9,624 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 582.3 | 72.1 | 201 ms | 13.9 ms | 9.7 | 0 | 17,464 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 990.1 | 61.5 | 354 ms | 16.3 ms | 21.3 | 0 | 29,696 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,412.8 | 44.2 | 661 ms | 22.6 ms | 48.0 | 0 | 42,368 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 87.5 | 87.5 | 106 ms | 11.4 ms | 1.0 | 0 | 2,624 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 90.9 | 44.8 | 166 ms | 22.3 ms | 34.5 | 0 | 2,726 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 239.6 | 57.8 | 219 ms | 17.3 ms | 22.0 | 0 | 7,188 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 477.2 | 59.3 | 342 ms | 16.8 ms | 11.8 | 0 | 14,312 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 796.7 | 50.0 | 702 ms | 20.0 ms | 24.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,120.7 | 35.3 | 1,251 ms | 28.3 ms | 48.0 | 0 | 33,600 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 85.6 | 85.7 | 158 ms | 11.7 ms | 1.0 | 0 | 2,568 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 66.9 | 33.5 | 279 ms | 29.9 ms | 47.9 | 0 | 2,006 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 138.9 | 34.8 | 450 ms | 28.7 ms | 38.0 | 0 | 4,164 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 354.0 | 42.9 | 801 ms | 23.3 ms | 20.9 | 0 | 10,616 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 587.3 | 37.2 | 1,610 ms | 26.9 ms | 27.9 | 0 | 17,616 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 82.1 | 82.1 | 284 ms | 12.2 ms | 1.0 | 0 | 2,462 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 72.0 | 36.1 | 567 ms | 27.7 ms | 25.2 | 0 | 2,159 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 139.4 | 35.1 | 957 ms | 28.5 ms | 22.0 | 0 | 4,180 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 273.5 | 34.3 | 1,918 ms | 29.2 ms | 14.0 | 0 | 8,200 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 76.1 | 76.1 | 579 ms | 13.1 ms | 1.0 | 0 | 2,283 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 72.5 | 35.3 | 967 ms | 28.4 ms | 14.2 | 0 | 2,174 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 103.6 | 26.3 | 1,912 ms | 38.1 ms | 14.4 | 0 | 3,104 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 66.4 | 66.4 | 994 ms | 15.1 ms | 1.0 | 0 | 1,991 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 48.3 | 24.2 | 1,941 ms | 41.3 ms | 12.8 | 0 | 1,448 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.