Speed test matrix
Qwen3.6-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.858781 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.4K567 ms TTFT
16K context14.5K1,114 ms TTFT
32K context13.6K2,359 ms TTFT
64K context11.9K5,387 ms TTFT
128K context9,53113,427 ms TTFT
251K context6,80536,853 ms TTFT
256K context6,76937,787 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 88.6Requests queued83 ms TTFT | 137.2139 ms TTFT | 321.81,637 ms TTFT | 584.6206 ms TTFT | 990.7350 ms TTFT | 1,414.4673 ms TTFT | Over capacity | Over capacity |
| 16K | 87.6106 ms TTFT | 91.0165 ms TTFT | 239.9224 ms TTFT | 478.3344 ms TTFT | 796.4685 ms TTFT | 1,122.61,271 ms TTFT | Over capacity | Over capacity |
| 32K | 85.8159 ms TTFT | 66.8262 ms TTFT | 138.9414 ms TTFT | 353.7847 ms TTFT | 586.91,550 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 82.1309 ms TTFT | 72.0543 ms TTFT | 139.61,042 ms TTFT | 273.31,945 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 76.0525 ms TTFT | 72.31,031 ms TTFT | 103.51,891 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 66.31,000 ms TTFT | 48.21,936 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.4K | — | 567 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.5K | — | 1,114 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.6K | — | 2,359 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.9K | — | 5,387 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,531.0 | — | 13,427 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,805.0 | — | 36,853 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,769.0 | — | 37,787 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 88.6 | 88.6 | 83 ms | 11.3 ms | 1.0 | 0 | 2,658 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 137.2 | 67.8 | 139 ms | 14.7 ms | 2.7 | 0 | 4,114 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 321.8 | 80.3 | 1,637 ms | 12.4 ms | 5.1 | 0 | 9,652 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 584.6 | 72.4 | 206 ms | 13.8 ms | 9.7 | 0 | 17,536 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 990.7 | 61.6 | 350 ms | 16.2 ms | 21.1 | 0 | 29,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,414.4 | 44.3 | 673 ms | 22.6 ms | 48.0 | 0 | 42,432 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 87.6 | 87.6 | 106 ms | 11.4 ms | 1.0 | 0 | 2,628 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 91.0 | 44.9 | 165 ms | 22.3 ms | 34.5 | 0 | 2,728 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 239.9 | 57.8 | 224 ms | 17.3 ms | 22.0 | 0 | 7,196 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 478.3 | 59.4 | 344 ms | 16.8 ms | 11.8 | 0 | 14,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 796.4 | 49.9 | 685 ms | 20.0 ms | 24.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,122.6 | 35.2 | 1,271 ms | 28.4 ms | 48.0 | 0 | 33,664 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 85.8 | 85.7 | 159 ms | 11.7 ms | 1.0 | 0 | 2,572 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 66.8 | 33.4 | 262 ms | 29.9 ms | 48.4 | 0 | 2,002 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 138.9 | 34.8 | 414 ms | 28.7 ms | 38.0 | 0 | 4,168 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 353.7 | 42.9 | 847 ms | 23.3 ms | 20.4 | 0 | 10,608 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 586.9 | 37.2 | 1,550 ms | 26.9 ms | 27.7 | 0 | 17,600 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 82.1 | 82.1 | 309 ms | 12.2 ms | 1.0 | 0 | 2,463 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 72.0 | 36.1 | 543 ms | 27.7 ms | 25.4 | 0 | 2,160 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 139.6 | 35.2 | 1,042 ms | 28.4 ms | 22.0 | 0 | 4,184 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 273.3 | 34.1 | 1,945 ms | 29.3 ms | 14.0 | 0 | 8,192 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 76.0 | 76.0 | 525 ms | 13.2 ms | 1.0 | 0 | 2,281 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 72.3 | 35.3 | 1,031 ms | 28.3 ms | 14.2 | 0 | 2,168 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 103.5 | 26.2 | 1,891 ms | 38.1 ms | 14.5 | 0 | 3,104 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 66.3 | 66.3 | 1,000 ms | 15.1 ms | 1.0 | 0 | 1,988 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 48.2 | 24.2 | 1,936 ms | 41.3 ms | 12.8 | 0 | 1,444 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.