Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-incoai-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.859135 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --cuda-graph-max-bs-prefill 2048 --attention-backend flashinfer --speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8
Prefill
prompt tok/s by context length8K context9,412871 ms TTFT
16K context9,0151,793 ms TTFT
32K context7,7934,125 ms TTFT
64K context6,11610,482 ms TTFT
128K context4,35929,370 ms TTFT
251K context2,80089,581 ms TTFT
256K context2,75792,782 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 130.899 ms TTFT | 238.9163 ms TTFT | 418.51,675 ms TTFT | 734.2209 ms TTFT | 1,101.1Requests queued2,571 ms TTFT | 996.2Requests queued31,160 ms TTFT | Not run | Over capacity |
| 16K | 134.7125 ms TTFT | 224.81,111 ms TTFT | 401.3215 ms TTFT | 705.5370 ms TTFT | 1,025.8Requests queued4,394 ms TTFT | 1,023.2Requests queued58,437 ms TTFT | Over capacity | Over capacity |
| 32K | 107.7181 ms TTFT | 156.21,676 ms TTFT | 376.0446 ms TTFT | 627.4863 ms TTFT | 896.4Requests queued9,030 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 114.9331 ms TTFT | 180.73,768 ms TTFT | 372.11,004 ms TTFT | 526.51,816 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 107.4556 ms TTFT | 128.11,035 ms TTFT | 274.91,923 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 96.71,062 ms TTFT | 124.41,996 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,412.0 | — | 871 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,015.0 | — | 1,793 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,793.0 | — | 4,125 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,116.0 | — | 10,482 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,359.0 | — | 29,370 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,800.0 | — | 89,581 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,757.0 | — | 92,782 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 130.8 | 153.9 | 99 ms | 6.7 ms | 1.0 | 1 | 3,923 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 238.9 | 130.0 | 163 ms | 7.7 ms | 2.0 | 1 | 7,161 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 418.5 | 108.1 | 1,675 ms | 9.3 ms | 4.9 | 0 | 12,551 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 734.2 | 93.5 | 209 ms | 10.7 ms | 8.0 | 0 | 22,024 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 1,101.1 | 76.8 | 2,571 ms | 13.0 ms | 14.7 | 14 | 33,002 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 996.2 | 77.4 | 31,160 ms | 12.9 ms | 13.3 | 13 | 29,886 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 134.7 | 144.4 | 125 ms | 7.0 ms | 1.0 | 1 | 4,040 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 224.8 | 111.2 | 1,111 ms | 9.0 ms | 4.0 | 0 | 6,745 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 401.3 | 104.0 | 215 ms | 9.6 ms | 4.0 | 0 | 12,031 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 705.5 | 90.6 | 370 ms | 11.0 ms | 8.1 | 0 | 21,149 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 1,025.8 | 70.6 | 4,394 ms | 14.2 ms | 14.1 | 15 | 30,769 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 1,023.2 | 71.8 | 58,437 ms | 13.9 ms | 14.3 | 13 | 30,688 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 107.7 | 115.1 | 181 ms | 8.7 ms | 1.0 | 0 | 3,228 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 156.2 | 79.7 | 1,676 ms | 12.6 ms | 9.1 | 0 | 4,684 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 376.0 | 94.9 | 446 ms | 10.5 ms | 5.4 | 0 | 11,279 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 627.4 | 81.5 | 863 ms | 12.3 ms | 8.7 | 0 | 18,814 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 896.4 | 64.8 | 9,030 ms | 15.4 ms | 13.7 | 13 | 26,863 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 114.9 | 118.8 | 331 ms | 8.4 ms | 1.0 | 0 | 3,448 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 180.7 | 88.3 | 3,768 ms | 11.3 ms | 4.9 | 0 | 5,416 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 372.1 | 96.6 | 1,004 ms | 10.4 ms | 4.2 | 0 | 11,163 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 526.5 | 67.8 | 1,816 ms | 14.8 ms | 9.1 | 0 | 15,796 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 107.4 | 111.2 | 556 ms | 9.0 ms | 1.0 | 0 | 3,221 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 128.1 | 65.4 | 1,035 ms | 15.3 ms | 8.4 | 0 | 3,843 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 274.9 | 71.8 | 1,923 ms | 13.9 ms | 5.1 | 0 | 8,241 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 96.7 | 99.8 | 1,062 ms | 10.0 ms | 1.0 | 0 | 2,899 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 124.4 | 64.2 | 1,996 ms | 15.6 ms | 4.9 | 0 | 3,731 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.