Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.822808 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --attention-backend flashinfer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context9,033907 ms TTFT
16K context8,4961,903 ms TTFT
32K context7,3374,382 ms TTFT
64K context5,83610,984 ms TTFT
128K context4,15230,831 ms TTFT
251K context2,65994,320 ms TTFT
256K context2,62797,392 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 109.690 ms TTFT | 158.9150 ms TTFT | 370.01,635 ms TTFT | 633.0208 ms TTFT | 1,027.7366 ms TTFT | 1,147.3Requests queued15,387 ms TTFT | Not run | Over capacity |
| 16K | 102.7114 ms TTFT | 163.52,234 ms TTFT | 339.1237 ms TTFT | 592.5362 ms TTFT | 872.8685 ms TTFT | 986.6Requests queued19,067 ms TTFT | Over capacity | Over capacity |
| 32K | 98.3186 ms TTFT | 101.77,525 ms TTFT | 228.7420 ms TTFT | 402.1855 ms TTFT | 687.3Requests queued1,681 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 94.3326 ms TTFT | 107.8739 ms TTFT | 213.31,030 ms TTFT | 305.01,932 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 87.8737 ms TTFT | 63.51,039 ms TTFT | 107.51,926 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 70.31,074 ms TTFT | 56.62,065 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,033.0 | — | 907 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,496.0 | — | 1,903 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,337.0 | — | 4,382 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,836.0 | — | 10,984 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,152.0 | — | 30,831 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,659.0 | — | 94,320 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,627.0 | — | 97,392 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 109.6 | 109.2 | 90 ms | 9.2 ms | 1.0 | 0 | 3,286 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 158.9 | 78.1 | 150 ms | 12.8 ms | 2.6 | 0 | 4,767 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 370.0 | 92.0 | 1,635 ms | 10.9 ms | 4.7 | 0 | 11,092 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 633.0 | 79.2 | 208 ms | 12.6 ms | 8.8 | 0 | 18,981 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,027.7 | 62.3 | 366 ms | 16.1 ms | 16.1 | 0 | 30,829 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,147.3 | 57.7 | 15,387 ms | 17.3 ms | 19.7 | 16 | 34,399 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 102.7 | 103.7 | 114 ms | 9.6 ms | 1.0 | 0 | 3,081 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 163.5 | 80.2 | 2,234 ms | 12.5 ms | 6.3 | 0 | 4,905 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 339.1 | 84.7 | 237 ms | 11.8 ms | 4.7 | 0 | 10,172 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 592.5 | 73.3 | 362 ms | 13.7 ms | 9.5 | 0 | 17,762 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 872.8 | 51.6 | 685 ms | 19.4 ms | 16.3 | 0 | 26,181 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 986.6 | 49.5 | 19,067 ms | 20.2 ms | 19.8 | 4 | 29,575 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 98.3 | 99.4 | 186 ms | 10.1 ms | 1.0 | 0 | 2,949 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 101.7 | 51.2 | 7,525 ms | 19.5 ms | 18.0 | 0 | 3,052 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 228.7 | 57.3 | 420 ms | 17.5 ms | 12.9 | 0 | 6,858 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 402.1 | 51.0 | 855 ms | 19.6 ms | 12.0 | 0 | 12,049 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 687.3 | 40.0 | 1,681 ms | 25.0 ms | 16.6 | 0 | 20,618 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 94.3 | 96.2 | 326 ms | 10.4 ms | 1.0 | 0 | 2,826 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 107.8 | 52.8 | 739 ms | 19.0 ms | 9.2 | 0 | 3,235 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 213.3 | 53.6 | 1,030 ms | 18.7 ms | 7.5 | 0 | 6,397 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 305.0 | 39.3 | 1,932 ms | 25.4 ms | 12.1 | 0 | 9,150 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 87.8 | 88.5 | 737 ms | 11.3 ms | 1.0 | 0 | 2,633 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 63.5 | 32.8 | 1,039 ms | 30.6 ms | 12.4 | 0 | 1,902 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 107.5 | 28.4 | 1,926 ms | 35.2 ms | 14.0 | 0 | 3,224 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 70.3 | 70.8 | 1,074 ms | 14.1 ms | 1.0 | 0 | 2,109 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 56.6 | 28.5 | 2,065 ms | 35.1 ms | 10.3 | 0 | 1,696 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.