Speed test matrix
Qwen3.8-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.817563 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --attention-backend flashinfer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context13K632 ms TTFT
16K context11.7K1,377 ms TTFT
32K context9,7413,300 ms TTFT
64K context7,2808,805 ms TTFT
128K context4,83126,500 ms TTFT
251K context2,92485,777 ms TTFT
256K context2,87888,888 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 135.796 ms TTFT | 239.8165 ms TTFT | 474.7877 ms TTFT | 744.1213 ms TTFT | 1,189.7359 ms TTFT | 1,471.5Requests queued13,551 ms TTFT | Not run | Over capacity |
| 16K | 141.3115 ms TTFT | 135.7197 ms TTFT | 392.9202 ms TTFT | 691.8352 ms TTFT | 918.3668 ms TTFT | 1,184.0Requests queued43,765 ms TTFT | Not run | Over capacity |
| 32K | 127.5169 ms TTFT | 87.5309 ms TTFT | 260.8466 ms TTFT | 497.1841 ms TTFT | 526.71,615 ms TTFT | 381.2Requests queued15,467 ms TTFT | Over capacity | Over capacity |
| 64K | 116.3294 ms TTFT | 76.03,720 ms TTFT | 126.91,003 ms TTFT | 255.01,929 ms TTFT | 318.43,736 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 98.7531 ms TTFT | 38.93,364 ms TTFT | 76.98,603 ms TTFT | 155.33,862 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 83.41,015 ms TTFT | 32.32,094 ms TTFT | 47.73,907 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13K | — | 632 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.7K | — | 1,377 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,741.0 | — | 3,300 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,280.0 | — | 8,805 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,831.0 | — | 26,500 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,924.0 | — | 85,777 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,878.0 | — | 88,888 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 135.7 | 141.5 | 96 ms | 7.1 ms | 1.0 | 1 | 4,070 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 239.8 | 115.8 | 165 ms | 8.6 ms | 2.3 | 0 | 7,193 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 474.7 | 120.7 | 877 ms | 8.3 ms | 4.0 | 2 | 14,235 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 744.1 | 92.9 | 213 ms | 10.8 ms | 10.0 | 0 | 22,310 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,189.7 | 73.4 | 359 ms | 13.6 ms | 18.0 | 0 | 35,688 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,471.5 | 53.3 | 13,551 ms | 18.8 ms | 27.7 | 12 | 44,090 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 141.3 | 139.9 | 115 ms | 7.2 ms | 1.0 | 1 | 4,237 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 135.7 | 68.0 | 197 ms | 14.7 ms | 18.2 | 0 | 4,070 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 392.9 | 89.5 | 202 ms | 11.2 ms | 8.2 | 0 | 11,782 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 691.8 | 86.5 | 352 ms | 11.6 ms | 8.6 | 0 | 20,755 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 918.3 | 56.8 | 668 ms | 17.6 ms | 20.2 | 0 | 27,522 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 1,184.0 | 42.8 | 43,765 ms | 23.4 ms | 27.9 | 1 | 35,523 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 127.5 | 142.7 | 169 ms | 7.1 ms | 1.0 | 1 | 3,824 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 87.5 | 44.3 | 309 ms | 22.6 ms | 23.1 | 0 | 2,622 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 260.8 | 62.8 | 466 ms | 15.9 ms | 12.5 | 0 | 7,819 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 497.1 | 63.0 | 841 ms | 15.9 ms | 11.4 | 0 | 14,902 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 526.7 | 34.2 | 1,615 ms | 29.2 ms | 24.0 | 0 | 15,771 | 29.9 s |
| Sustained decode | 32K | 32 | Requests queued | 381.2 | 31.3 | 15,467 ms | 32.0 ms | 27.9 | 0 | 11,435 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 116.3 | 117.1 | 294 ms | 8.5 ms | 1.0 | 0 | 3,489 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 76.0 | 37.4 | 3,720 ms | 26.8 ms | 20.7 | 0 | 2,278 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 126.9 | 32.5 | 1,003 ms | 30.8 ms | 22.0 | 0 | 3,801 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 255.0 | 32.5 | 1,929 ms | 30.7 ms | 18.0 | 0 | 7,645 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 318.4 | 21.1 | 3,736 ms | 47.5 ms | 28.0 | 0 | 9,549 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 98.7 | 99.7 | 531 ms | 10.0 ms | 1.0 | 0 | 2,959 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 38.9 | 20.2 | 3,364 ms | 49.6 ms | 26.5 | 0 | 1,165 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 76.9 | 20.3 | 8,603 ms | 49.2 ms | 23.7 | 0 | 2,301 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 155.3 | 15.9 | 3,862 ms | 62.8 ms | 18.9 | 0 | 4,644 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 83.4 | 83.7 | 1,015 ms | 11.9 ms | 1.0 | 0 | 2,500 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 32.3 | 17.5 | 2,094 ms | 57.2 ms | 16.0 | 0 | 968 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 47.7 | 13.3 | 3,907 ms | 75.4 ms | 19.5 | 0 | 1,423 | 29.8 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.