Speed test matrix
Qwen3.8-27B-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.810972 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context13.4K614 ms TTFT
16K context12.1K1,340 ms TTFT
32K context10K3,205 ms TTFT
64K context7,5598,481 ms TTFT
128K context5,06825,259 ms TTFT
251K context3,08581,319 ms TTFT
256K context3,03484,326 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.991 ms TTFT | 140.4144 ms TTFT | 314.31,654 ms TTFT | 579.6207 ms TTFT | 972.0357 ms TTFT | 1,397.6662 ms TTFT | 2,172.9Requests queued62,100 ms TTFT | Over capacity |
| 16K | 85.4111 ms TTFT | 84.4172 ms TTFT | 217.2194 ms TTFT | 504.0346 ms TTFT | 788.4650 ms TTFT | 1,099.51,281 ms TTFT | 727.5Requests queued32,304 ms TTFT | Over capacity |
| 32K | 84.0171 ms TTFT | 65.8305 ms TTFT | 133.8432 ms TTFT | 369.0867 ms TTFT | 578.61,559 ms TTFT | 757.03,136 ms TTFT | Over capacity | Over capacity |
| 64K | 81.0309 ms TTFT | 46.9563 ms TTFT | 95.81,011 ms TTFT | 251.61,924 ms TTFT | 399.93,778 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 75.3536 ms TTFT | 48.81,060 ms TTFT | 64.41,929 ms TTFT | 123.53,659 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 67.91,027 ms TTFT | 30.71,996 ms TTFT | 55.63,880 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.4K | — | 614 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.1K | — | 1,340 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10K | — | 3,205 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,559.0 | — | 8,481 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,068.0 | — | 25,259 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,085.0 | — | 81,319 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,034.0 | — | 84,326 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.9 | 86.0 | 91 ms | 11.6 ms | 1.0 | 0 | 2,577 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 140.4 | 69.4 | 144 ms | 14.4 ms | 2.7 | 0 | 4,212 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 314.3 | 78.4 | 1,654 ms | 12.8 ms | 5.0 | 0 | 9,428 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 579.6 | 72.1 | 207 ms | 13.9 ms | 9.9 | 0 | 17,384 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 972.0 | 60.5 | 357 ms | 16.5 ms | 21.6 | 0 | 29,152 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,397.6 | 43.8 | 662 ms | 22.9 ms | 48.0 | 0 | 41,920 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 2,172.9 | 41.0 | 62,100 ms | 24.4 ms | 53.9 | 0 | 65,192 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 85.4 | 85.5 | 111 ms | 11.7 ms | 1.0 | 0 | 2,563 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.4 | 42.3 | 172 ms | 23.7 ms | 43.0 | 0 | 2,532 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 217.2 | 52.5 | 194 ms | 19.0 ms | 26.7 | 0 | 6,512 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 504.0 | 61.9 | 346 ms | 16.2 ms | 13.0 | 0 | 15,121 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 788.4 | 49.4 | 650 ms | 20.3 ms | 24.0 | 0 | 23,648 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,099.5 | 34.5 | 1,281 ms | 28.9 ms | 48.0 | 0 | 32,960 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 727.5 | 31.9 | 32,304 ms | 31.3 ms | 54.9 | 0 | 21,828 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 84.0 | 83.9 | 171 ms | 11.9 ms | 1.0 | 0 | 2,518 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 65.8 | 32.9 | 305 ms | 30.4 ms | 48.9 | 0 | 1,972 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 133.8 | 33.5 | 432 ms | 29.8 ms | 38.0 | 0 | 4,012 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 369.0 | 44.6 | 867 ms | 22.4 ms | 23.3 | 0 | 11,064 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 578.6 | 36.7 | 1,559 ms | 27.3 ms | 27.9 | 0 | 17,359 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 757.0 | 23.9 | 3,136 ms | 41.9 ms | 48.0 | 0 | 22,688 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 81.0 | 81.0 | 309 ms | 12.4 ms | 1.0 | 0 | 2,428 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 46.9 | 23.5 | 563 ms | 42.6 ms | 50.0 | 0 | 1,406 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 95.8 | 24.1 | 1,011 ms | 41.4 ms | 38.0 | 0 | 2,872 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 251.6 | 31.0 | 1,924 ms | 32.3 ms | 30.0 | 0 | 7,544 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 399.9 | 26.1 | 3,778 ms | 38.3 ms | 30.2 | 0 | 11,984 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 75.3 | 75.3 | 536 ms | 13.3 ms | 1.0 | 0 | 2,258 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 48.8 | 24.2 | 1,060 ms | 41.4 ms | 30.3 | 0 | 1,464 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 64.4 | 16.3 | 1,929 ms | 61.4 ms | 30.5 | 0 | 1,932 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 123.5 | 16.0 | 3,659 ms | 62.5 ms | 35.7 | 0 | 3,704 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 67.9 | 67.8 | 1,027 ms | 14.7 ms | 1.0 | 0 | 2,035 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 30.7 | 15.5 | 1,996 ms | 64.5 ms | 32.0 | 0 | 920 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 55.6 | 14.3 | 3,880 ms | 70.0 ms | 32.2 | 0 | 1,668 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.