Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.809178 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context9,219889 ms TTFT
16K context8,7101,857 ms TTFT
32K context7,5654,250 ms TTFT
64K context6,02910,633 ms TTFT
128K context4,32929,570 ms TTFT
251K context2,79889,665 ms TTFT
256K context2,75892,766 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 62.3Requests queued85 ms TTFT | 100.1146 ms TTFT | 223.51,641 ms TTFT | 415.2217 ms TTFT | 707.6353 ms TTFT | 1,178.5Requests queued8,094 ms TTFT | 1,153.0Requests queued18,500 ms TTFT | Over capacity |
| 16K | 61.7112 ms TTFT | 90.13,149 ms TTFT | 183.6237 ms TTFT | 386.0355 ms TTFT | 614.0650 ms TTFT | 695.5Requests queued1,307 ms TTFT | Over capacity | Over capacity |
| 32K | 61.1170 ms TTFT | 60.5294 ms TTFT | 150.8440 ms TTFT | 289.6838 ms TTFT | 473.41,697 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 59.4297 ms TTFT | 60.4540 ms TTFT | 103.8940 ms TTFT | 182.41,925 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 56.5514 ms TTFT | 57.71,026 ms TTFT | 92.61,906 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 52.11,079 ms TTFT | 46.02,010 ms TTFT | 73.63,816 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,219.0 | — | 889 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,710.0 | — | 1,857 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,565.0 | — | 4,250 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,029.0 | — | 10,633 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,329.0 | — | 29,570 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,798.0 | — | 89,665 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,758.0 | — | 92,766 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 62.3 | 62.3 | 85 ms | 16.1 ms | 1.0 | 0 | 1,867 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 100.1 | 50.1 | 146 ms | 20.0 ms | 3.0 | 0 | 3,000 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 223.5 | 54.8 | 1,641 ms | 18.2 ms | 6.1 | 0 | 6,704 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 415.2 | 51.7 | 217 ms | 19.3 ms | 12.1 | 0 | 12,448 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 707.6 | 44.0 | 353 ms | 22.7 ms | 24.1 | 0 | 21,216 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,178.5 | 37.2 | 8,094 ms | 26.8 ms | 34.7 | 0 | 35,344 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,153.0 | 35.9 | 18,500 ms | 27.9 ms | 39.7 | 0 | 34,592 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 61.7 | 61.7 | 112 ms | 16.2 ms | 1.0 | 0 | 1,850 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 90.1 | 44.0 | 3,149 ms | 22.7 ms | 20.7 | 0 | 2,700 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 183.6 | 45.5 | 237 ms | 22.0 ms | 18.0 | 0 | 5,504 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 386.0 | 47.8 | 355 ms | 20.9 ms | 12.9 | 0 | 11,576 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 614.0 | 38.3 | 650 ms | 26.1 ms | 25.7 | 0 | 18,416 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 695.5 | 29.7 | 1,307 ms | 33.7 ms | 40.0 | 0 | 20,856 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 61.1 | 61.1 | 170 ms | 16.4 ms | 1.0 | 0 | 1,832 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 60.5 | 30.3 | 294 ms | 33.0 ms | 34.0 | 0 | 1,816 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 150.8 | 36.6 | 440 ms | 27.3 ms | 23.6 | 0 | 4,520 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 289.6 | 36.4 | 838 ms | 27.5 ms | 22.0 | 0 | 8,688 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 473.4 | 29.2 | 1,697 ms | 34.2 ms | 29.6 | 0 | 14,192 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 59.4 | 59.4 | 297 ms | 16.8 ms | 1.0 | 0 | 1,783 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 60.4 | 30.2 | 540 ms | 33.2 ms | 26.8 | 0 | 1,810 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 103.8 | 25.8 | 940 ms | 38.8 ms | 27.3 | 0 | 3,112 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 182.4 | 23.2 | 1,925 ms | 43.1 ms | 30.0 | 0 | 5,472 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 56.5 | 56.5 | 514 ms | 17.7 ms | 1.0 | 0 | 1,694 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 57.7 | 28.5 | 1,026 ms | 35.1 ms | 16.0 | 0 | 1,730 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 92.6 | 23.5 | 1,906 ms | 42.5 ms | 18.0 | 0 | 2,776 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 52.1 | 52.1 | 1,079 ms | 19.2 ms | 1.0 | 0 | 1,562 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 46.0 | 23.1 | 2,010 ms | 43.3 ms | 17.1 | 0 | 1,380 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 73.6 | 18.7 | 3,816 ms | 53.4 ms | 16.4 | 0 | 2,208 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.