Speed test matrix
Qwen3.6-27B-NVFP4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Qwen3.6-27B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.878057 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-cuda-graph
Prefill
prompt tok/s by context length8K context6,0111,356 ms TTFT
16K context6,3682,532 ms TTFT
32K context6,3575,050 ms TTFT
64K context6,19410,342 ms TTFT
128K context5,69822,460 ms TTFT
251K context4,95150,660 ms TTFT
256K context4,92751,916 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 19.4Requests queued198 ms TTFT | 38.9261 ms TTFT | 78.41,748 ms TTFT | 156.1305 ms TTFT | 312.9421 ms TTFT | 633.5713 ms TTFT | 196.9Requests queued26,853 ms TTFT | 913.1Requests queued38,101 ms TTFT |
| 16K | 19.3Requests queued223 ms TTFT | 37.8280 ms TTFT | 79.41,022 ms TTFT | 158.2449 ms TTFT | 312.7796 ms TTFT | 653.71,538 ms TTFT | 200.9Requests queued26,308 ms TTFT | Over capacity |
| 32K | 19.4Requests queued278 ms TTFT | 39.3386 ms TTFT | 78.6545 ms TTFT | 157.9932 ms TTFT | 311.71,822 ms TTFT | 625.53,581 ms TTFT | Over capacity | Over capacity |
| 64K | 19.3Requests queued411 ms TTFT | 39.2655 ms TTFT | 74.41,114 ms TTFT | 147.12,050 ms TTFT | 294.64,341 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 19.2Requests queued664 ms TTFT | 34.61,145 ms TTFT | 68.32,024 ms TTFT | 134.63,992 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 19.0Requests queued1,208 ms TTFT | 33.12,194 ms TTFT | 52.24,132 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,011.0 | — | 1,356 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,368.0 | — | 2,532 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,357.0 | — | 5,050 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,194.0 | — | 10,342 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,698.0 | — | 22,460 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,951.0 | — | 50,660 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,927.0 | — | 51,916 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 19.4 | 19.3 | 198 ms | 51.7 ms | 0.9 | 0 | 580 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 38.9 | 19.4 | 261 ms | 51.5 ms | 3.0 | 0 | 1,166 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 78.4 | 19.6 | 1,748 ms | 51.1 ms | 7.0 | 0 | 2,348 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 156.1 | 19.5 | 305 ms | 51.3 ms | 15.0 | 0 | 4,680 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 312.9 | 19.5 | 421 ms | 51.3 ms | 31.0 | 0 | 9,376 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 633.5 | 19.7 | 713 ms | 50.8 ms | 62.9 | 0 | 18,976 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 196.9 | 19.0 | 26,853 ms | 52.8 ms | 67.0 | 0 | 5,899 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 913.1 | 19.0 | 38,101 ms | 52.6 ms | 67.0 | 0 | 27,360 | 30.0 s |
| Sustained decode | 16K | 1 | Requests queued | 19.3 | 19.3 | 223 ms | 51.9 ms | 1.0 | 0 | 578 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 37.8 | 18.8 | 280 ms | 53.1 ms | 65.0 | 0 | 1,132 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 79.4 | 19.8 | 1,022 ms | 50.5 ms | 61.0 | 0 | 2,380 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 158.2 | 19.7 | 449 ms | 50.7 ms | 53.0 | 0 | 4,744 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 312.7 | 19.5 | 796 ms | 51.4 ms | 37.0 | 0 | 9,376 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 653.7 | 20.3 | 1,538 ms | 49.2 ms | 63.4 | 0 | 19,584 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 200.9 | 18.9 | 26,308 ms | 52.8 ms | 66.9 | 0 | 6,025 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Requests queued | 19.4 | 19.4 | 278 ms | 51.7 ms | 1.0 | 0 | 580 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 39.3 | 19.6 | 386 ms | 51.0 ms | 53.0 | 0 | 1,180 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 78.6 | 19.6 | 545 ms | 51.0 ms | 25.0 | 0 | 2,356 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 157.9 | 19.7 | 932 ms | 50.8 ms | 27.5 | 0 | 4,736 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 311.7 | 19.5 | 1,822 ms | 51.3 ms | 40.9 | 0 | 9,344 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 625.5 | 19.3 | 3,581 ms | 51.7 ms | 62.0 | 0 | 18,752 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Requests queued | 19.3 | 19.3 | 411 ms | 51.8 ms | 0.9 | 0 | 579 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 39.2 | 19.6 | 655 ms | 51.0 ms | 62.0 | 0 | 1,174 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 74.4 | 18.7 | 1,114 ms | 53.5 ms | 62.0 | 0 | 2,232 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 147.1 | 18.6 | 2,050 ms | 53.8 ms | 62.0 | 0 | 4,408 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 294.6 | 18.3 | 4,341 ms | 54.5 ms | 62.0 | 0 | 8,832 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Requests queued | 19.2 | 19.3 | 664 ms | 51.8 ms | 1.0 | 0 | 576 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 34.6 | 17.3 | 1,145 ms | 57.8 ms | 34.6 | 0 | 1,036 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 68.3 | 17.3 | 2,024 ms | 57.8 ms | 36.0 | 0 | 2,048 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 134.6 | 17.5 | 3,992 ms | 57.3 ms | 42.0 | 0 | 4,032 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 19.0 | 19.4 | 1,208 ms | 51.5 ms | 0.9 | 0 | 569 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 33.1 | 16.7 | 2,194 ms | 59.8 ms | 40.0 | 0 | 990 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 52.2 | 13.7 | 4,132 ms | 73.1 ms | 36.0 | 0 | 1,564 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.