Speed test matrix
Qwen3.8-27B-NVFP4 on H200 NVL 141GB
sglang · 0.0.0.dev16762+g0c088029c · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev16762+g0c088029c
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/alphabetc1/sglang
- Build
- cuda-13.2
- Commit
0c088029ccba
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-radixark-sglang/bin/sglang" serve --model-path RadixArk/Qwen3.8-27B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.845041 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context6,3031,300 ms TTFT
16K context6,2802,575 ms TTFT
32K context6,2035,183 ms TTFT
64K context5,89210,880 ms TTFT
128K context5,44523,512 ms TTFT
251K context4,68153,585 ms TTFT
256K context4,66754,820 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 163.185 ms TTFT | 259.0117 ms TTFT | 487.9120 ms TTFT | 750.0206 ms TTFT | 1,150.7353 ms TTFT | 1,559.6707 ms TTFT | Not run | Not run |
| 16K | 156.9110 ms TTFT | 148.95,790 ms TTFT | 502.1171 ms TTFT | 639.9392 ms TTFT | 938.1644 ms TTFT | 1,475.31,232 ms TTFT | Not run | Over capacity |
| 32K | 147.9173 ms TTFT | 108.23,322 ms TTFT | 464.0461 ms TTFT | 644.0811 ms TTFT | 783.71,580 ms TTFT | 1,206.3Requests queued3,320 ms TTFT | Over capacity | Over capacity |
| 64K | 150.3300 ms TTFT | 91.68,007 ms TTFT | 305.0944 ms TTFT | 555.31,838 ms TTFT | 554.93,566 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 131.8532 ms TTFT | 68.81,035 ms TTFT | 135.41,864 ms TTFT | 326.33,658 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 147.0998 ms TTFT | 100.41,981 ms TTFT | 176.43,781 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,303.0 | — | 1,300 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,280.0 | — | 2,575 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,203.0 | — | 5,183 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,892.0 | — | 10,880 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,445.0 | — | 23,512 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,681.0 | — | 53,585 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,667.0 | — | 54,820 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 163.1 | 162.2 | 85 ms | 6.2 ms | 1.0 | 1 | 4,891 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 259.0 | 150.5 | 117 ms | 6.8 ms | 2.6 | 2 | 7,770 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 487.9 | 126.7 | 120 ms | 7.9 ms | 5.6 | 3 | 14,635 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 750.0 | 94.8 | 206 ms | 10.6 ms | 11.0 | 0 | 22,498 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,150.7 | 69.0 | 353 ms | 14.5 ms | 17.9 | 0 | 34,502 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,559.6 | 47.2 | 707 ms | 21.2 ms | 32.0 | 0 | 46,783 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 156.9 | 163.9 | 110 ms | 6.1 ms | 1.0 | 1 | 4,706 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 148.9 | 68.4 | 5,790 ms | 14.6 ms | 15.3 | 0 | 4,467 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 502.1 | 134.3 | 171 ms | 7.5 ms | 4.3 | 4 | 15,060 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 639.9 | 81.5 | 392 ms | 12.3 ms | 12.0 | 0 | 19,192 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 938.1 | 57.7 | 644 ms | 17.3 ms | 22.0 | 0 | 28,121 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,475.3 | 43.5 | 1,232 ms | 23.0 ms | 32.0 | 0 | 44,206 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 147.9 | 150.9 | 173 ms | 6.6 ms | 1.0 | 1 | 4,436 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 108.2 | 52.6 | 3,322 ms | 19.0 ms | 18.5 | 0 | 3,246 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 464.0 | 113.3 | 461 ms | 8.8 ms | 5.2 | 1 | 13,913 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 644.0 | 81.8 | 811 ms | 12.2 ms | 9.0 | 0 | 19,316 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 783.7 | 49.6 | 1,580 ms | 20.2 ms | 21.6 | 0 | 23,512 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 1,206.3 | 39.5 | 3,320 ms | 25.3 ms | 32.0 | 0 | 36,160 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 150.3 | 153.6 | 300 ms | 6.5 ms | 1.0 | 1 | 4,507 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 91.6 | 44.3 | 8,007 ms | 22.7 ms | 18.8 | 0 | 2,743 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 305.0 | 73.6 | 944 ms | 13.6 ms | 10.9 | 0 | 9,144 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 555.3 | 69.7 | 1,838 ms | 14.4 ms | 10.2 | 0 | 16,654 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 554.9 | 37.1 | 3,566 ms | 26.9 ms | 24.0 | 0 | 16,628 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.8 | 141.5 | 532 ms | 7.1 ms | 1.0 | 1 | 3,953 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 68.8 | 35.8 | 1,035 ms | 28.0 ms | 18.0 | 0 | 2,060 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 135.4 | 35.6 | 1,864 ms | 28.1 ms | 21.8 | 0 | 4,059 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 326.3 | 43.2 | 3,658 ms | 23.2 ms | 14.0 | 0 | 9,780 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 147.0 | 150.5 | 998 ms | 6.6 ms | 1.0 | 1 | 4,409 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 100.4 | 51.2 | 1,981 ms | 19.6 ms | 13.1 | 0 | 3,006 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 176.4 | 44.4 | 3,781 ms | 22.7 ms | 10.5 | 0 | 5,290 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.