Speed test matrix
Qwen3.8-27B-NVFP4-BF16-LMHead on H200 NVL 141GB
sglang · 0.5.6.post3.dev9023+g73a255206 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9023+g73a255206
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
73a255206f91
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-radixark-qwen-sglang/bin/sglang" serve --model-path RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead --served-model-name qwen --tp 1 --reasoning-parser auto --tool-call-parser auto --context-length 262144 --mem-fraction-static 0.866754 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --chunked-prefill-size 4096 --max-running-requests 36 --allow-auto-truncate --fp4-gemm-backend marlin --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context6,4361,273 ms TTFT
16K context6,4802,495 ms TTFT
32K context6,3415,070 ms TTFT
64K context6,05210,591 ms TTFT
128K context5,59422,884 ms TTFT
251K context4,86451,568 ms TTFT
256K context4,86452,599 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 94.178 ms TTFT | 167.3112 ms TTFT | 351.4138 ms TTFT | 634.8196 ms TTFT | 1,027.0350 ms TTFT | 1,752.5658 ms TTFT | Not run | Not run |
| 16K | 93.7102 ms TTFT | 138.2145 ms TTFT | 336.3215 ms TTFT | 583.3353 ms TTFT | 880.7700 ms TTFT | 1,551.31,262 ms TTFT | Not run | Over capacity |
| 32K | 91.7152 ms TTFT | 122.4291 ms TTFT | 310.2435 ms TTFT | 479.4819 ms TTFT | 674.61,701 ms TTFT | 559.6Requests queued9,190 ms TTFT | Over capacity | Over capacity |
| 64K | 88.8291 ms TTFT | 57.3570 ms TTFT | 168.37,097 ms TTFT | 250.51,888 ms TTFT | 483.13,830 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 84.1515 ms TTFT | 60.11,000 ms TTFT | 88.31,873 ms TTFT | 157.43,593 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 76.21,022 ms TTFT | 41.61,916 ms TTFT | 79.73,715 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,436.0 | — | 1,273 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,480.0 | — | 2,495 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,341.0 | — | 5,070 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,052.0 | — | 10,591 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,594.0 | — | 22,884 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,864.0 | — | 51,568 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,864.0 | — | 52,599 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 94.1 | 94.1 | 78 ms | 10.6 ms | 1.0 | 0 | 2,821 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 167.3 | 82.3 | 112 ms | 12.1 ms | 2.5 | 0 | 5,018 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 351.4 | 87.4 | 138 ms | 11.4 ms | 4.5 | 0 | 10,544 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 634.8 | 78.1 | 196 ms | 12.8 ms | 9.1 | 0 | 19,040 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,027.0 | 63.5 | 350 ms | 15.7 ms | 20.3 | 0 | 30,800 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,752.5 | 50.2 | 658 ms | 19.9 ms | 32.0 | 0 | 52,576 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 93.7 | 93.7 | 102 ms | 10.7 ms | 1.0 | 0 | 2,811 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 138.2 | 66.0 | 145 ms | 15.1 ms | 15.2 | 0 | 4,146 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 336.3 | 83.8 | 215 ms | 11.9 ms | 5.0 | 0 | 10,088 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 583.3 | 71.8 | 353 ms | 13.9 ms | 9.6 | 0 | 17,496 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 880.7 | 54.9 | 700 ms | 18.2 ms | 22.1 | 0 | 26,416 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,551.3 | 42.1 | 1,262 ms | 23.8 ms | 32.0 | 0 | 46,528 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 91.7 | 91.7 | 152 ms | 10.9 ms | 1.0 | 0 | 2,750 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 122.4 | 58.3 | 291 ms | 17.2 ms | 16.9 | 0 | 3,672 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 310.2 | 75.2 | 435 ms | 13.3 ms | 5.5 | 0 | 9,304 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 479.4 | 59.6 | 819 ms | 16.8 ms | 10.7 | 0 | 14,376 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 674.6 | 42.9 | 1,701 ms | 23.3 ms | 24.0 | 0 | 20,240 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 559.6 | 28.2 | 9,190 ms | 35.5 ms | 36.0 | 0 | 16,787 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 88.8 | 88.9 | 291 ms | 11.2 ms | 1.0 | 0 | 2,663 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 57.3 | 28.7 | 570 ms | 34.9 ms | 34.0 | 0 | 1,716 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 168.3 | 39.9 | 7,097 ms | 25.1 ms | 18.9 | 0 | 5,048 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 250.5 | 31.9 | 1,888 ms | 31.4 ms | 25.1 | 0 | 7,512 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 483.1 | 31.6 | 3,830 ms | 31.7 ms | 27.7 | 0 | 14,480 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 84.1 | 84.1 | 515 ms | 11.9 ms | 1.0 | 0 | 2,523 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 60.1 | 30.1 | 1,000 ms | 33.2 ms | 26.1 | 0 | 1,800 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 88.3 | 22.2 | 1,873 ms | 45.0 ms | 28.2 | 0 | 2,648 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 157.4 | 20.5 | 3,593 ms | 48.8 ms | 30.0 | 0 | 4,720 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 76.2 | 76.2 | 1,022 ms | 13.1 ms | 1.0 | 0 | 2,285 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 41.6 | 20.9 | 1,916 ms | 47.9 ms | 17.5 | 0 | 1,248 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 79.7 | 20.7 | 3,715 ms | 48.4 ms | 19.3 | 0 | 2,392 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.