Speed test matrix
Qwen3.8-27B-FP8 on H200 NVL 141GB
sglang · 0.0.0.dev16762+g0c088029c · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev16762+g0c088029c
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/alphabetc1/sglang
- Build
- cuda-13.2
- Commit
0c088029ccba
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-radixark-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.841325 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --attention-backend fa3 --speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark --speculative-draft-model-quantization unquant --speculative-dspark-block-size 7
Prefill
prompt tok/s by context length8K context13.9K589 ms TTFT
16K context13.8K1,175 ms TTFT
32K context13K2,482 ms TTFT
64K context11.4K5,637 ms TTFT
128K context9,15813,979 ms TTFT
251K context6,63237,822 ms TTFT
256K context6,57938,884 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 155.385 ms TTFT | 299.4122 ms TTFT | 495.6133 ms TTFT | 702.0203 ms TTFT | 1,268.95,145 ms TTFT | 1,297.6Requests queued12,379 ms TTFT | Over capacity | Over capacity |
| 16K | 135.1107 ms TTFT | 195.0166 ms TTFT | 531.2161 ms TTFT | 665.1348 ms TTFT | 1,205.61,784 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 151.5174 ms TTFT | 253.7230 ms TTFT | 482.8412 ms TTFT | 556.2829 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 155.8289 ms TTFT | 195.5568 ms TTFT | 444.8951 ms TTFT | 496.31,840 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 133.1529 ms TTFT | 123.51,047 ms TTFT | 295.91,862 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 107.91,050 ms TTFT | 175.11,925 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.9K | — | 589 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.8K | — | 1,175 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13K | — | 2,482 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.4K | — | 5,637 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,158.0 | — | 13,979 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,632.0 | — | 37,822 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,579.0 | — | 38,884 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 155.3 | 153.0 | 85 ms | 6.6 ms | 1.0 | 1 | 4,657 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 299.4 | 146.6 | 122 ms | 6.8 ms | 2.0 | 2 | 8,979 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 495.6 | 149.7 | 133 ms | 6.9 ms | 5.1 | 4 | 14,863 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 702.0 | 89.3 | 203 ms | 11.2 ms | 12.0 | 0 | 21,044 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,268.9 | 78.8 | 5,145 ms | 12.7 ms | 16.0 | 0 | 38,044 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 1,297.6 | 68.7 | 12,379 ms | 14.6 ms | 18.8 | 17 | 38,904 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 135.1 | 172.3 | 107 ms | 6.1 ms | 1.0 | 1 | 4,052 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 195.0 | 97.7 | 166 ms | 10.3 ms | 7.3 | 0 | 5,849 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 531.2 | 139.7 | 161 ms | 7.2 ms | 4.1 | 4 | 15,937 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 665.1 | 86.4 | 348 ms | 11.6 ms | 11.9 | 0 | 19,947 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,205.6 | 71.8 | 1,784 ms | 13.9 ms | 16.0 | 0 | 36,135 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 151.5 | 146.2 | 174 ms | 6.9 ms | 1.0 | 1 | 4,544 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 253.7 | 129.0 | 230 ms | 7.8 ms | 5.3 | 1 | 7,609 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 482.8 | 124.4 | 412 ms | 8.0 ms | 4.8 | 4 | 14,420 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 556.2 | 73.5 | 829 ms | 13.6 ms | 11.6 | 0 | 16,687 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 155.8 | 150.4 | 289 ms | 6.7 ms | 1.0 | 1 | 4,672 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 195.5 | 96.5 | 568 ms | 10.4 ms | 6.9 | 0 | 5,864 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 444.8 | 110.8 | 951 ms | 9.0 ms | 4.1 | 1 | 13,335 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 496.3 | 65.0 | 1,840 ms | 15.4 ms | 9.0 | 0 | 14,886 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 133.1 | 143.7 | 529 ms | 7.0 ms | 1.0 | 1 | 3,991 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 123.5 | 63.5 | 1,047 ms | 15.8 ms | 9.1 | 0 | 3,703 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 295.9 | 77.6 | 1,862 ms | 12.9 ms | 5.2 | 0 | 8,873 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 107.9 | 112.7 | 1,050 ms | 8.9 ms | 1.0 | 0 | 3,237 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 175.1 | 87.7 | 1,925 ms | 11.4 ms | 3.7 | 0 | 5,254 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.