Speed test matrix
Qwen3.8-Flash-Next-FP8 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9023+g73a255206 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9023+g73a255206
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
73a255206f91
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-flash-next-sglang/bin/sglang" serve --model-path Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tp 2 --reasoning-parser auto --tool-call-parser auto --context-length 262144 --mem-fraction-static 0.867313 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --ep 2 --chunked-prefill-size 8192 --linear-attn-prefill-backend flashinfer --linear-attn-decode-backend flashinfer --mamba-ssm-dtype bfloat16 --max-running-requests 96
Prefill
prompt tok/s by context length8K context24.6K333 ms TTFT
16K context26.8K603 ms TTFT
32K context27.4K1,175 ms TTFT
64K context26.3K2,435 ms TTFT
128K context25.6K4,993 ms TTFT
251K context21.3K11,795 ms TTFT
256K context21K12,166 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 117.7145 ms TTFT | 227.8229 ms TTFT | 410.71,758 ms TTFT | 704.1317 ms TTFT | 1,113.0397 ms TTFT | 1,501.2698 ms TTFT | 2,135.31,332 ms TTFT | Not run |
| 16K | 117.5181 ms TTFT | 75.0301 ms TTFT | 306.7375 ms TTFT | 669.8414 ms TTFT | 1,089.7674 ms TTFT | 1,473.01,301 ms TTFT | 2,130.72,635 ms TTFT | Over capacity |
| 32K | 117.1313 ms TTFT | 74.7372 ms TTFT | 309.9440 ms TTFT | 666.4835 ms TTFT | 1,068.51,689 ms TTFT | 1,465.53,126 ms TTFT | 2,104.16,218 ms TTFT | Over capacity |
| 64K | 116.4419 ms TTFT | 73.9584 ms TTFT | 312.31,021 ms TTFT | 666.51,852 ms TTFT | 1,072.63,659 ms TTFT | 1,485.97,644 ms TTFT | Over capacity | Over capacity |
| 128K | 115.3629 ms TTFT | 94.41,027 ms TTFT | 319.91,942 ms TTFT | 655.63,810 ms TTFT | 1,069.97,373 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 112.01,102 ms TTFT | 144.31,990 ms TTFT | 368.23,918 ms TTFT | 649.57,772 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.6K | — | 333 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.8K | — | 603 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.4K | — | 1,175 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.3K | — | 2,435 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 25.6K | — | 4,993 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 21.3K | — | 11,795 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 21K | — | 12,166 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 117.7 | 117.7 | 145 ms | 8.5 ms | 1.0 | 0 | 3,533 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 227.8 | 113.9 | 229 ms | 8.8 ms | 2.0 | 0 | 6,836 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 410.7 | 102.7 | 1,758 ms | 9.7 ms | 4.0 | 0 | 12,324 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 704.1 | 86.8 | 317 ms | 11.5 ms | 8.1 | 0 | 21,096 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,113.0 | 68.5 | 397 ms | 14.6 ms | 18.9 | 0 | 33,376 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,501.2 | 46.7 | 698 ms | 21.4 ms | 47.5 | 0 | 44,992 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,135.3 | 33.4 | 1,332 ms | 30.0 ms | 96.0 | 0 | 64,064 | 30.0 s |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 117.5 | 117.6 | 181 ms | 8.5 ms | 1.0 | 0 | 3,526 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 75.0 | 37.5 | 301 ms | 26.7 ms | 66.0 | 0 | 2,246 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 306.7 | 71.8 | 375 ms | 13.9 ms | 18.8 | 0 | 9,201 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 669.8 | 82.7 | 414 ms | 12.1 ms | 9.9 | 0 | 20,104 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,089.7 | 67.1 | 674 ms | 14.9 ms | 19.5 | 0 | 32,688 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,473.0 | 46.0 | 1,301 ms | 21.7 ms | 48.0 | 0 | 44,128 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,130.7 | 33.3 | 2,635 ms | 30.0 ms | 96.0 | 0 | 63,872 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 117.1 | 117.1 | 313 ms | 8.5 ms | 1.0 | 0 | 3,509 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 74.7 | 37.3 | 372 ms | 26.8 ms | 66.0 | 0 | 2,240 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 309.9 | 72.5 | 440 ms | 13.8 ms | 16.7 | 0 | 9,296 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 666.4 | 82.4 | 835 ms | 12.1 ms | 9.9 | 0 | 19,992 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,068.5 | 66.1 | 1,689 ms | 15.1 ms | 19.7 | 0 | 32,048 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,465.5 | 45.9 | 3,126 ms | 21.8 ms | 48.0 | 0 | 43,913 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,104.1 | 33.1 | 6,218 ms | 30.2 ms | 96.0 | 0 | 63,104 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 116.4 | 116.4 | 419 ms | 8.6 ms | 1.0 | 0 | 3,490 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 73.9 | 36.9 | 584 ms | 27.1 ms | 66.0 | 0 | 2,214 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 312.3 | 73.8 | 1,021 ms | 13.5 ms | 16.7 | 0 | 9,360 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 666.5 | 83.5 | 1,852 ms | 12.0 ms | 9.7 | 0 | 19,984 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,072.6 | 67.9 | 3,659 ms | 14.7 ms | 19.3 | 0 | 32,176 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,485.9 | 49.1 | 7,644 ms | 20.3 ms | 46.0 | 0 | 44,576 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 115.3 | 115.3 | 629 ms | 8.7 ms | 1.0 | 0 | 3,456 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 94.4 | 47.4 | 1,027 ms | 21.1 ms | 34.0 | 0 | 2,832 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 319.9 | 77.1 | 1,942 ms | 13.0 ms | 11.0 | 0 | 9,596 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 655.6 | 83.9 | 3,810 ms | 11.9 ms | 9.2 | 0 | 19,656 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,069.9 | 70.1 | 7,373 ms | 14.3 ms | 18.0 | 0 | 32,080 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 112.0 | 112.0 | 1,102 ms | 8.9 ms | 1.0 | 0 | 3,361 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 144.3 | 70.9 | 1,990 ms | 14.1 ms | 12.8 | 0 | 4,326 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 368.2 | 93.9 | 3,918 ms | 10.7 ms | 4.6 | 0 | 11,044 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 649.5 | 86.9 | 7,772 ms | 11.5 ms | 8.0 | 0 | 19,464 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.