Speed test matrix

Qwen3.8-Flash-Next-NVFP4 on H200 NVL 141GB ×2

sglang · 0.5.6.post3.dev9023+g73a255206 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 1, 2026, 02:35 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9023+g73a255206
Quantization: NVFP4
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 73a255206f91

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-radixark-qwen-sglang/bin/sglang" serve --model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen --tp 2 --reasoning-parser auto --tool-call-parser auto --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --chunked-prefill-size 4096 --max-running-requests 36 --allow-auto-truncate --fp4-gemm-backend flashinfer_cutlass --quantization modelopt_fp4

Prefill

prompt tok/s by context length

8K context18.5K443 ms TTFT

16K context19.4K834 ms TTFT

32K context19.7K1,630 ms TTFT

64K context19.5K3,288 ms TTFT

128K context18.6K6,879 ms TTFT

251K context16.1K15,611 ms TTFT

256K context16K16,024 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	129.2126 ms TTFT	216.5196 ms TTFT	456.5206 ms TTFT	824.7267 ms TTFT	1,415.3387 ms TTFT	2,235.0753 ms TTFT	Not run	Not run
16K	128.9160 ms TTFT	200.9248 ms TTFT	455.9309 ms TTFT	827.2418 ms TTFT	1,405.2723 ms TTFT	2,228.51,404 ms TTFT	Not run	Over capacity
32K	128.2217 ms TTFT	203.1287 ms TTFT	454.1440 ms TTFT	817.6837 ms TTFT	1,391.91,793 ms TTFT	2,206.63,199 ms TTFT	Over capacity	Over capacity
64K	127.0346 ms TTFT	195.8558 ms TTFT	449.3949 ms TTFT	808.51,983 ms TTFT	1,374.43,681 ms TTFT	Over capacity	Over capacity	Over capacity
128K	124.4587 ms TTFT	220.1997 ms TTFT	444.61,948 ms TTFT	795.63,755 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	122.91,059 ms TTFT	237.61,984 ms TTFT	426.83,828 ms TTFT	759.27,603 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	18.5K	—	443 ms	—	—	1	—	—
Prefill	16K	1	Valid	19.4K	—	834 ms	—	—	1	—	—
Prefill	32K	1	Valid	19.7K	—	1,630 ms	—	—	1	—	—
Prefill	64K	1	Valid	19.5K	—	3,288 ms	—	—	1	—	—
Prefill	128K	1	Valid	18.6K	—	6,879 ms	—	—	1	—	—
Prefill	251K	1	Valid	16.1K	—	15,611 ms	—	—	1	—	—
Prefill	256K	1	Valid	16K	—	16,024 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	129.2	129.9	126 ms	7.7 ms	1.0	1	3,877	30.0 s
Sustained decode	8K	2	Valid	216.5	108.2	196 ms	9.2 ms	3.0	0	6,496	30.0 s
Sustained decode	8K	4	Valid	456.5	113.9	206 ms	8.8 ms	4.0	0	13,692	30.0 s
Sustained decode	8K	8	Valid	824.7	103.1	267 ms	9.7 ms	8.0	0	24,712	30.0 s
Sustained decode	8K	16	Valid	1,415.3	86.9	387 ms	11.5 ms	16.0	0	42,464	30.0 s
Sustained decode	8K	32	Valid	2,235.0	68.7	753 ms	14.5 ms	32.0	0	67,008	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	128.9	129.7	160 ms	7.7 ms	1.0	1	3,863	30.0 s
Sustained decode	16K	2	Valid	200.9	95.7	248 ms	10.4 ms	12.4	0	6,028	30.0 s
Sustained decode	16K	4	Valid	455.9	112.3	309 ms	8.9 ms	4.1	0	13,664	30.0 s
Sustained decode	16K	8	Valid	827.2	103.1	418 ms	9.7 ms	8.0	0	24,816	30.0 s
Sustained decode	16K	16	Valid	1,405.2	86.2	723 ms	11.6 ms	16.3	0	42,144	30.0 s
Sustained decode	16K	32	Valid	2,228.5	68.9	1,404 ms	14.5 ms	32.0	0	66,848	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	128.2	129.2	217 ms	7.7 ms	1.0	1	3,842	30.0 s
Sustained decode	32K	2	Valid	203.1	96.7	287 ms	10.3 ms	11.7	0	6,086	30.0 s
Sustained decode	32K	4	Valid	454.1	111.9	440 ms	8.9 ms	4.1	0	13,620	30.0 s
Sustained decode	32K	8	Valid	817.6	102.3	837 ms	9.8 ms	8.0	0	24,528	30.0 s
Sustained decode	32K	16	Valid	1,391.9	86.0	1,793 ms	11.6 ms	16.3	0	41,760	30.0 s
Sustained decode	32K	32	Valid	2,206.6	68.1	3,199 ms	14.7 ms	32.0	0	66,208	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	127.0	128.5	346 ms	7.8 ms	1.0	1	3,807	30.0 s
Sustained decode	64K	2	Valid	195.8	93.9	558 ms	10.7 ms	12.8	0	5,870	30.0 s
Sustained decode	64K	4	Valid	449.3	111.2	949 ms	9.0 ms	4.1	0	13,480	30.0 s
Sustained decode	64K	8	Valid	808.5	102.6	1,983 ms	9.7 ms	8.0	0	24,256	30.0 s
Sustained decode	64K	16	Valid	1,374.4	86.9	3,681 ms	11.5 ms	16.0	0	41,233	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	124.4	126.9	587 ms	7.9 ms	1.0	1	3,732	30.0 s
Sustained decode	128K	2	Valid	220.1	105.7	997 ms	9.5 ms	6.3	0	6,604	30.0 s
Sustained decode	128K	4	Valid	444.6	112.8	1,948 ms	8.9 ms	4.0	0	13,328	30.0 s
Sustained decode	128K	8	Valid	795.6	103.1	3,755 ms	9.7 ms	8.0	0	23,864	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	122.9	122.8	1,059 ms	8.1 ms	1.0	1	3,603	29.3 s
Sustained decode	251K	2	Valid	237.6	118.2	1,984 ms	8.5 ms	2.0	0	7,130	30.0 s
Sustained decode	251K	4	Valid	426.8	110.6	3,828 ms	9.0 ms	4.0	0	12,804	30.0 s
Sustained decode	251K	8	Valid	759.2	103.4	7,603 ms	9.7 ms	8.0	0	22,776	30.0 s
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.