Speed test matrix

Qwen3.8-Flash-Next-NVFP4 on H200 SXM 141GB ×4

vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 13, 2026, 05:36 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.28.1rc1.dev562+g9c2d21046
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 9c2d21046bb3

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nemotron-ultra-vllm-9c2d21046/bin/vllm" serve nvidia/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.840483 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --kv-cache-dtype bfloat16 --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-config '{"indexer_kv_dtype":"bf16"}' --no-enable-flashinfer-autotune --moe-backend emulation --speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}' --quantization modelopt

Prefill

prompt tok/s by context length

8K context19.2K430 ms TTFT

16K context23.2K700 ms TTFT

32K context26.2K1,236 ms TTFT

64K context28.4K2,268 ms TTFT

128K context28.5K4,521 ms TTFT

251K context27.3K9,247 ms TTFT

256K context26.9K9,570 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	174.6177 ms TTFT	286.8202 ms TTFT	495.0375 ms TTFT	904.5520 ms TTFT	1,520.6627 ms TTFT	2,252.71,008 ms TTFT	3,786.31,561 ms TTFT	5,893.32,728 ms TTFT
16K	169.5211 ms TTFT	288.4242 ms TTFT	504.6325 ms TTFT	894.6508 ms TTFT	1,491.9912 ms TTFT	2,241.31,237 ms TTFT	3,753.41,729 ms TTFT	5,873.42,746 ms TTFT
32K	172.9257 ms TTFT	279.9268 ms TTFT	502.3283 ms TTFT	903.3644 ms TTFT	1,493.51,059 ms TTFT	2,245.71,368 ms TTFT	3,751.22,349 ms TTFT	Looped
64K	171.0355 ms TTFT	283.1356 ms TTFT	502.4393 ms TTFT	886.1921 ms TTFT	1,497.31,331 ms TTFT	2,196.12,494 ms TTFT	3,713.84,362 ms TTFT	Over capacity
128K	172.6544 ms TTFT	274.1697 ms TTFT	492.8736 ms TTFT	885.21,521 ms TTFT	1,432.22,553 ms TTFT	2,196.74,465 ms TTFT	Over capacity	Over capacity
251K	172.3997 ms TTFT	274.7986 ms TTFT	489.71,081 ms TTFT	877.32,844 ms TTFT	1,406.34,980 ms TTFT	Over capacity	Over capacity	Over capacity

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	19.2K	—	430 ms	—	—	1	—	—
Prefill	16K	1	Valid	23.2K	—	700 ms	—	—	1	—	—
Prefill	32K	1	Valid	26.2K	—	1,236 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.4K	—	2,268 ms	—	—	1	—	—
Prefill	128K	1	Valid	28.5K	—	4,521 ms	—	—	1	—	—
Prefill	251K	1	Valid	27.3K	—	9,247 ms	—	—	1	—	—
Prefill	256K	1	Valid	26.9K	—	9,570 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	174.6	175.9	177 ms	5.7 ms	1.0	1	5,236	30.0 s
Sustained decode	8K	2	Valid	286.8	143.8	202 ms	7.0 ms	2.0	2	8,605	30.0 s
Sustained decode	8K	4	Valid	495.0	127.1	375 ms	7.9 ms	4.0	4	14,845	30.0 s
Sustained decode	8K	8	Valid	904.5	112.8	520 ms	8.9 ms	8.0	0	27,125	30.0 s
Sustained decode	8K	16	Valid	1,520.6	95.0	627 ms	10.5 ms	16.0	0	45,650	30.0 s
Sustained decode	8K	32	Valid	2,252.7	70.6	1,008 ms	14.2 ms	32.0	0	67,639	30.0 s
Sustained decode	8K	64	Valid	3,786.3	58.7	1,561 ms	17.0 ms	64.0	0	113,577	30.0 s
Sustained decode	8K	128	Valid	5,893.3	44.3	2,728 ms	22.6 ms	128.0	0	177,076	30.0 s
Sustained decode	16K	1	Valid	169.5	171.0	211 ms	5.8 ms	1.0	1	5,084	30.0 s
Sustained decode	16K	2	Valid	288.4	144.7	242 ms	6.9 ms	2.0	2	8,653	30.0 s
Sustained decode	16K	4	Valid	504.6	127.0	325 ms	7.9 ms	4.0	4	15,133	30.0 s
Sustained decode	16K	8	Valid	894.6	112.1	508 ms	8.9 ms	8.0	0	26,828	30.0 s
Sustained decode	16K	16	Valid	1,491.9	92.5	912 ms	10.8 ms	16.0	0	44,740	30.0 s
Sustained decode	16K	32	Valid	2,241.3	69.8	1,237 ms	14.3 ms	32.0	0	67,223	30.0 s
Sustained decode	16K	64	Valid	3,753.4	57.9	1,729 ms	17.3 ms	64.0	0	112,579	30.0 s
Sustained decode	16K	128	Valid	5,873.4	44.1	2,746 ms	22.7 ms	128.0	0	176,207	30.0 s
Sustained decode	32K	1	Valid	172.9	175.5	257 ms	5.7 ms	1.0	1	5,187	30.0 s
Sustained decode	32K	2	Valid	279.9	141.2	268 ms	7.1 ms	2.0	2	8,395	30.0 s
Sustained decode	32K	4	Valid	502.3	123.7	283 ms	8.1 ms	4.0	4	15,065	30.0 s
Sustained decode	32K	8	Valid	903.3	112.4	644 ms	8.9 ms	8.0	0	27,098	30.0 s
Sustained decode	32K	16	Valid	1,493.5	92.4	1,059 ms	10.8 ms	16.0	0	44,808	30.0 s
Sustained decode	32K	32	Valid	2,245.7	68.9	1,368 ms	14.5 ms	32.0	0	67,340	30.0 s
Sustained decode	32K	64	Valid	3,751.2	57.2	2,349 ms	17.5 ms	64.0	0	112,520	30.0 s
Sustained decode	32K	128	Looped	—	—	—	—	—	0	0	—
Sustained decode	64K	1	Valid	171.0	173.4	355 ms	5.8 ms	1.0	1	5,128	30.0 s
Sustained decode	64K	2	Valid	283.1	141.1	356 ms	7.1 ms	2.0	2	8,489	30.0 s
Sustained decode	64K	4	Valid	502.4	125.6	393 ms	8.0 ms	4.0	4	15,067	30.0 s
Sustained decode	64K	8	Valid	886.1	109.5	921 ms	9.1 ms	8.0	0	26,584	30.0 s
Sustained decode	64K	16	Valid	1,497.3	91.7	1,331 ms	10.9 ms	16.0	0	44,916	30.0 s
Sustained decode	64K	32	Valid	2,196.1	66.6	2,494 ms	15.0 ms	32.0	0	65,895	30.0 s
Sustained decode	64K	64	Valid	3,713.8	53.6	4,362 ms	18.7 ms	64.0	0	111,356	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	172.6	175.1	544 ms	5.7 ms	1.0	1	5,178	30.0 s
Sustained decode	128K	2	Valid	274.1	141.7	697 ms	7.1 ms	1.9	2	8,220	30.0 s
Sustained decode	128K	4	Valid	492.8	124.6	736 ms	8.0 ms	4.0	4	14,759	29.9 s
Sustained decode	128K	8	Valid	885.2	108.8	1,521 ms	9.2 ms	8.0	0	26,572	30.0 s
Sustained decode	128K	16	Valid	1,432.2	86.6	2,553 ms	11.5 ms	16.0	0	42,948	30.0 s
Sustained decode	128K	32	Valid	2,196.7	64.4	4,465 ms	15.5 ms	32.0	0	65,901	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	172.3	179.3	997 ms	5.6 ms	1.0	1	5,167	30.0 s
Sustained decode	251K	2	Valid	274.7	142.1	986 ms	7.0 ms	1.9	2	8,240	30.0 s
Sustained decode	251K	4	Valid	489.7	122.2	1,081 ms	8.2 ms	4.0	4	14,692	30.0 s
Sustained decode	251K	8	Valid	877.3	107.0	2,844 ms	9.3 ms	8.0	0	26,314	30.0 s
Sustained decode	251K	16	Valid	1,406.3	84.8	4,980 ms	11.8 ms	16.0	0	42,172	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.