Speed test matrix

Qwen3.5-397B-A17B-NVFP4-V2 on H200 SXM 141GB ×4

vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 13, 2026, 02:25 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.28.1rc1.dev562+g9c2d21046
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 9c2d21046bb3

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nemotron-ultra-vllm-9c2d21046/bin/vllm" serve nvidia/Qwen3.5-397B-A17B-NVFP4-V2 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.863147 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype fp8_e4m3 --attention-config '{"disable_flashinfer_q_quantization":true}' --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-backend FLASHINFER --linear-backend cutlass --moe-backend emulation --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton","kv_cache_dtype":"bfloat16","attention_backend":"TRITON_ATTN"}' --quantization modelopt_mixed

Prefill

prompt tok/s by context length

8K context13.7K599 ms TTFT

16K context15K1,079 ms TTFT

32K context14.3K2,258 ms TTFT

64K context12.7K5,078 ms TTFT

128K context9,76913,174 ms TTFT

251K context6,62038,096 ms TTFT

256K context6,50939,516 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	82.9584 ms TTFT	164.0878 ms TTFT	310.71,429 ms TTFT	502.32,411 ms TTFT	803.04,191 ms TTFT	1,078.87,837 ms TTFT	1,646.015,236 ms TTFT	2,244.4Requests queued30,178 ms TTFT
16K	84.51,189 ms TTFT	162.11,449 ms TTFT	300.81,660 ms TTFT	492.82,614 ms TTFT	788.74,431 ms TTFT	1,067.28,062 ms TTFT	1,579.215,368 ms TTFT	2,363.0Requests queued30,272 ms TTFT
32K	81.2770 ms TTFT	162.51,075 ms TTFT	294.71,903 ms TTFT	483.92,960 ms TTFT	761.94,970 ms TTFT	1,039.28,982 ms TTFT	1,512.917,007 ms TTFT	1,923.1Requests queued33,448 ms TTFT
64K	87.35,223 ms TTFT	159.75,588 ms TTFT	276.32,304 ms TTFT	447.03,037 ms TTFT	705.85,149 ms TTFT	962.48,630 ms TTFT	1,410.916,401 ms TTFT	2,076.931,760 ms TTFT
128K	81.513,353 ms TTFT	155.613,899 ms TTFT	251.13,037 ms TTFT	402.94,878 ms TTFT	629.37,813 ms TTFT	862.414,349 ms TTFT	1,242.427,077 ms TTFT	1,746.853,406 ms TTFT
251K	64.538,366 ms TTFT	121.339,471 ms TTFT	195.75,822 ms TTFT	324.18,300 ms TTFT	498.313,289 ms TTFT	673.424,148 ms TTFT	920.245,962 ms TTFT	1,199.7Requests queued88,742 ms TTFT

Requests queued: the server could not run all requests at once, so some waited in a queue.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	13.7K	—	599 ms	—	—	1	—	—
Prefill	16K	1	Valid	15K	—	1,079 ms	—	—	1	—	—
Prefill	32K	1	Valid	14.3K	—	2,258 ms	—	—	1	—	—
Prefill	64K	1	Valid	12.7K	—	5,078 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,769.0	—	13,174 ms	—	—	1	—	—
Prefill	251K	1	Valid	6,620.0	—	38,096 ms	—	—	1	—	—
Prefill	256K	1	Valid	6,509.0	—	39,516 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	82.9	84.2	584 ms	11.9 ms	1.0	0	2,485	30.0 s
Sustained decode	8K	2	Valid	164.0	82.5	878 ms	12.1 ms	2.0	0	4,921	30.0 s
Sustained decode	8K	4	Valid	310.7	77.6	1,429 ms	12.9 ms	4.0	0	9,316	30.0 s
Sustained decode	8K	8	Valid	502.3	60.8	2,411 ms	16.5 ms	8.0	0	15,045	30.0 s
Sustained decode	8K	16	Valid	803.0	47.4	4,191 ms	21.1 ms	16.0	0	24,080	30.0 s
Sustained decode	8K	32	Valid	1,078.8	30.1	7,837 ms	33.2 ms	32.0	0	32,320	30.0 s
Sustained decode	8K	64	Valid	1,646.0	21.2	15,236 ms	47.3 ms	64.0	0	49,440	30.0 s
Sustained decode	8K	128	Requests queued	2,244.4	13.4	30,178 ms	74.5 ms	124.7	0	67,291	30.0 s
Sustained decode	16K	1	Valid	84.5	86.0	1,189 ms	11.6 ms	1.0	0	2,534	30.0 s
Sustained decode	16K	2	Valid	162.1	81.4	1,449 ms	12.3 ms	2.0	0	4,860	30.0 s
Sustained decode	16K	4	Valid	300.8	75.5	1,660 ms	13.2 ms	4.0	0	9,015	30.0 s
Sustained decode	16K	8	Valid	492.8	60.4	2,614 ms	16.6 ms	8.0	0	14,777	30.0 s
Sustained decode	16K	16	Valid	788.7	46.4	4,431 ms	21.5 ms	16.0	0	23,646	30.0 s
Sustained decode	16K	32	Valid	1,067.2	29.4	8,062 ms	34.1 ms	32.0	0	32,002	30.0 s
Sustained decode	16K	64	Valid	1,579.2	19.8	15,368 ms	50.5 ms	64.0	0	47,266	29.9 s
Sustained decode	16K	128	Requests queued	2,363.0	12.8	30,272 ms	78.1 ms	127.9	0	70,869	30.0 s
Sustained decode	32K	1	Valid	81.2	82.7	770 ms	12.1 ms	1.0	0	2,433	30.0 s
Sustained decode	32K	2	Valid	162.5	81.1	1,075 ms	12.3 ms	2.0	0	4,870	30.0 s
Sustained decode	32K	4	Valid	294.7	74.2	1,903 ms	13.5 ms	4.0	0	8,831	30.0 s
Sustained decode	32K	8	Valid	483.9	59.4	2,960 ms	16.8 ms	8.0	0	14,523	30.0 s
Sustained decode	32K	16	Valid	761.9	43.7	4,970 ms	22.9 ms	16.0	0	22,823	30.0 s
Sustained decode	32K	32	Valid	1,039.2	28.6	8,982 ms	35.0 ms	32.0	0	31,100	29.9 s
Sustained decode	32K	64	Valid	1,512.9	18.9	17,007 ms	53.0 ms	64.0	0	45,253	29.9 s
Sustained decode	32K	128	Requests queued	1,923.1	11.2	33,448 ms	89.4 ms	126.4	0	57,433	29.9 s
Sustained decode	64K	1	Valid	87.3	88.4	5,223 ms	11.3 ms	1.0	0	2,617	30.0 s
Sustained decode	64K	2	Valid	159.7	81.2	5,588 ms	12.3 ms	2.0	0	4,792	30.0 s
Sustained decode	64K	4	Valid	276.3	70.1	2,304 ms	14.3 ms	4.0	0	8,289	30.0 s
Sustained decode	64K	8	Valid	447.0	55.1	3,037 ms	18.1 ms	8.0	0	13,410	30.0 s
Sustained decode	64K	16	Valid	705.8	42.7	5,149 ms	23.4 ms	16.0	0	21,135	29.9 s
Sustained decode	64K	32	Valid	962.4	26.5	8,630 ms	37.8 ms	32.0	0	28,867	30.0 s
Sustained decode	64K	64	Valid	1,410.9	17.4	16,401 ms	57.4 ms	64.0	0	42,215	29.9 s
Sustained decode	64K	128	Valid	2,076.9	10.7	31,760 ms	93.2 ms	128.0	0	62,066	29.9 s
Sustained decode	128K	1	Valid	81.5	83.0	13,353 ms	12.0 ms	1.0	0	2,443	30.0 s
Sustained decode	128K	2	Valid	155.6	79.2	13,899 ms	12.6 ms	2.0	0	4,666	30.0 s
Sustained decode	128K	4	Valid	251.1	61.7	3,037 ms	16.2 ms	4.0	0	7,526	30.0 s
Sustained decode	128K	8	Valid	402.9	48.4	4,878 ms	20.7 ms	8.0	0	12,078	30.0 s
Sustained decode	128K	16	Valid	629.3	35.7	7,813 ms	28.0 ms	16.0	0	18,842	29.9 s
Sustained decode	128K	32	Valid	862.4	21.2	14,349 ms	47.1 ms	32.0	0	25,858	30.0 s
Sustained decode	128K	64	Valid	1,242.4	12.8	27,077 ms	78.4 ms	64.0	0	37,098	29.9 s
Sustained decode	128K	128	Valid	1,746.8	7.6	53,406 ms	132.5 ms	128.0	0	52,356	30.0 s
Sustained decode	251K	1	Valid	64.5	65.2	38,366 ms	15.3 ms	1.0	0	1,933	30.0 s
Sustained decode	251K	2	Valid	121.3	61.6	39,471 ms	16.2 ms	2.0	0	3,637	30.0 s
Sustained decode	251K	4	Valid	195.7	48.7	5,822 ms	20.5 ms	4.0	0	5,863	30.0 s
Sustained decode	251K	8	Valid	324.1	37.7	8,300 ms	26.5 ms	8.0	0	9,721	30.0 s
Sustained decode	251K	16	Valid	498.3	25.9	13,289 ms	38.6 ms	16.0	0	14,906	29.9 s
Sustained decode	251K	32	Valid	673.4	14.2	24,148 ms	70.6 ms	32.0	0	20,125	29.9 s
Sustained decode	251K	64	Valid	920.2	7.7	45,962 ms	129.4 ms	64.0	0	27,545	29.9 s
Sustained decode	251K	128	Requests queued	1,199.7	4.0	88,742 ms	247.0 ms	128.0	0	35,794	29.8 s

Requests queued: the server could not run all requests at once, so some waited in a queue.