Speed test matrix

Qwen3.5-397B-A17B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 13, 2026, 16:45 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.7.9
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-397B-A17B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Prefill

prompt tok/s by context length

8K context9,947823 ms TTFT

16K context10.3K1,580 ms TTFT

32K context10.2K3,156 ms TTFT

64K context9,8456,543 ms TTFT

128K context9,28113,866 ms TTFT

251K context8,25430,554 ms TTFT

256K context8,10831,723 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	129.5144 ms TTFT	235.1216 ms TTFT	423.2417 ms TTFT	702.0830 ms TTFT	1,033.6919 ms TTFT	1,375.61,632 ms TTFT	2,267.53,141 ms TTFT	2,942.35,509 ms TTFT
16K	102.1166 ms TTFT	198.0267 ms TTFT	354.7475 ms TTFT	603.2829 ms TTFT	911.21,002 ms TTFT	1,284.01,972 ms TTFT	1,991.53,496 ms TTFT	Over capacity
32K	72.3191 ms TTFT	142.1278 ms TTFT	265.8450 ms TTFT	472.1519 ms TTFT	762.81,193 ms TTFT	1,119.91,512 ms TTFT	Over capacity	Over capacity
64K	46.0247 ms TTFT	93.6360 ms TTFT	170.9500 ms TTFT	328.1796 ms TTFT	559.01,239 ms TTFT	Over capacity	Over capacity	Over capacity
128K	26.3444 ms TTFT	53.1629 ms TTFT	103.3874 ms TTFT	204.71,365 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	14.9832 ms TTFT	29.01,141 ms TTFT	56.41,684 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,947.0	—	823 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.3K	—	1,580 ms	—	—	1	—	—
Prefill	32K	1	Valid	10.2K	—	3,156 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,845.0	—	6,543 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,281.0	—	13,866 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,254.0	—	30,554 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,108.0	—	31,723 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	129.5	139.5	144 ms	7.2 ms	1.0	1	3,883	30.0 s
Sustained decode	8K	2	Valid	235.1	114.8	216 ms	8.7 ms	2.0	0	7,050	30.0 s
Sustained decode	8K	4	Valid	423.2	102.8	417 ms	9.7 ms	4.0	0	12,694	30.0 s
Sustained decode	8K	8	Valid	702.0	87.2	830 ms	11.5 ms	8.0	0	21,044	30.0 s
Sustained decode	8K	16	Valid	1,033.6	63.1	919 ms	15.9 ms	16.0	0	30,974	30.0 s
Sustained decode	8K	32	Valid	1,375.6	43.1	1,632 ms	23.2 ms	32.0	0	41,221	30.0 s
Sustained decode	8K	64	Valid	2,267.5	34.6	3,141 ms	28.9 ms	64.0	0	67,895	29.9 s
Sustained decode	8K	128	Valid	2,942.3	21.1	5,509 ms	47.4 ms	128.0	0	87,991	29.9 s
Sustained decode	16K	1	Valid	102.1	103.2	166 ms	9.7 ms	1.0	0	3,060	30.0 s
Sustained decode	16K	2	Valid	198.0	100.1	267 ms	10.0 ms	2.0	0	5,939	30.0 s
Sustained decode	16K	4	Valid	354.7	89.5	475 ms	11.2 ms	4.0	0	10,636	30.0 s
Sustained decode	16K	8	Valid	603.2	76.6	829 ms	13.1 ms	8.0	0	18,096	30.0 s
Sustained decode	16K	16	Valid	911.2	57.4	1,002 ms	17.4 ms	16.0	0	27,300	30.0 s
Sustained decode	16K	32	Valid	1,284.0	39.9	1,972 ms	25.1 ms	32.0	0	38,502	30.0 s
Sustained decode	16K	64	Valid	1,991.5	30.0	3,496 ms	33.3 ms	64.0	0	59,679	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	72.3	73.1	191 ms	13.7 ms	1.0	0	2,166	30.0 s
Sustained decode	32K	2	Valid	142.1	72.0	278 ms	13.9 ms	2.0	0	4,264	30.0 s
Sustained decode	32K	4	Valid	265.8	67.1	450 ms	14.9 ms	4.0	0	7,971	30.0 s
Sustained decode	32K	8	Valid	472.1	59.5	519 ms	16.8 ms	8.0	0	14,150	30.0 s
Sustained decode	32K	16	Valid	762.8	48.1	1,193 ms	20.8 ms	16.0	0	22,864	30.0 s
Sustained decode	32K	32	Valid	1,119.9	35.1	1,512 ms	28.5 ms	32.0	0	33,575	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	46.0	46.7	247 ms	21.4 ms	1.0	0	1,379	30.0 s
Sustained decode	64K	2	Valid	93.6	47.6	360 ms	21.0 ms	2.0	0	2,805	30.0 s
Sustained decode	64K	4	Valid	170.9	43.0	500 ms	23.3 ms	4.0	0	5,120	30.0 s
Sustained decode	64K	8	Valid	328.1	41.3	796 ms	24.2 ms	8.0	0	9,825	29.9 s
Sustained decode	64K	16	Valid	559.0	35.1	1,239 ms	28.5 ms	16.0	0	16,764	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	26.3	26.8	444 ms	37.4 ms	1.0	0	786	29.9 s
Sustained decode	128K	2	Valid	53.1	27.1	629 ms	36.9 ms	2.0	0	1,590	29.9 s
Sustained decode	128K	4	Valid	103.3	25.7	874 ms	38.9 ms	4.0	0	3,091	29.9 s
Sustained decode	128K	8	Valid	204.7	25.9	1,365 ms	38.6 ms	8.0	0	6,126	29.9 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	14.9	15.1	832 ms	66.1 ms	1.0	0	445	29.9 s
Sustained decode	251K	2	Valid	29.0	14.7	1,141 ms	67.9 ms	2.0	0	865	29.8 s
Sustained decode	251K	4	Valid	56.4	14.4	1,684 ms	69.7 ms	4.0	0	1,691	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.