Speed test matrix

DeepSeek-V4.1-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

vllm · 0.30.1rc1.dev147+g02ee13be2 · NVFP4+MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 27, 2026, 19:59 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev147+g02ee13be2
Quantization: NVFP4+MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 02ee13be2107

Run

Served model: nvidia/DeepSeek-V4.1-Flash-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-deepseek-v41-vllm-pr-56509/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-NVFP4 --served-model-name nvidia/DeepSeek-V4.1-Flash-NVFP4 --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":false}' --enable-chunked-prefill --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --no-enable-prefix-caching

Prefill

prompt tok/s by context length

8K context9,946823 ms TTFT

16K context10.1K1,610 ms TTFT

32K context10K3,217 ms TTFT

64K context9,8736,518 ms TTFT

128K context9,45713,589 ms TTFT

251K context8,64129,148 ms TTFT

256K context8,52930,119 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	115.3816 ms TTFT	207.61,214 ms TTFT	268.92,013 ms TTFT	323.73,998 ms TTFT	206.97,562 ms TTFT	376.313,910 ms TTFT	597.226,641 ms TTFT	Over capacity
16K	114.31,610 ms TTFT	207.32,409 ms TTFT	268.34,001 ms TTFT	326.97,175 ms TTFT	207.413,520 ms TTFT	371.526,210 ms TTFT	Over capacity	Over capacity
32K	113.73,207 ms TTFT	206.34,805 ms TTFT	267.78,035 ms TTFT	326.014,429 ms TTFT	211.127,208 ms TTFT	Over capacity	Over capacity	Over capacity
64K	113.36,512 ms TTFT	205.39,768 ms TTFT	268.816,382 ms TTFT	323.329,405 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	114.213,586 ms TTFT	204.220,418 ms TTFT	265.234,271 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	114.129,174 ms TTFT	199.843,573 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,946.0	—	823 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.1K	—	1,610 ms	—	—	1	—	—
Prefill	32K	1	Valid	10K	—	3,217 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,873.0	—	6,518 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,457.0	—	13,589 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,641.0	—	29,148 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,529.0	—	30,119 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	115.3	115.4	816 ms	8.7 ms	1.0	0	3,458	30.0 s
Sustained decode	8K	2	Valid	207.6	102.7	1,214 ms	9.7 ms	2.0	0	6,226	30.0 s
Sustained decode	8K	4	Valid	268.9	65.1	2,013 ms	15.4 ms	4.0	0	8,066	30.0 s
Sustained decode	8K	8	Valid	323.7	37.8	3,998 ms	26.4 ms	8.0	0	9,705	30.0 s
Sustained decode	8K	16	Valid	206.9	11.4	7,562 ms	87.9 ms	16.0	0	6,192	29.9 s
Sustained decode	8K	32	Valid	376.3	9.0	13,910 ms	110.6 ms	32.0	0	11,263	29.9 s
Sustained decode	8K	64	Valid	597.2	6.0	26,641 ms	166.5 ms	64.0	0	17,854	29.9 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	114.3	114.3	1,610 ms	8.8 ms	1.0	0	3,429	30.0 s
Sustained decode	16K	2	Valid	207.3	101.4	2,409 ms	9.9 ms	2.0	0	6,218	30.0 s
Sustained decode	16K	4	Valid	268.3	62.9	4,001 ms	15.9 ms	4.0	0	8,049	30.0 s
Sustained decode	16K	8	Valid	326.9	35.5	7,175 ms	28.2 ms	8.0	0	9,807	30.0 s
Sustained decode	16K	16	Valid	207.4	10.0	13,520 ms	100.3 ms	16.0	0	6,208	29.9 s
Sustained decode	16K	32	Valid	371.5	7.3	26,210 ms	137.1 ms	32.0	0	11,136	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	113.7	113.7	3,207 ms	8.8 ms	1.0	0	3,409	30.0 s
Sustained decode	32K	2	Valid	206.3	98.9	4,805 ms	10.1 ms	2.0	0	6,188	30.0 s
Sustained decode	32K	4	Valid	267.7	59.3	8,035 ms	16.9 ms	4.0	0	8,028	30.0 s
Sustained decode	32K	8	Valid	326.0	31.2	14,429 ms	32.1 ms	8.0	0	9,776	30.0 s
Sustained decode	32K	16	Valid	211.1	8.3	27,208 ms	121.0 ms	16.0	0	6,319	29.9 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	113.3	113.6	6,512 ms	8.8 ms	1.0	0	3,400	30.0 s
Sustained decode	64K	2	Valid	205.3	94.8	9,768 ms	10.6 ms	2.0	0	6,160	30.0 s
Sustained decode	64K	4	Valid	268.8	53.1	16,382 ms	19.0 ms	4.0	0	8,060	30.0 s
Sustained decode	64K	8	Valid	323.3	25.1	29,405 ms	40.0 ms	8.0	0	9,697	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	114.2	114.5	13,586 ms	8.7 ms	1.0	0	3,425	30.0 s
Sustained decode	128K	2	Valid	204.2	88.1	20,418 ms	11.7 ms	2.0	0	6,125	30.0 s
Sustained decode	128K	4	Valid	265.2	43.0	34,271 ms	23.6 ms	4.0	0	7,956	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	114.1	114.4	29,174 ms	8.7 ms	1.0	0	3,423	30.0 s
Sustained decode	251K	2	Valid	199.8	77.3	43,573 ms	14.2 ms	2.0	0	5,994	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.