Speed test matrix

DeepSeek-V4.1-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

vllm · 0.30.1rc1.dev147+g02ee13be2 · NVFP4+MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 27, 2026, 22:02 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev147+g02ee13be2
Quantization: NVFP4+MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 02ee13be2107

Run

Served model: nvidia/DeepSeek-V4.1-Flash-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-deepseek-v41-vllm-pr-56509/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-NVFP4 --served-model-name nvidia/DeepSeek-V4.1-Flash-NVFP4 --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --enable-chunked-prefill --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --no-enable-prefix-caching

Prefill

prompt tok/s by context length

8K context9,714843 ms TTFT

16K context9,8851,640 ms TTFT

32K context9,8813,264 ms TTFT

64K context9,7026,633 ms TTFT

128K context9,28213,846 ms TTFT

256K context8,49130,255 ms TTFT

512K context7,23570,994 ms TTFT

1019K context5,790176,509 ms TTFT

1024K context5,765178,156 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	115.7828 ms TTFT	205.41,249 ms TTFT	269.02,052 ms TTFT	322.54,063 ms TTFT	209.17,720 ms TTFT	370.914,142 ms TTFT	590.627,192 ms TTFT	112.8Requests queued45,063 ms TTFT
16K	114.01,632 ms TTFT	204.32,451 ms TTFT	264.54,050 ms TTFT	322.47,287 ms TTFT	208.413,771 ms TTFT	370.226,636 ms TTFT	57.0Requests queued44,428 ms TTFT	56.6Requests queued44,477 ms TTFT
32K	112.43,257 ms TTFT	202.24,900 ms TTFT	261.18,202 ms TTFT	320.114,666 ms TTFT	208.727,666 ms TTFT	27.3Requests queued44,096 ms TTFT	27.0Requests queued43,939 ms TTFT	27.6Requests queued44,038 ms TTFT
64K	112.26,613 ms TTFT	199.99,940 ms TTFT	261.116,697 ms TTFT	317.729,862 ms TTFT	207.455,536 ms TTFT	23.7Requests queued72,131 ms TTFT	23.5Requests queued72,249 ms TTFT	Over capacity
128K	112.513,835 ms TTFT	198.720,741 ms TTFT	258.034,931 ms TTFT	311.861,709 ms TTFT	15.0Requests queued102,878 ms TTFT	14.8Requests queued102,599 ms TTFT	Over capacity	Over capacity
256K	112.730,245 ms TTFT	195.445,383 ms TTFT	252.975,637 ms TTFT	5.4Requests queued90,821 ms TTFT	5.4Requests queued90,966 ms TTFT	Over capacity	Over capacity	Over capacity
512K	110.570,977 ms TTFT	0.8Requests queued71,171 ms TTFT	0.8Requests queued71,056 ms TTFT	0.8Requests queued71,172 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
1019K	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,714.0	—	843 ms	—	—	1	—	—
Prefill	16K	1	Valid	9,885.0	—	1,640 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,881.0	—	3,264 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,702.0	—	6,633 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,282.0	—	13,846 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,491.0	—	30,255 ms	—	—	1	—	—
Prefill	512K	1	Valid	7,235.0	—	70,994 ms	—	—	1	—	—
Prefill	1019K	1	Valid	5,790.0	—	176,509 ms	—	—	1	—	—
Prefill	1024K	1	Valid	5,765.0	—	178,156 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	115.7	115.7	828 ms	8.6 ms	1.0	0	3,470	30.0 s
Sustained decode	8K	2	Valid	205.4	101.6	1,249 ms	9.8 ms	2.0	0	6,160	30.0 s
Sustained decode	8K	4	Valid	269.0	65.0	2,052 ms	15.4 ms	4.0	0	8,068	30.0 s
Sustained decode	8K	8	Valid	322.5	37.7	4,063 ms	26.5 ms	8.0	0	9,673	30.0 s
Sustained decode	8K	16	Valid	209.1	11.4	7,720 ms	87.4 ms	16.0	0	6,256	29.9 s
Sustained decode	8K	32	Valid	370.9	8.9	14,142 ms	112.0 ms	32.0	0	11,104	29.9 s
Sustained decode	8K	64	Valid	590.6	5.9	27,192 ms	170.3 ms	64.0	0	17,665	29.9 s
Sustained decode	8K	128	Requests queued	112.8	1.2	45,063 ms	811.8 ms	91.2	0	3,328	29.5 s
Sustained decode	16K	1	Valid	114.0	114.1	1,632 ms	8.8 ms	1.0	0	3,421	30.0 s
Sustained decode	16K	2	Valid	204.3	99.9	2,451 ms	10.0 ms	2.0	0	6,130	30.0 s
Sustained decode	16K	4	Valid	264.5	61.9	4,050 ms	16.2 ms	4.0	0	7,932	30.0 s
Sustained decode	16K	8	Valid	322.4	34.9	7,287 ms	28.7 ms	8.0	0	9,672	30.0 s
Sustained decode	16K	16	Valid	208.4	9.9	13,771 ms	100.9 ms	16.0	0	6,239	29.9 s
Sustained decode	16K	32	Valid	370.2	7.2	26,636 ms	138.5 ms	32.0	0	11,105	30.0 s
Sustained decode	16K	64	Requests queued	57.0	1.2	44,428 ms	807.6 ms	46.0	0	1,710	30.0 s
Sustained decode	16K	128	Requests queued	56.6	1.2	44,477 ms	807.9 ms	46.0	0	1,656	29.3 s
Sustained decode	32K	1	Valid	112.4	112.4	3,257 ms	8.9 ms	1.0	0	3,371	30.0 s
Sustained decode	32K	2	Valid	202.2	96.8	4,900 ms	10.3 ms	2.0	0	6,067	30.0 s
Sustained decode	32K	4	Valid	261.1	57.7	8,202 ms	17.4 ms	4.0	0	7,832	30.0 s
Sustained decode	32K	8	Valid	320.1	30.4	14,666 ms	32.9 ms	8.0	0	9,599	30.0 s
Sustained decode	32K	16	Valid	208.7	8.1	27,666 ms	122.8 ms	16.0	0	6,256	30.0 s
Sustained decode	32K	32	Requests queued	27.3	1.2	44,096 ms	815.2 ms	22.9	0	807	29.5 s
Sustained decode	32K	64	Requests queued	27.0	1.2	43,939 ms	813.2 ms	23.0	0	791	29.3 s
Sustained decode	32K	128	Requests queued	27.6	1.2	44,038 ms	812.7 ms	23.0	0	809	29.3 s
Sustained decode	64K	1	Valid	112.2	112.4	6,613 ms	8.9 ms	1.0	0	3,364	30.0 s
Sustained decode	64K	2	Valid	199.9	92.1	9,940 ms	10.9 ms	2.0	0	5,995	30.0 s
Sustained decode	64K	4	Valid	261.1	51.5	16,697 ms	19.5 ms	4.0	0	7,833	30.0 s
Sustained decode	64K	8	Valid	317.7	24.5	29,862 ms	40.9 ms	8.0	0	9,528	30.0 s
Sustained decode	64K	16	Valid	207.4	6.0	55,536 ms	167.9 ms	16.0	0	6,208	29.9 s
Sustained decode	64K	32	Requests queued	23.7	1.2	72,131 ms	828.6 ms	20.2	0	710	29.9 s
Sustained decode	64K	64	Requests queued	23.5	1.2	72,249 ms	828.3 ms	20.1	0	686	29.2 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	112.5	112.8	13,835 ms	8.9 ms	1.0	0	3,376	30.0 s
Sustained decode	128K	2	Valid	198.7	85.4	20,741 ms	12.0 ms	2.0	0	5,961	30.0 s
Sustained decode	128K	4	Valid	258.0	41.6	34,931 ms	24.5 ms	4.0	0	7,736	30.0 s
Sustained decode	128K	8	Valid	311.8	16.9	61,709 ms	59.4 ms	8.0	0	9,353	30.0 s
Sustained decode	128K	16	Requests queued	15.0	1.2	102,878 ms	868.2 ms	13.8	0	447	29.9 s
Sustained decode	128K	32	Requests queued	14.8	1.2	102,599 ms	865.5 ms	13.8	0	435	29.3 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	112.7	113.0	30,245 ms	8.9 ms	1.0	0	3,382	30.0 s
Sustained decode	256K	2	Valid	195.4	75.1	45,383 ms	14.7 ms	2.0	0	5,860	30.0 s
Sustained decode	256K	4	Valid	252.9	29.0	75,637 ms	35.8 ms	4.0	0	7,587	30.0 s
Sustained decode	256K	8	Requests queued	5.4	1.0	90,821 ms	952.6 ms	6.0	0	160	29.8 s
Sustained decode	256K	16	Requests queued	5.4	1.0	90,966 ms	952.6 ms	6.0	0	160	29.8 s
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	110.5	110.9	70,977 ms	9.0 ms	1.0	0	3,315	30.0 s
Sustained decode	512K	2	Requests queued	0.8	0.9	71,171 ms	1106.6 ms	2.0	0	23	29.2 s
Sustained decode	512K	4	Requests queued	0.8	0.9	71,056 ms	1104.9 ms	2.0	0	23	29.1 s
Sustained decode	512K	8	Requests queued	0.8	0.9	71,172 ms	1104.7 ms	2.0	0	23	29.1 s
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Over capacity	—	—	—	—	0.9	0	0	30.0 s
Sustained decode	1019K	2	Over capacity	—	—	—	—	1.0	0	0	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	1.0	0	0	30.0 s
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.