Speed test matrix
DeepSeek-V4.1-Flash-1017-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.33.0rc1.dev141 · NVFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.33.0rc1.dev141
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.0
- Commit
48dcffbb7c8f
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
# Reconstructed from current recipes/nvidia-rtxpro6k/nvidia/vllm_nvidia_DeepSeek-V4.1-Flash-1017-NVFP4_ngram_ram_rtxpro6kx4.sh; not captured runtime argv.
env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-deepseek-v41-1017-vllm-pr-60762/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-1017-NVFP4 --served-model-name deepseek --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --max-model-len 524288 --gpu-memory-utilization 0.950000 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"fp8","indexer_sparse_logits":false}' --kv-cache-dtype fp8 --max-num-seqs 128 --engram-config '{"cpu_offload":true}' --enable-prefix-cachingPrefill
prompt tok/s by context length8K context21.6K379 ms TTFT
16K context22.3K728 ms TTFT
32K context22.4K1,439 ms TTFT
64K context22.3K2,881 ms TTFT
128K context21.9K5,865 ms TTFT
256K context20.8K12,382 ms TTFT
507K context18.8K27,027 ms TTFT
512K context18.7K27,528 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 115.176 ms TTFT | 203.399 ms TTFT | 354.1203 ms TTFT | 595.2218 ms TTFT | 927.0242 ms TTFT | 1,391.7520 ms TTFT | 2,031.21,037 ms TTFT | 3,170.71,273 ms TTFT |
| 16K | 113.477 ms TTFT | 203.1102 ms TTFT | 352.6203 ms TTFT | 594.4269 ms TTFT | 935.2376 ms TTFT | 1,370.1554 ms TTFT | 2,000.2895 ms TTFT | 3,162.41,486 ms TTFT |
| 32K | 112.499 ms TTFT | 201.3130 ms TTFT | 349.9210 ms TTFT | 589.2352 ms TTFT | 921.4494 ms TTFT | 1,355.0777 ms TTFT | 1,984.61,434 ms TTFT | Over capacity |
| 64K | 112.3154 ms TTFT | 200.8196 ms TTFT | 347.7259 ms TTFT | 586.0413 ms TTFT | 902.4699 ms TTFT | 1,337.01,260 ms TTFT | Over capacity | Over capacity |
| 128K | 111.5271 ms TTFT | 200.3356 ms TTFT | 347.9508 ms TTFT | 579.5832 ms TTFT | 909.11,445 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 111.9514 ms TTFT | 198.0730 ms TTFT | 342.31,106 ms TTFT | 575.01,778 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 109.71,157 ms TTFT | 194.01,545 ms TTFT | 337.32,360 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.6K | — | 379 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.3K | — | 728 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.4K | — | 1,439 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.3K | — | 2,881 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.9K | — | 5,865 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.8K | — | 12,382 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 18.8K | — | 27,027 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 18.7K | — | 27,528 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 115.1 | 115.1 | 76 ms | 8.7 ms | 1.0 | 0 | 3,453 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 203.3 | 101.6 | 99 ms | 9.8 ms | 2.0 | 0 | 6,100 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 354.1 | 88.6 | 203 ms | 11.3 ms | 4.0 | 0 | 10,620 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 595.2 | 74.8 | 218 ms | 13.4 ms | 8.0 | 0 | 17,856 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 927.0 | 58.1 | 242 ms | 17.2 ms | 16.0 | 0 | 27,808 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,391.7 | 43.8 | 520 ms | 22.8 ms | 32.0 | 0 | 41,730 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,031.2 | 32.2 | 1,037 ms | 31.1 ms | 64.0 | 0 | 60,926 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,170.7 | 24.8 | 1,273 ms | 40.4 ms | 128.0 | 0 | 95,106 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.4 | 113.4 | 77 ms | 8.8 ms | 1.0 | 0 | 3,403 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 203.1 | 101.5 | 102 ms | 9.8 ms | 2.0 | 0 | 6,094 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 352.6 | 88.3 | 203 ms | 11.3 ms | 4.0 | 0 | 10,576 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 594.4 | 74.4 | 269 ms | 13.4 ms | 8.0 | 0 | 17,832 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 935.2 | 58.8 | 376 ms | 17.0 ms | 16.0 | 0 | 28,047 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,370.1 | 43.0 | 554 ms | 23.2 ms | 32.0 | 0 | 41,087 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,000.2 | 31.4 | 895 ms | 31.9 ms | 64.0 | 0 | 59,970 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,162.4 | 24.5 | 1,486 ms | 40.7 ms | 128.0 | 0 | 94,850 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 112.4 | 112.5 | 99 ms | 8.9 ms | 1.0 | 0 | 3,372 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 201.3 | 100.8 | 130 ms | 9.9 ms | 2.0 | 0 | 6,038 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 349.9 | 87.5 | 210 ms | 11.4 ms | 4.0 | 0 | 10,496 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 589.2 | 73.8 | 352 ms | 13.5 ms | 8.0 | 0 | 17,671 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 921.4 | 57.8 | 494 ms | 17.3 ms | 16.0 | 0 | 27,641 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,355.0 | 42.2 | 777 ms | 23.7 ms | 32.0 | 0 | 40,638 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,984.6 | 30.8 | 1,434 ms | 32.5 ms | 64.0 | 0 | 59,517 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 112.3 | 112.4 | 154 ms | 8.9 ms | 1.0 | 0 | 3,369 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 200.8 | 100.5 | 196 ms | 10.0 ms | 2.0 | 0 | 6,022 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 347.7 | 86.9 | 259 ms | 11.5 ms | 4.0 | 0 | 10,432 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 586.0 | 73.2 | 413 ms | 13.7 ms | 8.0 | 0 | 17,577 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 902.4 | 56.3 | 699 ms | 17.7 ms | 16.0 | 0 | 27,072 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,337.0 | 41.6 | 1,260 ms | 24.0 ms | 32.0 | 0 | 40,096 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 111.5 | 111.6 | 271 ms | 9.0 ms | 1.0 | 0 | 3,345 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 200.3 | 100.3 | 356 ms | 10.0 ms | 2.0 | 0 | 6,008 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 347.9 | 87.0 | 508 ms | 11.5 ms | 4.0 | 0 | 10,436 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 579.5 | 72.4 | 832 ms | 13.8 ms | 8.0 | 0 | 17,384 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 909.1 | 56.8 | 1,445 ms | 17.6 ms | 16.0 | 0 | 27,263 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 111.9 | 112.1 | 514 ms | 8.9 ms | 1.0 | 0 | 3,356 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 198.0 | 99.2 | 730 ms | 10.1 ms | 2.0 | 0 | 5,940 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 342.3 | 85.7 | 1,106 ms | 11.7 ms | 4.0 | 0 | 10,269 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 575.0 | 71.9 | 1,778 ms | 13.9 ms | 8.0 | 0 | 17,248 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 109.7 | 110.2 | 1,157 ms | 9.1 ms | 1.0 | 0 | 3,292 | 30.0 s |
| Sustained decode | 507K | 2 | Valid | 194.0 | 97.3 | 1,545 ms | 10.3 ms | 2.0 | 0 | 5,820 | 30.0 s |
| Sustained decode | 507K | 4 | Valid | 337.3 | 84.5 | 2,360 ms | 11.8 ms | 4.0 | 0 | 10,117 | 30.0 s |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.