Speed test matrix
DeepSeek-V4.1-Flash-1017-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.33.0rc1.dev141 · NVFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.33.0rc1.dev141
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.0
- Commit
48dcffbb7c8f
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
# Reconstructed from current recipes/nvidia-rtxpro6k/nvidia/vllm_nvidia_DeepSeek-V4.1-Flash-1017-NVFP4_speculative_dspark_7_ngram_ram_rtxpro6kx4.sh; not captured runtime argv.
env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-deepseek-v41-1017-vllm-pr-60762/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-1017-NVFP4 --served-model-name deepseek --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --max-model-len 524288 --gpu-memory-utilization 0.950000 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"fp8","indexer_sparse_logits":false}' --kv-cache-dtype fp8 --max-num-seqs 128 --engram-config '{"cpu_offload":true}' --speculative-config '{"method":"dspark","num_speculative_tokens":7}' --enable-prefix-cachingPrefill
prompt tok/s by context length8K context21.5K380 ms TTFT
16K context22.1K735 ms TTFT
32K context22.3K1,448 ms TTFT
64K context22.1K2,909 ms TTFT
128K context21.7K5,919 ms TTFT
256K context20.6K12,493 ms TTFT
507K context18.7K27,249 ms TTFT
512K context18.3K28,053 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 174.674 ms TTFT | 268.788 ms TTFT | 378.5191 ms TTFT | 533.4232 ms TTFT | 859.7254 ms TTFT | 1,187.1555 ms TTFT | 1,605.61,122 ms TTFT | Over capacity |
| 16K | 165.282 ms TTFT | 259.7101 ms TTFT | 396.8176 ms TTFT | 549.0202 ms TTFT | 849.0330 ms TTFT | 1,173.4559 ms TTFT | Over capacity | Over capacity |
| 32K | 162.1110 ms TTFT | 265.3115 ms TTFT | 380.0209 ms TTFT | 530.3301 ms TTFT | 820.6470 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 159.6160 ms TTFT | 259.8171 ms TTFT | 365.9304 ms TTFT | 536.4463 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 161.8275 ms TTFT | 276.9289 ms TTFT | 364.8555 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 168.4554 ms TTFT | 254.8547 ms TTFT | 367.61,131 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 165.51,134 ms TTFT | 244.21,590 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.5K | — | 380 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.1K | — | 735 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.3K | — | 1,448 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.1K | — | 2,909 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.7K | — | 5,919 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.6K | — | 12,493 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 18.7K | — | 27,249 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 18.3K | — | 28,053 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 174.6 | 170.1 | 74 ms | 5.9 ms | 1.0 | 1 | 5,235 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 268.7 | 128.4 | 88 ms | 7.8 ms | 2.0 | 2 | 8,059 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 378.5 | 94.8 | 191 ms | 10.6 ms | 4.0 | 0 | 11,348 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 533.4 | 67.3 | 232 ms | 14.9 ms | 8.0 | 0 | 15,995 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 859.7 | 54.3 | 254 ms | 18.4 ms | 16.0 | 0 | 25,790 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,187.1 | 37.4 | 555 ms | 26.7 ms | 32.0 | 0 | 35,588 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,605.6 | 25.5 | 1,122 ms | 39.3 ms | 64.0 | 0 | 48,114 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 165.2 | 167.1 | 82 ms | 6.0 ms | 1.0 | 1 | 4,955 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 259.7 | 130.9 | 101 ms | 7.6 ms | 2.0 | 2 | 7,789 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 396.8 | 100.7 | 176 ms | 9.9 ms | 4.0 | 0 | 11,902 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 549.0 | 68.7 | 202 ms | 14.6 ms | 8.0 | 0 | 16,452 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 849.0 | 54.3 | 330 ms | 18.4 ms | 16.0 | 0 | 25,445 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,173.4 | 37.2 | 559 ms | 26.9 ms | 32.0 | 0 | 35,187 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 162.1 | 168.1 | 110 ms | 6.0 ms | 1.0 | 1 | 4,861 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 265.3 | 128.1 | 115 ms | 7.8 ms | 2.0 | 2 | 7,957 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 380.0 | 96.3 | 209 ms | 10.4 ms | 4.0 | 0 | 11,397 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 530.3 | 68.0 | 301 ms | 14.7 ms | 8.0 | 0 | 15,910 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 820.6 | 52.3 | 470 ms | 19.1 ms | 16.0 | 0 | 24,599 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 159.6 | 162.9 | 160 ms | 6.1 ms | 1.0 | 1 | 4,788 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 259.8 | 129.1 | 171 ms | 7.7 ms | 2.0 | 2 | 7,792 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 365.9 | 94.3 | 304 ms | 10.6 ms | 4.0 | 0 | 10,973 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 536.4 | 66.7 | 463 ms | 15.0 ms | 8.0 | 0 | 16,091 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 161.8 | 163.7 | 275 ms | 6.1 ms | 1.0 | 1 | 4,853 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 276.9 | 135.0 | 289 ms | 7.4 ms | 2.0 | 2 | 8,305 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 364.8 | 91.9 | 555 ms | 10.9 ms | 4.0 | 0 | 10,937 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 168.4 | 168.0 | 554 ms | 6.0 ms | 1.0 | 1 | 5,053 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 254.8 | 128.7 | 547 ms | 7.8 ms | 2.0 | 2 | 7,645 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 367.6 | 91.5 | 1,131 ms | 10.9 ms | 4.0 | 0 | 11,021 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 165.5 | 167.9 | 1,134 ms | 6.0 ms | 1.0 | 1 | 4,962 | 30.0 s |
| Sustained decode | 507K | 2 | Valid | 244.2 | 122.8 | 1,590 ms | 8.1 ms | 2.0 | 2 | 7,245 | 29.7 s |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.