Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.33.0rc1.dev141 · MXFP4+FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.33.0rc1.dev141
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.0
- Commit
48dcffbb7c8f
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
# Reconstructed from current recipes/nvidia-rtxpro6k/deepseek-ai/vllm_deepseek-ai_DeepSeek-V4.1-Flash-1017_speculative_dspark_7_ngram_ram_rtxpro6kx4.sh; not captured runtime argv.
env VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_B12X_MOE_FP4_FORCE_A16=1 "$HOME/env_rtxpro6k-deepseek-v41-1017-vllm-pr-60762/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash-1017 --served-model-name deepseek --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 524288 --gpu-memory-utilization 0.950000 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --kv-cache-dtype fp8 --engram-config '{"cpu_offload":true}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --speculative-config '{"method":"dspark","num_speculative_tokens":7}' --enable-prefix-cachingPrefill
prompt tok/s by context length8K context18.5K442 ms TTFT
16K context19.1K850 ms TTFT
32K context19.3K1,674 ms TTFT
64K context19.2K3,352 ms TTFT
128K context18.9K6,804 ms TTFT
256K context18K14,268 ms TTFT
507K context16.5K30,744 ms TTFT
512K context16.4K31,277 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 153.782 ms TTFT | 245.997 ms TTFT | 364.0245 ms TTFT | 488.1301 ms TTFT | 769.0298 ms TTFT | 1,037.5792 ms TTFT | 1,198.5827 ms TTFT | Over capacity |
| 16K | 151.485 ms TTFT | 250.8105 ms TTFT | 357.8212 ms TTFT | 505.4241 ms TTFT | 753.9456 ms TTFT | 1,020.5765 ms TTFT | 1,163.7973 ms TTFT | Over capacity |
| 32K | 152.5108 ms TTFT | 242.4125 ms TTFT | 347.4207 ms TTFT | 498.4317 ms TTFT | 732.9502 ms TTFT | 997.8834 ms TTFT | Over capacity | Over capacity |
| 64K | 167.0166 ms TTFT | 241.7185 ms TTFT | 361.5281 ms TTFT | 491.6469 ms TTFT | 721.9799 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 150.6277 ms TTFT | 245.7297 ms TTFT | 349.0548 ms TTFT | 478.2889 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 149.6559 ms TTFT | 239.7750 ms TTFT | 341.11,124 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 197.41,144 ms TTFT | 232.91,556 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.5K | — | 442 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 19.1K | — | 850 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.3K | — | 1,674 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.2K | — | 3,352 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.9K | — | 6,804 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18K | — | 14,268 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 16.5K | — | 30,744 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 16.4K | — | 31,277 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 153.7 | 159.1 | 82 ms | 6.3 ms | 1.0 | 1 | 4,612 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 245.9 | 123.8 | 97 ms | 8.1 ms | 2.0 | 2 | 7,376 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 364.0 | 91.4 | 245 ms | 10.9 ms | 4.0 | 0 | 10,911 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 488.1 | 62.2 | 301 ms | 16.1 ms | 8.0 | 0 | 14,641 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 769.0 | 48.5 | 298 ms | 20.6 ms | 16.0 | 0 | 23,038 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,037.5 | 32.7 | 792 ms | 30.6 ms | 32.0 | 0 | 31,107 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,198.5 | 18.9 | 827 ms | 53.0 ms | 64.0 | 0 | 35,858 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 151.4 | 148.3 | 85 ms | 6.8 ms | 1.0 | 1 | 4,541 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 250.8 | 120.3 | 105 ms | 8.3 ms | 2.0 | 2 | 7,524 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 357.8 | 89.8 | 212 ms | 11.1 ms | 4.0 | 0 | 10,732 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 505.4 | 62.8 | 241 ms | 15.9 ms | 8.0 | 0 | 15,151 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 753.9 | 47.8 | 456 ms | 20.9 ms | 16.0 | 0 | 22,589 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,020.5 | 32.4 | 765 ms | 30.9 ms | 32.0 | 0 | 30,601 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,163.7 | 18.5 | 973 ms | 54.0 ms | 64.0 | 0 | 34,799 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 152.5 | 157.3 | 108 ms | 6.4 ms | 1.0 | 1 | 4,576 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 242.4 | 118.1 | 125 ms | 8.5 ms | 2.0 | 1 | 7,269 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 347.4 | 85.8 | 207 ms | 11.7 ms | 4.0 | 0 | 10,415 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 498.4 | 62.6 | 317 ms | 16.0 ms | 8.0 | 0 | 14,936 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 732.9 | 46.6 | 502 ms | 21.5 ms | 16.0 | 0 | 21,956 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 997.8 | 31.5 | 834 ms | 31.7 ms | 32.0 | 0 | 29,904 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 167.0 | 161.7 | 166 ms | 6.2 ms | 1.0 | 1 | 5,007 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 241.7 | 126.3 | 185 ms | 7.9 ms | 2.0 | 1 | 7,250 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 361.5 | 90.4 | 281 ms | 11.1 ms | 4.0 | 0 | 10,841 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 491.6 | 61.4 | 469 ms | 16.3 ms | 8.0 | 0 | 14,747 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 721.9 | 45.6 | 799 ms | 21.9 ms | 16.0 | 0 | 21,651 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 150.6 | 153.7 | 277 ms | 6.5 ms | 1.0 | 1 | 4,517 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 245.7 | 126.8 | 297 ms | 7.9 ms | 2.0 | 2 | 7,369 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 349.0 | 86.0 | 548 ms | 11.6 ms | 4.0 | 0 | 10,464 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 478.2 | 59.7 | 889 ms | 16.7 ms | 8.0 | 0 | 14,334 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 149.6 | 152.0 | 559 ms | 6.6 ms | 1.0 | 1 | 4,488 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 239.7 | 119.3 | 750 ms | 8.4 ms | 2.0 | 0 | 7,186 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 341.1 | 86.2 | 1,124 ms | 11.6 ms | 4.0 | 0 | 10,228 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 197.4 | 194.9 | 1,144 ms | 5.4 ms | 1.0 | 1 | 5,921 | 30.0 s |
| Sustained decode | 507K | 2 | Valid | 232.9 | 117.3 | 1,556 ms | 8.5 ms | 2.0 | 1 | 6,985 | 30.0 s |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.