Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.33.0rc1.dev141 · MXFP4+FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.33.0rc1.dev141
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.0
- Commit
48dcffbb7c8f
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
# Reconstructed from current recipes/nvidia-rtxpro6k/deepseek-ai/vllm_deepseek-ai_DeepSeek-V4.1-Flash-1017_ngram_ram_rtxpro6kx4.sh; not captured runtime argv.
env VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_B12X_MOE_FP4_FORCE_A16=1 "$HOME/env_rtxpro6k-deepseek-v41-1017-vllm-pr-60762/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash-1017 --served-model-name deepseek --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 524288 --gpu-memory-utilization 0.950000 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --enable-prefix-cachingPrefill
prompt tok/s by context length8K context18.7K438 ms TTFT
16K context19.2K847 ms TTFT
32K context19.4K1,665 ms TTFT
64K context19.3K3,331 ms TTFT
128K context19K6,758 ms TTFT
256K context18.2K14,141 ms TTFT
507K context16.6K30,561 ms TTFT
512K context16.6K31,038 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 114.783 ms TTFT | 198.4116 ms TTFT | 337.9258 ms TTFT | 548.3276 ms TTFT | 852.8287 ms TTFT | 1,277.1714 ms TTFT | 1,816.1911 ms TTFT | 2,801.21,620 ms TTFT |
| 16K | 113.786 ms TTFT | 197.4124 ms TTFT | 334.6186 ms TTFT | 548.8212 ms TTFT | 857.8395 ms TTFT | 1,267.3872 ms TTFT | 1,821.41,374 ms TTFT | 2,790.32,476 ms TTFT |
| 32K | 112.9105 ms TTFT | 196.7177 ms TTFT | 331.1221 ms TTFT | 542.1347 ms TTFT | 839.7567 ms TTFT | 1,245.11,017 ms TTFT | 1,775.61,842 ms TTFT | Over capacity |
| 64K | 113.3163 ms TTFT | 195.8270 ms TTFT | 328.7382 ms TTFT | 540.2602 ms TTFT | 847.11,053 ms TTFT | 1,242.11,950 ms TTFT | Over capacity | Over capacity |
| 128K | 112.7278 ms TTFT | 195.4511 ms TTFT | 326.1537 ms TTFT | 542.1834 ms TTFT | 831.51,462 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 111.7530 ms TTFT | 193.4750 ms TTFT | 325.11,124 ms TTFT | 532.71,799 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 109.41,134 ms TTFT | 189.81,503 ms TTFT | 318.42,343 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.7K | — | 438 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 19.2K | — | 847 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.4K | — | 1,665 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.3K | — | 3,331 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19K | — | 6,758 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18.2K | — | 14,141 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 16.6K | — | 30,561 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 16.6K | — | 31,038 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 114.7 | 114.7 | 83 ms | 8.7 ms | 1.0 | 0 | 3,442 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 198.4 | 99.3 | 116 ms | 10.1 ms | 2.0 | 0 | 5,952 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 337.9 | 84.5 | 258 ms | 11.8 ms | 4.0 | 0 | 10,135 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 548.3 | 68.9 | 276 ms | 14.5 ms | 8.0 | 0 | 16,448 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 852.8 | 53.4 | 287 ms | 18.7 ms | 16.0 | 0 | 25,584 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,277.1 | 40.4 | 714 ms | 24.7 ms | 32.0 | 0 | 38,302 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,816.1 | 28.5 | 911 ms | 35.1 ms | 64.0 | 0 | 54,465 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,801.2 | 21.9 | 1,620 ms | 45.7 ms | 128.0 | 0 | 83,964 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.7 | 113.7 | 86 ms | 8.8 ms | 1.0 | 0 | 3,411 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 197.4 | 98.8 | 124 ms | 10.1 ms | 2.0 | 0 | 5,922 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 334.6 | 83.8 | 186 ms | 11.9 ms | 4.0 | 0 | 10,036 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 548.8 | 68.8 | 212 ms | 14.5 ms | 8.0 | 0 | 16,456 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 857.8 | 53.8 | 395 ms | 18.6 ms | 16.0 | 0 | 25,730 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,267.3 | 39.7 | 872 ms | 25.2 ms | 32.0 | 0 | 38,017 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,821.4 | 28.5 | 1,374 ms | 35.1 ms | 64.0 | 0 | 54,588 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,790.3 | 21.6 | 2,476 ms | 46.3 ms | 128.0 | 0 | 83,711 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 112.9 | 113.0 | 105 ms | 8.9 ms | 1.0 | 0 | 3,387 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 196.7 | 98.5 | 177 ms | 10.2 ms | 2.0 | 0 | 5,901 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 331.1 | 82.9 | 221 ms | 12.1 ms | 4.0 | 0 | 9,932 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 542.1 | 67.8 | 347 ms | 14.7 ms | 8.0 | 0 | 16,256 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 839.7 | 52.4 | 567 ms | 19.1 ms | 16.0 | 0 | 25,185 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,245.1 | 38.8 | 1,017 ms | 25.8 ms | 32.0 | 0 | 37,345 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,775.6 | 27.5 | 1,842 ms | 36.4 ms | 64.0 | 0 | 53,249 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.3 | 113.3 | 163 ms | 8.8 ms | 1.0 | 0 | 3,398 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 195.8 | 98.0 | 270 ms | 10.2 ms | 2.0 | 0 | 5,874 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 328.7 | 82.3 | 382 ms | 12.2 ms | 4.0 | 0 | 9,860 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 540.2 | 67.5 | 602 ms | 14.8 ms | 8.0 | 0 | 16,200 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 847.1 | 52.7 | 1,053 ms | 19.0 ms | 16.0 | 0 | 25,407 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,242.1 | 38.6 | 1,950 ms | 25.9 ms | 32.0 | 0 | 37,249 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.7 | 112.8 | 278 ms | 8.9 ms | 1.0 | 0 | 3,380 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 195.4 | 97.8 | 511 ms | 10.2 ms | 2.0 | 0 | 5,863 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 326.1 | 81.5 | 537 ms | 12.3 ms | 4.0 | 0 | 9,780 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 542.1 | 67.6 | 834 ms | 14.8 ms | 8.0 | 0 | 16,264 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 831.5 | 51.9 | 1,462 ms | 19.3 ms | 16.0 | 0 | 24,944 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 111.7 | 111.8 | 530 ms | 8.9 ms | 1.0 | 0 | 3,349 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 193.4 | 96.8 | 750 ms | 10.3 ms | 2.0 | 0 | 5,800 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 325.1 | 81.4 | 1,124 ms | 12.3 ms | 4.0 | 0 | 9,752 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 532.7 | 66.6 | 1,799 ms | 15.0 ms | 8.0 | 0 | 15,977 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 109.4 | 109.8 | 1,134 ms | 9.1 ms | 1.0 | 0 | 3,283 | 30.0 s |
| Sustained decode | 507K | 2 | Valid | 189.8 | 95.2 | 1,503 ms | 10.5 ms | 2.0 | 0 | 5,694 | 30.0 s |
| Sustained decode | 507K | 4 | Valid | 318.4 | 79.9 | 2,343 ms | 12.5 ms | 4.0 | 0 | 9,548 | 30.0 s |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.