Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.33.0rc1.dev141 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.33.0rc1.dev141
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.0
- Commit
48dcffbb7c8f
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
# Reconstructed from current recipes/nvidia-rtxpro6k/deepseek-ai/vllm_deepseek-ai_DeepSeek-V4.1-Flash-1017_rtxpro6kx8.sh; not captured runtime argv.
env VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_B12X_MOE_FP4_FORCE_A16=1 "$HOME/env_rtxpro6k-deepseek-v41-1017-vllm-pr-60762/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash-1017 --served-model-name deepseek --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":false}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --no-enable-prefix-cachingPrefill
prompt tok/s by context length8K context10.7K769 ms TTFT
16K context11K1,472 ms TTFT
32K context11.1K2,918 ms TTFT
64K context11K5,824 ms TTFT
128K context10.9K11,753 ms TTFT
256K context10.6K24,162 ms TTFT
512K context10.1K50,733 ms TTFT
1019K context9,249110,502 ms TTFT
1024K context9,209111,533 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 78.1752 ms TTFT | 152.61,124 ms TTFT | 274.01,876 ms TTFT | 482.13,353 ms TTFT | 825.97,047 ms TTFT | 1,239.012,961 ms TTFT | 1,772.324,947 ms TTFT | 132.3Requests queued44,866 ms TTFT |
| 16K | 78.01,466 ms TTFT | 152.22,216 ms TTFT | 271.73,701 ms TTFT | 484.26,684 ms TTFT | 820.612,635 ms TTFT | 1,230.324,551 ms TTFT | 67.1Requests queued44,661 ms TTFT | 66.7Requests queued44,727 ms TTFT |
| 32K | 77.72,913 ms TTFT | 151.14,426 ms TTFT | 268.27,431 ms TTFT | 474.113,354 ms TTFT | 809.925,236 ms TTFT | 33.0Requests queued44,587 ms TTFT | 33.2Requests queued44,588 ms TTFT | 33.0Requests queued44,669 ms TTFT |
| 64K | 77.55,833 ms TTFT | 150.58,866 ms TTFT | 265.114,866 ms TTFT | 467.726,775 ms TTFT | 792.849,836 ms TTFT | 29.2Requests queued73,722 ms TTFT | 29.1Requests queued73,761 ms TTFT | Over capacity |
| 128K | 76.911,742 ms TTFT | 149.217,845 ms TTFT | 262.430,006 ms TTFT | 458.853,302 ms TTFT | 255.3Requests queued100,686 ms TTFT | 20.3Requests queued100,775 ms TTFT | Over capacity | Over capacity |
| 256K | 76.624,099 ms TTFT | 146.236,556 ms TTFT | 255.960,801 ms TTFT | 8.6Requests queued97,010 ms TTFT | 8.6Requests queued97,072 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 75.550,770 ms TTFT | 142.276,434 ms TTFT | 2.7Requests queued102,199 ms TTFT | 2.8Requests queued102,062 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 1.4Requests queued | 0.0Requests queued | 0.0Requests queued | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.7K | — | 769 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11K | — | 1,472 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.1K | — | 2,918 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11K | — | 5,824 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.9K | — | 11,753 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.6K | — | 24,162 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.1K | — | 50,733 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,249.0 | — | 110,502 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,209.0 | — | 111,533 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 78.1 | 78.1 | 752 ms | 12.8 ms | 1.0 | 0 | 2,343 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 152.6 | 75.5 | 1,124 ms | 13.2 ms | 2.0 | 0 | 4,578 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 274.0 | 66.4 | 1,876 ms | 15.1 ms | 4.0 | 0 | 8,216 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 482.1 | 56.2 | 3,353 ms | 17.8 ms | 8.0 | 0 | 14,457 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 825.9 | 45.5 | 7,047 ms | 22.0 ms | 16.0 | 0 | 24,769 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,239.0 | 29.8 | 12,961 ms | 33.6 ms | 32.0 | 0 | 37,151 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,772.3 | 17.1 | 24,947 ms | 58.4 ms | 64.0 | 0 | 53,120 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 132.3 | 1.3 | 44,866 ms | 751.6 ms | 98.8 | 0 | 3,899 | 29.5 s |
| Sustained decode | 16K | 1 | Valid | 78.0 | 78.0 | 1,466 ms | 12.8 ms | 1.0 | 0 | 2,340 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 152.2 | 74.6 | 2,216 ms | 13.4 ms | 2.0 | 0 | 4,566 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 271.7 | 64.0 | 3,701 ms | 15.6 ms | 4.0 | 0 | 8,152 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 484.2 | 52.9 | 6,684 ms | 18.9 ms | 8.0 | 0 | 14,522 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 820.6 | 39.0 | 12,635 ms | 25.6 ms | 16.0 | 0 | 24,608 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,230.3 | 23.7 | 24,551 ms | 42.2 ms | 32.0 | 0 | 36,895 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 67.1 | 1.3 | 44,661 ms | 744.8 ms | 50.6 | 0 | 1,981 | 29.5 s |
| Sustained decode | 16K | 128 | Requests queued | 66.7 | 1.3 | 44,727 ms | 744.9 ms | 50.4 | 0 | 1,979 | 29.7 s |
| Sustained decode | 32K | 1 | Valid | 77.7 | 77.8 | 2,913 ms | 12.9 ms | 1.0 | 0 | 2,332 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 151.1 | 72.7 | 4,426 ms | 13.8 ms | 2.0 | 0 | 4,532 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 268.2 | 59.9 | 7,431 ms | 16.7 ms | 4.0 | 0 | 8,044 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 474.1 | 46.1 | 13,354 ms | 21.7 ms | 8.0 | 0 | 14,217 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 809.9 | 31.3 | 25,236 ms | 32.0 ms | 16.0 | 0 | 24,288 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 33.0 | 1.3 | 44,587 ms | 742.3 ms | 25.2 | 0 | 987 | 29.9 s |
| Sustained decode | 32K | 64 | Requests queued | 33.2 | 1.3 | 44,588 ms | 742.3 ms | 25.2 | 0 | 987 | 29.8 s |
| Sustained decode | 32K | 128 | Requests queued | 33.0 | 1.3 | 44,669 ms | 742.3 ms | 25.2 | 0 | 988 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 77.5 | 77.5 | 5,833 ms | 12.9 ms | 1.0 | 0 | 2,323 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 150.5 | 69.9 | 8,866 ms | 14.4 ms | 2.0 | 0 | 4,513 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 265.1 | 53.4 | 14,866 ms | 18.8 ms | 4.0 | 0 | 7,952 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 467.7 | 37.2 | 26,775 ms | 26.9 ms | 8.0 | 0 | 14,024 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 792.8 | 22.5 | 49,836 ms | 44.6 ms | 16.0 | 0 | 23,777 | 30.0 s |
| Sustained decode | 64K | 32 | Requests queued | 29.2 | 1.3 | 73,722 ms | 744.6 ms | 22.6 | 0 | 867 | 29.7 s |
| Sustained decode | 64K | 64 | Requests queued | 29.1 | 1.3 | 73,761 ms | 744.7 ms | 22.6 | 0 | 867 | 29.8 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 76.9 | 77.0 | 11,742 ms | 13.0 ms | 1.0 | 0 | 2,308 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 149.2 | 65.2 | 17,845 ms | 15.7 ms | 2.0 | 0 | 4,474 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 262.4 | 44.8 | 30,006 ms | 22.6 ms | 4.0 | 0 | 7,868 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 458.8 | 26.7 | 53,302 ms | 37.7 ms | 8.0 | 0 | 13,759 | 30.0 s |
| Sustained decode | 128K | 16 | Requests queued | 255.3 | 5.9 | 100,686 ms | 170.9 ms | 15.7 | 0 | 7,658 | 30.0 s |
| Sustained decode | 128K | 32 | Requests queued | 20.3 | 1.3 | 100,775 ms | 751.1 ms | 16.0 | 0 | 600 | 29.5 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 76.6 | 76.8 | 24,099 ms | 13.0 ms | 1.0 | 0 | 2,299 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 146.2 | 58.5 | 36,556 ms | 18.3 ms | 2.0 | 0 | 4,386 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 255.9 | 32.5 | 60,801 ms | 31.7 ms | 4.0 | 0 | 7,675 | 30.0 s |
| Sustained decode | 256K | 8 | Requests queued | 8.6 | 1.3 | 97,010 ms | 767.1 ms | 7.6 | 0 | 257 | 29.9 s |
| Sustained decode | 256K | 16 | Requests queued | 8.6 | 1.3 | 97,072 ms | 767.2 ms | 7.6 | 0 | 257 | 29.9 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 75.5 | 75.8 | 50,770 ms | 13.2 ms | 1.0 | 0 | 2,266 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 142.2 | 50.4 | 76,434 ms | 24.0 ms | 2.0 | 0 | 4,264 | 30.0 s |
| Sustained decode | 512K | 4 | Requests queued | 2.7 | 1.3 | 102,199 ms | 799.3 ms | 3.2 | 0 | 80 | 29.3 s |
| Sustained decode | 512K | 8 | Requests queued | 2.8 | 1.3 | 102,062 ms | 797.7 ms | 3.3 | 0 | 82 | 29.5 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 1.4 | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 0.0 | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 0.0 | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.