Speed test matrix
DeepSeek-V4.1-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev147+g02ee13be2 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev147+g02ee13be2
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
02ee13be2107
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-deepseek-v41-vllm-pr-56509/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.85 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12x --speculative-config '{"method":"dspark","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context10.6K776 ms TTFT
16K context10.7K1,512 ms TTFT
32K context10.7K3,018 ms TTFT
64K context10.5K6,138 ms TTFT
128K context9,99812,855 ms TTFT
256K context9,06228,349 ms TTFT
512K context7,66167,047 ms TTFT
1019K context6,053168,841 ms TTFT
1024K context6,043169,958 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 181.181 ms TTFT | 266.799 ms TTFT | 410.2326 ms TTFT | 565.8328 ms TTFT | 900.0341 ms TTFT | 1,295.91,097 ms TTFT | 1,779.51,206 ms TTFT | 1,578.22,616 ms TTFT |
| 16K | 173.492 ms TTFT | 276.9106 ms TTFT | 401.5334 ms TTFT | 574.2338 ms TTFT | 907.9676 ms TTFT | 1,305.2723 ms TTFT | 1,749.01,244 ms TTFT | 1,659.22,437 ms TTFT |
| 32K | 169.5127 ms TTFT | 272.4139 ms TTFT | 394.0242 ms TTFT | 542.9473 ms TTFT | 877.9745 ms TTFT | 1,245.81,062 ms TTFT | 1,640.61,580 ms TTFT | Over capacity |
| 64K | 173.9185 ms TTFT | 271.0208 ms TTFT | 413.4294 ms TTFT | 540.9503 ms TTFT | 858.2872 ms TTFT | 1,201.81,686 ms TTFT | Over capacity | Over capacity |
| 128K | 165.7320 ms TTFT | 277.1330 ms TTFT | 422.3557 ms TTFT | 540.1912 ms TTFT | 818.21,652 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 175.8641 ms TTFT | 258.2634 ms TTFT | 386.81,247 ms TTFT | 526.02,080 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 159.91,332 ms TTFT | 244.41,292 ms TTFT | 386.92,808 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 140.02,872 ms TTFT | 238.33,776 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.6K | — | 776 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.7K | — | 1,512 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.7K | — | 3,018 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.5K | — | 6,138 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,998.0 | — | 12,855 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,062.0 | — | 28,349 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 7,661.0 | — | 67,047 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,053.0 | — | 168,841 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,043.0 | — | 169,958 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 181.1 | 179.3 | 81 ms | 5.6 ms | 1.0 | 1 | 5,433 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 266.7 | 136.1 | 99 ms | 7.3 ms | 2.0 | 2 | 7,999 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 410.2 | 101.7 | 326 ms | 9.8 ms | 4.0 | 0 | 12,304 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 565.8 | 71.5 | 328 ms | 14.0 ms | 8.0 | 0 | 16,972 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 900.0 | 57.3 | 341 ms | 17.5 ms | 16.0 | 0 | 26,987 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,295.9 | 41.1 | 1,097 ms | 24.3 ms | 32.0 | 0 | 38,811 | 29.9 s |
| Sustained decode | 8K | 64 | Valid | 1,779.5 | 28.3 | 1,206 ms | 35.4 ms | 64.0 | 0 | 53,343 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,578.2 | 11.5 | 2,616 ms | 87.0 ms | 128.0 | 0 | 47,333 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 173.4 | 172.4 | 92 ms | 5.8 ms | 1.0 | 1 | 5,200 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 276.9 | 136.2 | 106 ms | 7.3 ms | 2.0 | 2 | 8,305 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 401.5 | 99.7 | 334 ms | 10.0 ms | 4.0 | 0 | 12,037 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 574.2 | 71.4 | 338 ms | 14.0 ms | 8.0 | 0 | 17,215 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 907.9 | 57.3 | 676 ms | 17.4 ms | 16.0 | 0 | 27,222 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,305.2 | 41.7 | 723 ms | 24.0 ms | 32.0 | 0 | 39,136 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,749.0 | 27.9 | 1,244 ms | 35.8 ms | 64.0 | 0 | 52,439 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,659.2 | 13.3 | 2,437 ms | 75.0 ms | 128.0 | 0 | 49,761 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 169.5 | 166.7 | 127 ms | 6.0 ms | 1.0 | 1 | 5,085 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 272.4 | 133.1 | 139 ms | 7.5 ms | 2.0 | 2 | 8,169 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 394.0 | 98.1 | 242 ms | 10.2 ms | 4.0 | 0 | 11,812 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 542.9 | 68.3 | 473 ms | 14.6 ms | 8.0 | 0 | 16,271 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 877.9 | 55.1 | 745 ms | 18.2 ms | 16.0 | 0 | 26,331 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,245.8 | 39.1 | 1,062 ms | 25.6 ms | 32.0 | 0 | 37,325 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,640.6 | 26.2 | 1,580 ms | 38.2 ms | 64.0 | 0 | 49,142 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 173.9 | 172.9 | 185 ms | 5.8 ms | 1.0 | 1 | 5,217 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 271.0 | 136.7 | 208 ms | 7.3 ms | 2.0 | 2 | 8,128 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 413.4 | 99.8 | 294 ms | 10.0 ms | 4.0 | 0 | 12,402 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 540.9 | 67.8 | 503 ms | 14.7 ms | 8.0 | 0 | 16,217 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 858.2 | 55.1 | 872 ms | 18.2 ms | 16.0 | 0 | 25,738 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,201.8 | 37.9 | 1,686 ms | 26.4 ms | 32.0 | 0 | 36,048 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 165.7 | 163.3 | 320 ms | 6.1 ms | 1.0 | 1 | 4,970 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 277.1 | 132.7 | 330 ms | 7.5 ms | 2.0 | 2 | 8,312 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 422.3 | 102.3 | 557 ms | 9.8 ms | 4.0 | 0 | 12,665 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 540.1 | 67.5 | 912 ms | 14.8 ms | 8.0 | 0 | 16,199 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 818.2 | 51.9 | 1,652 ms | 19.3 ms | 16.0 | 0 | 24,521 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 175.8 | 175.8 | 641 ms | 5.7 ms | 1.0 | 1 | 5,275 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 258.2 | 120.5 | 634 ms | 8.3 ms | 2.0 | 1 | 7,744 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 386.8 | 96.7 | 1,247 ms | 10.3 ms | 4.0 | 0 | 11,602 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 526.0 | 66.1 | 2,080 ms | 15.1 ms | 8.0 | 0 | 15,766 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 159.9 | 163.2 | 1,332 ms | 6.2 ms | 1.0 | 1 | 4,798 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 244.4 | 129.1 | 1,292 ms | 7.7 ms | 1.9 | 2 | 7,330 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 386.9 | 97.9 | 2,808 ms | 10.2 ms | 4.0 | 0 | 11,604 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 140.0 | 153.4 | 2,872 ms | 6.5 ms | 0.9 | 1 | 4,199 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 238.3 | 124.0 | 3,776 ms | 8.1 ms | 1.9 | 1 | 7,124 | 29.9 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.