Speed test matrix
DeepSeek-V4.1-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev147+g02ee13be2 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev147+g02ee13be2
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
02ee13be2107
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-deepseek-v41-vllm-pr-56509/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --attention-backend FLASHINFER_MLA_SPARSE_DSV41 --linear-backend flashinfer_cutlass --moe-backend b12xPrefill
prompt tok/s by context length8K context10.4K789 ms TTFT
16K context10.6K1,524 ms TTFT
32K context10.6K3,042 ms TTFT
64K context10.4K6,188 ms TTFT
128K context9,92212,953 ms TTFT
256K context9,00628,523 ms TTFT
512K context7,60567,534 ms TTFT
1019K context6,027169,580 ms TTFT
1024K context6,005171,022 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 116.681 ms TTFT | 217.2131 ms TTFT | 386.1302 ms TTFT | 581.1315 ms TTFT | 828.6606 ms TTFT | 1,278.31,039 ms TTFT | 1,743.41,204 ms TTFT | 3,066.12,584 ms TTFT |
| 16K | 114.893 ms TTFT | 213.4152 ms TTFT | 381.0192 ms TTFT | 569.0430 ms TTFT | 826.9728 ms TTFT | 1,270.4884 ms TTFT | 1,718.51,291 ms TTFT | 3,041.92,724 ms TTFT |
| 32K | 113.0121 ms TTFT | 213.6204 ms TTFT | 376.4259 ms TTFT | 569.3576 ms TTFT | 813.5963 ms TTFT | 1,254.51,201 ms TTFT | 1,692.71,684 ms TTFT | Over capacity |
| 64K | 112.6174 ms TTFT | 210.4223 ms TTFT | 373.7308 ms TTFT | 566.5481 ms TTFT | 810.9837 ms TTFT | 1,231.41,496 ms TTFT | Over capacity | Over capacity |
| 128K | 112.4298 ms TTFT | 209.6396 ms TTFT | 372.0585 ms TTFT | 559.5955 ms TTFT | 801.61,704 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 112.8575 ms TTFT | 208.4835 ms TTFT | 367.81,247 ms TTFT | 552.62,141 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 111.01,275 ms TTFT | 204.91,853 ms TTFT | 357.02,859 ms TTFT | 540.44,940 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 107.22,800 ms TTFT | 198.23,781 ms TTFT | 349.86,207 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 789 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.6K | — | 1,524 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.6K | — | 3,042 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.4K | — | 6,188 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,922.0 | — | 12,953 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,006.0 | — | 28,523 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 7,605.0 | — | 67,534 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,027.0 | — | 169,580 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,005.0 | — | 171,022 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 116.6 | 116.6 | 81 ms | 8.6 ms | 1.0 | 0 | 3,499 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 217.2 | 108.6 | 131 ms | 9.2 ms | 2.0 | 0 | 6,516 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 386.1 | 96.6 | 302 ms | 10.4 ms | 4.0 | 0 | 11,584 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 581.1 | 72.6 | 315 ms | 13.8 ms | 8.0 | 0 | 17,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 828.6 | 52.0 | 606 ms | 19.2 ms | 16.0 | 0 | 24,847 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,278.3 | 40.0 | 1,039 ms | 25.0 ms | 32.0 | 0 | 38,335 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,743.4 | 27.0 | 1,204 ms | 37.0 ms | 64.0 | 0 | 52,288 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,066.1 | 23.6 | 2,584 ms | 42.4 ms | 128.0 | 0 | 91,906 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 114.8 | 114.8 | 93 ms | 8.7 ms | 1.0 | 0 | 3,443 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 213.4 | 106.7 | 152 ms | 9.4 ms | 2.0 | 0 | 6,400 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 381.0 | 95.2 | 192 ms | 10.5 ms | 4.0 | 0 | 11,429 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 569.0 | 71.3 | 430 ms | 14.0 ms | 8.0 | 0 | 17,064 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 826.9 | 51.9 | 728 ms | 19.3 ms | 16.0 | 0 | 24,800 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,270.4 | 39.7 | 884 ms | 25.2 ms | 32.0 | 0 | 38,112 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,718.5 | 26.7 | 1,291 ms | 37.5 ms | 64.0 | 0 | 51,520 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,041.9 | 23.3 | 2,724 ms | 43.0 ms | 128.0 | 0 | 91,263 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 113.0 | 113.1 | 121 ms | 8.8 ms | 1.0 | 0 | 3,391 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 213.6 | 106.8 | 204 ms | 9.4 ms | 2.0 | 0 | 6,407 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 376.4 | 93.9 | 259 ms | 10.7 ms | 4.0 | 0 | 11,288 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 569.3 | 71.0 | 576 ms | 14.1 ms | 8.0 | 0 | 17,072 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 813.5 | 50.8 | 963 ms | 19.7 ms | 16.0 | 0 | 24,399 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,254.5 | 38.8 | 1,201 ms | 25.8 ms | 32.0 | 0 | 37,632 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,692.7 | 26.1 | 1,684 ms | 38.3 ms | 64.0 | 0 | 50,753 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 112.6 | 112.6 | 174 ms | 8.9 ms | 1.0 | 0 | 3,376 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 210.4 | 105.1 | 223 ms | 9.5 ms | 2.0 | 0 | 6,310 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 373.7 | 93.3 | 308 ms | 10.7 ms | 4.0 | 0 | 11,208 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 566.5 | 70.7 | 481 ms | 14.2 ms | 8.0 | 0 | 16,992 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 810.9 | 50.4 | 837 ms | 19.8 ms | 16.0 | 0 | 24,320 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,231.4 | 38.1 | 1,496 ms | 26.3 ms | 32.0 | 0 | 36,930 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.4 | 112.6 | 298 ms | 8.9 ms | 1.0 | 0 | 3,371 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 209.6 | 104.8 | 396 ms | 9.5 ms | 2.0 | 0 | 6,286 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 372.0 | 92.8 | 585 ms | 10.8 ms | 4.0 | 0 | 11,156 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 559.5 | 69.8 | 955 ms | 14.3 ms | 8.0 | 0 | 16,785 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 801.6 | 50.0 | 1,704 ms | 20.0 ms | 16.0 | 0 | 24,049 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 112.8 | 113.1 | 575 ms | 8.8 ms | 1.0 | 0 | 3,383 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 208.4 | 104.3 | 835 ms | 9.6 ms | 2.0 | 0 | 6,252 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 367.8 | 91.8 | 1,247 ms | 10.9 ms | 4.0 | 0 | 11,032 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 552.6 | 69.0 | 2,141 ms | 14.5 ms | 8.0 | 0 | 16,576 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 111.0 | 111.4 | 1,275 ms | 9.0 ms | 1.0 | 0 | 3,329 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 204.9 | 102.6 | 1,853 ms | 9.7 ms | 2.0 | 0 | 6,146 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 357.0 | 89.3 | 2,859 ms | 11.2 ms | 4.0 | 0 | 10,707 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 540.4 | 67.6 | 4,940 ms | 14.8 ms | 8.0 | 0 | 16,208 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 107.2 | 107.9 | 2,800 ms | 9.3 ms | 1.0 | 0 | 3,217 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 198.2 | 99.3 | 3,781 ms | 10.1 ms | 2.0 | 0 | 5,944 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 349.8 | 87.7 | 6,207 ms | 11.4 ms | 4.0 | 0 | 10,493 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.