Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 768GB ×8
vllm · 0.32.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1,048,512
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- CPU
- Intel(R) Xeon(R) Platinum 8559C
- RAM
- 2000 GB
- OS
- Ubuntu 24.04.3 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Build
- ghcr.io/alvarobartt/vllm@sha256:ebdd9351069705b01e49d3fe080a427729c5680c5fd87d13c7fef163d041838a
- Commit
889b538fc509
Run
- Served model
- deepseek-ai/DeepSeek-V4.1-Flash-1017
- Revision
- 1bae0e1088993b9b82de08f59de5be5a34e42170
- Benchmark tool
- llm-inference-bench v0.8.0+prefill-window
- Decode test
- Fixed duration
- Prefill mode
- Separate cold-cache pass
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
vllm serve /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash-1017/snapshots/1bae0e1088993b9b82de08f59de5be5a34e42170 --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --tokenizer-mode deepseek_v41 --tensor-parallel-size 8 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 --mm-encoder-tp-mode data --block-size 64 --speculative-config '{"method": "dspark", "num_speculative_tokens": 5, "revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "enable_adaptive_verification": true, "use_local_argmax_reduction": true}' --enable-expert-parallel --gpu-memory-utilization 0.95 --max-num-batched-tokens 3840 --compilation-config '{"max_cudagraph_capture_size":768,"cudagraph_capture_sizes":[1,2,4,6,8,12,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184,192,200,208,216,224,232,240,248,256,272,288,304,320,336,352,368,384,400,416,432,448,464,480,496,512,640,768]}' --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017Prefill
prompt tok/s by context length8K context14.3K574 ms TTFT
16K context15.4K1,055 ms TTFT
32K context15.9K2,035 ms TTFT
64K context15.9K4,051 ms TTFT
128K context15.6K8,251 ms TTFT
256K context15.1K17,065 ms TTFT
512K context14K36,755 ms TTFT
1019K context12.3K82,981 ms TTFT
1,048,512 context12.3K83,212 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 281.074 ms TTFT | 486.492 ms TTFT | 751.692 ms TTFT | 1,035.6115 ms TTFT | 1,353.6507 ms TTFT | 1,901.1878 ms TTFT | 2,863.71,170 ms TTFT | 4,156.31,961 ms TTFT |
| 16K | 274.889 ms TTFT | 474.799 ms TTFT | 739.599 ms TTFT | 1,026.1108 ms TTFT | 1,339.7425 ms TTFT | 1,882.7657 ms TTFT | 2,811.51,156 ms TTFT | 4,048.02,171 ms TTFT |
| 32K | 268.1132 ms TTFT | 469.5148 ms TTFT | 713.3163 ms TTFT | 1,011.8180 ms TTFT | 1,307.2576 ms TTFT | 1,819.41,042 ms TTFT | 2,598.12,018 ms TTFT | 3,807.23,758 ms TTFT |
| 64K | 266.8210 ms TTFT | 461.5226 ms TTFT | 723.5229 ms TTFT | 1,001.8263 ms TTFT | 1,307.7998 ms TTFT | 1,806.31,948 ms TTFT | 2,533.53,648 ms TTFT | 3,661.47,155 ms TTFT |
| 128K | 266.0376 ms TTFT | 448.3427 ms TTFT | 688.0392 ms TTFT | 985.8594 ms TTFT | 1,256.52,055 ms TTFT | 1,751.13,812 ms TTFT | 2,405.17,266 ms TTFT | 3,509.414,852 ms TTFT |
| 256K | 260.0697 ms TTFT | 431.0799 ms TTFT | 673.21,080 ms TTFT | 944.11,413 ms TTFT | 1,209.24,151 ms TTFT | 1,674.48,277 ms TTFT | 2,324.615,957 ms TTFT | Over capacity |
| 512K | 257.81,463 ms TTFT | 416.81,576 ms TTFT | 639.22,125 ms TTFT | 896.03,168 ms TTFT | 1,151.98,163 ms TTFT | 1,556.818,419 ms TTFT | Over capacity | Over capacity |
| 1019K | 222.53,059 ms TTFT | 398.3Requests queued3,142 ms TTFT | 545.5Requests queued4,018 ms TTFT | 799.7Requests queued7,261 ms TTFT | 1,027.714,557 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.3K | — | 574 ms | — | — | 30 | — | — |
| Prefill | 16K | 1 | Valid | 15.4K | — | 1,055 ms | — | — | 20 | — | — |
| Prefill | 32K | 1 | Valid | 15.9K | — | 2,035 ms | — | — | 12 | — | — |
| Prefill | 64K | 1 | Valid | 15.9K | — | 4,051 ms | — | — | 8 | — | — |
| Prefill | 128K | 1 | Valid | 15.6K | — | 8,251 ms | — | — | 4 | — | — |
| Prefill | 256K | 1 | Valid | 15.1K | — | 17,065 ms | — | — | 2 | — | — |
| Prefill | 512K | 1 | Valid | 14K | — | 36,755 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 12.3K | — | 82,981 ms | — | — | 1 | — | — |
| Prefill | 1,048,512 | 1 | Valid | 12.3K | — | 83,212 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 281.0 | 278.9 | 74 ms | 3.6 ms | 1.0 | 2 | 8,430 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 486.4 | 238.3 | 92 ms | 4.2 ms | 2.0 | 3 | 14,589 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 751.6 | 186.1 | 92 ms | 5.4 ms | 4.0 | 4 | 22,540 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,035.6 | 130.9 | 115 ms | 7.6 ms | 8.0 | 8 | 31,065 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,353.6 | 84.2 | 507 ms | 11.9 ms | 16.0 | 0 | 40,603 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,901.1 | 59.8 | 878 ms | 16.7 ms | 32.0 | 0 | 56,978 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,863.7 | 44.6 | 1,170 ms | 22.4 ms | 64.0 | 0 | 85,836 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,156.3 | 31.1 | 1,961 ms | 32.2 ms | 128.0 | 0 | 124,543 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 274.8 | 277.9 | 89 ms | 3.6 ms | 1.0 | 2 | 8,242 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 474.7 | 234.5 | 99 ms | 4.3 ms | 2.0 | 2 | 14,239 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 739.5 | 183.5 | 99 ms | 5.5 ms | 4.0 | 4 | 22,184 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,026.1 | 127.3 | 108 ms | 7.9 ms | 8.0 | 8 | 30,783 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,339.7 | 84.4 | 425 ms | 11.8 ms | 16.0 | 0 | 40,174 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,882.7 | 58.4 | 657 ms | 17.1 ms | 32.0 | 0 | 56,447 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,811.5 | 44.3 | 1,156 ms | 22.6 ms | 64.0 | 0 | 84,279 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,048.0 | 31.3 | 2,171 ms | 31.9 ms | 128.0 | 0 | 121,409 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 268.1 | 270.5 | 132 ms | 3.7 ms | 1.0 | 2 | 8,041 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 469.5 | 244.0 | 148 ms | 4.1 ms | 2.0 | 3 | 14,084 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 713.3 | 178.9 | 163 ms | 5.6 ms | 4.0 | 4 | 21,394 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,011.8 | 125.2 | 180 ms | 8.0 ms | 7.9 | 8 | 30,346 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,307.2 | 82.3 | 576 ms | 12.1 ms | 16.0 | 0 | 39,197 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,819.4 | 56.9 | 1,042 ms | 17.6 ms | 32.0 | 0 | 54,539 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,598.1 | 41.2 | 2,018 ms | 24.2 ms | 64.0 | 0 | 77,813 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,807.2 | 29.9 | 3,758 ms | 33.4 ms | 128.0 | 0 | 114,062 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 266.8 | 272.0 | 210 ms | 3.7 ms | 1.0 | 2 | 8,002 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 461.5 | 230.2 | 226 ms | 4.3 ms | 2.0 | 2 | 13,840 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 723.5 | 182.1 | 229 ms | 5.5 ms | 3.9 | 4 | 21,700 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,001.8 | 125.2 | 263 ms | 8.0 ms | 7.9 | 7 | 30,045 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,307.7 | 81.3 | 998 ms | 12.3 ms | 16.0 | 0 | 39,217 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,806.3 | 57.1 | 1,948 ms | 17.5 ms | 32.0 | 0 | 54,164 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,533.5 | 40.4 | 3,648 ms | 24.7 ms | 64.0 | 0 | 75,983 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,661.4 | 29.3 | 7,155 ms | 34.2 ms | 128.0 | 0 | 109,696 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 266.0 | 268.0 | 376 ms | 3.7 ms | 1.0 | 2 | 7,981 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 448.3 | 229.6 | 427 ms | 4.4 ms | 2.0 | 2 | 13,446 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 688.0 | 173.8 | 392 ms | 5.8 ms | 3.9 | 4 | 20,639 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 985.8 | 126.1 | 594 ms | 7.9 ms | 7.9 | 7 | 29,571 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,256.5 | 79.8 | 2,055 ms | 12.5 ms | 16.0 | 0 | 37,670 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,751.1 | 56.0 | 3,812 ms | 17.9 ms | 32.0 | 0 | 52,531 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,405.1 | 38.9 | 7,266 ms | 25.7 ms | 64.0 | 0 | 72,060 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,509.4 | 28.8 | 14,852 ms | 34.7 ms | 128.0 | 0 | 105,190 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 260.0 | 285.5 | 697 ms | 3.5 ms | 0.9 | 2 | 7,801 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 431.0 | 220.3 | 799 ms | 4.5 ms | 1.9 | 2 | 12,929 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 673.2 | 172.8 | 1,080 ms | 5.8 ms | 3.9 | 4 | 20,189 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 944.1 | 119.3 | 1,413 ms | 8.4 ms | 7.9 | 6 | 28,318 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,209.2 | 76.9 | 4,151 ms | 13.0 ms | 16.0 | 0 | 36,244 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,674.4 | 53.2 | 8,277 ms | 18.8 ms | 32.0 | 0 | 50,210 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 2,324.6 | 37.4 | 15,957 ms | 26.8 ms | 64.0 | 0 | 69,633 | 30.0 s |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 257.8 | 270.4 | 1,463 ms | 3.7 ms | 0.9 | 2 | 7,393 | 28.7 s |
| Sustained decode | 512K | 2 | Valid | 416.8 | 219.4 | 1,576 ms | 4.6 ms | 1.9 | 2 | 12,499 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 639.2 | 168.7 | 2,125 ms | 5.9 ms | 3.8 | 4 | 19,175 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 896.0 | 117.2 | 3,168 ms | 8.5 ms | 7.8 | 5 | 26,874 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,151.9 | 73.2 | 8,163 ms | 13.7 ms | 15.8 | 4 | 34,555 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,556.8 | 49.4 | 18,419 ms | 20.2 ms | 32.0 | 1 | 46,692 | 30.0 s |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 222.5 | 249.0 | 3,059 ms | 4.0 ms | 0.9 | 1 | 6,673 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 398.3 | 219.5 | 3,142 ms | 4.6 ms | 1.8 | 2 | 11,947 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 545.5 | 160.3 | 4,018 ms | 6.2 ms | 3.6 | 4 | 16,364 | 30.0 s |
| Sustained decode | 1019K | 8 | Requests queued | 799.7 | 108.4 | 7,261 ms | 9.2 ms | 7.5 | 6 | 23,987 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 1,027.7 | 66.4 | 14,557 ms | 15.1 ms | 15.6 | 8 | 30,802 | 30.0 s |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.