Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 384GB ×4
vllm · 0.32.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1,048,512
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- CPU
- Intel(R) Xeon(R) Platinum 8559C
- RAM
- 2000 GB
- OS
- Ubuntu 24.04.3 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Build
- ghcr.io/alvarobartt/vllm@sha256:ebdd9351069705b01e49d3fe080a427729c5680c5fd87d13c7fef163d041838a
- Commit
889b538fc509
Run
- Served model
- deepseek-ai/DeepSeek-V4.1-Flash-1017
- Revision
- 1bae0e1088993b9b82de08f59de5be5a34e42170
- Benchmark tool
- llm-inference-bench v0.8.0+prefill-window
- Decode test
- Fixed duration
- Prefill mode
- Separate cold-cache pass
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
vllm serve /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash-1017/snapshots/1bae0e1088993b9b82de08f59de5be5a34e42170 --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --tokenizer-mode deepseek_v41 --tensor-parallel-size 4 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 --mm-encoder-tp-mode data --block-size 64 --enable-expert-parallel --speculative-config '{"method":"dspark","num_speculative_tokens":5,"revision":"1bae0e1088993b9b82de08f59de5be5a34e42170","enable_adaptive_verification":true,"use_local_argmax_reduction":true}' --gpu-memory-utilization 0.95 --max-num-batched-tokens 4096 --compilation-config '{"max_cudagraph_capture_size":768,"cudagraph_capture_sizes":[1,2,4,6,8,12,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184,192,200,208,216,224,232,240,248,256,272,288,304,320,336,352,368,384,400,416,432,448,464,480,496,512,528,544,560,576,592,608,624,640,656,672,688,704,720,736,752,768]}' --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017Prefill
prompt tok/s by context length8K context17K481 ms TTFT
16K context17.9K905 ms TTFT
32K context17.9K1,797 ms TTFT
64K context17.9K3,604 ms TTFT
128K context17.5K7,347 ms TTFT
256K context16.8K15,305 ms TTFT
512K context15.4K33,392 ms TTFT
1019K context13.3K76,614 ms TTFT
1,048,512 context13.3K76,981 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 194.483 ms TTFT | 333.6100 ms TTFT | 516.8113 ms TTFT | 773.1353 ms TTFT | 1,111.0493 ms TTFT | 1,778.0637 ms TTFT | 2,627.11,037 ms TTFT | 3,724.71,801 ms TTFT |
| 16K | 197.099 ms TTFT | 322.4117 ms TTFT | 525.6117 ms TTFT | 771.2290 ms TTFT | 1,111.2406 ms TTFT | 1,753.2725 ms TTFT | 2,587.81,206 ms TTFT | Over capacity |
| 32K | 187.4146 ms TTFT | 321.9157 ms TTFT | 518.9162 ms TTFT | 785.9401 ms TTFT | 1,080.4633 ms TTFT | 1,681.21,119 ms TTFT | Over capacity | Over capacity |
| 64K | 198.5219 ms TTFT | 339.4228 ms TTFT | 511.8239 ms TTFT | 742.0617 ms TTFT | 1,067.81,131 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 202.1385 ms TTFT | 324.8393 ms TTFT | 501.9423 ms TTFT | 736.51,166 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 181.1725 ms TTFT | 311.6742 ms TTFT | 485.61,031 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 182.21,542 ms TTFT | 299.71,549 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 161.03,258 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17K | — | 481 ms | — | — | 59 | — | — |
| Prefill | 16K | 1 | Valid | 17.9K | — | 905 ms | — | — | 30 | — | — |
| Prefill | 32K | 1 | Valid | 17.9K | — | 1,797 ms | — | — | 15 | — | — |
| Prefill | 64K | 1 | Valid | 17.9K | — | 3,604 ms | — | — | 8 | — | — |
| Prefill | 128K | 1 | Valid | 17.5K | — | 7,347 ms | — | — | 4 | — | — |
| Prefill | 256K | 1 | Valid | 16.8K | — | 15,305 ms | — | — | 2 | — | — |
| Prefill | 512K | 1 | Valid | 15.4K | — | 33,392 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 13.3K | — | 76,614 ms | — | — | 1 | — | — |
| Prefill | 1,048,512 | 1 | Valid | 13.3K | — | 76,981 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 194.4 | 193.0 | 83 ms | 5.2 ms | 1.0 | 1 | 5,831 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 333.6 | 163.8 | 100 ms | 6.1 ms | 2.0 | 2 | 10,008 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 516.8 | 128.3 | 113 ms | 7.8 ms | 4.0 | 4 | 15,504 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 773.1 | 97.3 | 353 ms | 10.3 ms | 8.0 | 0 | 23,186 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,111.0 | 69.6 | 493 ms | 14.4 ms | 16.0 | 0 | 33,323 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,778.0 | 56.6 | 637 ms | 17.7 ms | 32.0 | 0 | 53,297 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,627.1 | 41.6 | 1,037 ms | 24.0 ms | 64.0 | 0 | 78,804 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,724.7 | 29.3 | 1,801 ms | 34.2 ms | 128.0 | 0 | 111,699 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 197.0 | 196.8 | 99 ms | 5.1 ms | 1.0 | 1 | 5,910 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 322.4 | 163.4 | 117 ms | 6.1 ms | 2.0 | 2 | 9,669 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 525.6 | 130.1 | 117 ms | 7.7 ms | 4.0 | 4 | 15,765 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 771.2 | 97.1 | 290 ms | 10.3 ms | 8.0 | 0 | 23,122 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,111.2 | 70.1 | 406 ms | 14.3 ms | 16.0 | 0 | 33,314 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,753.2 | 56.1 | 725 ms | 17.8 ms | 32.0 | 0 | 52,561 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,587.8 | 40.8 | 1,206 ms | 24.5 ms | 64.0 | 0 | 77,619 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 187.4 | 185.0 | 146 ms | 5.4 ms | 1.0 | 1 | 5,623 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 321.9 | 159.3 | 157 ms | 6.3 ms | 2.0 | 2 | 9,655 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 518.9 | 127.9 | 162 ms | 7.8 ms | 4.0 | 4 | 15,558 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 785.9 | 94.4 | 401 ms | 10.6 ms | 8.0 | 0 | 23,575 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,080.4 | 67.6 | 633 ms | 14.8 ms | 16.0 | 0 | 32,404 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,681.2 | 53.4 | 1,119 ms | 18.7 ms | 32.0 | 0 | 50,403 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 198.5 | 196.0 | 219 ms | 5.1 ms | 1.0 | 1 | 5,953 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 339.4 | 163.3 | 228 ms | 6.1 ms | 2.0 | 2 | 10,179 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 511.8 | 127.6 | 239 ms | 7.8 ms | 4.0 | 4 | 15,334 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 742.0 | 92.4 | 617 ms | 10.8 ms | 8.0 | 0 | 22,258 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,067.8 | 68.0 | 1,131 ms | 14.7 ms | 16.0 | 0 | 32,034 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 202.1 | 201.3 | 385 ms | 5.0 ms | 1.0 | 1 | 6,061 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 324.8 | 161.5 | 393 ms | 6.2 ms | 2.0 | 2 | 9,741 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 501.9 | 132.8 | 423 ms | 7.5 ms | 3.9 | 4 | 15,056 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 736.5 | 92.7 | 1,166 ms | 10.8 ms | 8.0 | 0 | 22,086 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 181.1 | 185.4 | 725 ms | 5.4 ms | 1.0 | 1 | 5,431 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 311.6 | 157.7 | 742 ms | 6.3 ms | 2.0 | 2 | 9,347 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 485.6 | 125.7 | 1,031 ms | 8.0 ms | 4.0 | 4 | 14,561 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 182.2 | 191.6 | 1,542 ms | 5.2 ms | 1.0 | 1 | 5,467 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 299.7 | 153.2 | 1,549 ms | 6.5 ms | 1.9 | 2 | 8,990 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 161.0 | 180.2 | 3,258 ms | 5.6 ms | 0.9 | 1 | 4,829 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.