Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 384GB ×4
vllm · 0.32.0 · MXFP4+FP82 contexts (0–16K) × 3 concurrency levels (1–32)
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- CPU
- Intel(R) Xeon(R) Platinum 8559C
- RAM
- 2000 GB
- OS
- Ubuntu 24.04.3 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Build
- ghcr.io/alvarobartt/vllm@sha256:11fd738a193ce4e97cc36f64b9f6d4eddb45d731179ac42099966a7d8ec2bb7e
- Commit
a1c8a45a431e
Run
- Served model
- deepseek-ai/DeepSeek-V4.1-Flash-1017
- Revision
- 1bae0e1088993b9b82de08f59de5be5a34e42170
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Not measured
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env NCCL_TUNER_PLUGIN=/usr/local/lib/libvllm-sm120-tuner.so VLLM_SM120_TARGET_GREEDY=1 VLLM_SM120_WOA_REGISTER=1 VLLM_SM120_DIRECT_DENSE=1 OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 RAYON_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false TORCHINDUCTOR_COMPILE_THREADS=1 MAX_JOBS=1 VLLM_SM120_EXPERT_TP=4 VLLM_SM120_COMPACT_EXPERTS=1 vllm serve /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash-1017/snapshots/1bae0e1088993b9b82de08f59de5be5a34e42170 --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --tokenizer-mode deepseek_v41 --tensor-parallel-size 4 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 --mm-encoder-tp-mode data --block-size 64 --enable-expert-parallel --speculative-config '{"method":"dspark","num_speculative_tokens":5,"revision":"1bae0e1088993b9b82de08f59de5be5a34e42170","enable_adaptive_verification":true,"use_local_argmax_reduction":true}' --gpu-memory-utilization 0.97 --max-num-batched-tokens 4096 --compilation-config '{"max_cudagraph_capture_size":768,"cudagraph_capture_sizes":[1,2,4,6,8,12,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184,192,200,208,216,224,232,240,248,256,272,288,304,320,336,352,368,384,400,416,432,448,464,480,496,512,528,544,560,576,592,608,624,640,656,672,688,704,720,736,752,768]}' --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 8 | 32 |
|---|---|---|---|
| 0 | 241.654 ms TTFT | 915.1155 ms TTFT | 1,838.4352 ms TTFT |
| 16K | 252.0103 ms TTFT | 884.3295 ms TTFT | 1,815.8739 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sustained decode | 0 | 1 | Valid | 241.6 | 240.9 | 54 ms | 4.2 ms | 1.0 | 2 | 7,247 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 915.1 | 112.9 | 155 ms | 8.9 ms | 8.0 | 3 | 27,445 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,838.4 | 58.9 | 352 ms | 17.0 ms | 32.0 | 0 | 55,123 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 252.0 | 230.7 | 103 ms | 4.3 ms | 1.0 | 2 | 7,559 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 884.3 | 109.6 | 295 ms | 9.1 ms | 8.0 | 0 | 26,520 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,815.8 | 56.8 | 739 ms | 17.6 ms | 32.0 | 0 | 54,417 | 30.0 s |