Speed test matrix
DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell Server Edition 768GB ×8
vllm · 0.32.0 · MXFP4+FP82 contexts (0–16K) × 3 concurrency levels (1–32)
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- CPU
- Intel(R) Xeon(R) Platinum 8559C
- RAM
- 2000 GB
- OS
- Ubuntu 24.04.3 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Build
- ghcr.io/alvarobartt/vllm@sha256:11fd738a193ce4e97cc36f64b9f6d4eddb45d731179ac42099966a7d8ec2bb7e
- Commit
a1c8a45a431e
Run
- Served model
- deepseek-ai/DeepSeek-V4.1-Flash-1017
- Revision
- 1bae0e1088993b9b82de08f59de5be5a34e42170
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Not measured
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env NCCL_TUNER_PLUGIN=/usr/local/lib/libvllm-sm120-tuner.so VLLM_SM120_TARGET_GREEDY=1 VLLM_SM120_WOA_REGISTER=1 VLLM_SM120_DIRECT_DENSE=1 OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 RAYON_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false TORCHINDUCTOR_COMPILE_THREADS=1 MAX_JOBS=1 VLLM_SM120_EXPERT_TP=8 VLLM_SM120_COMPACT_EXPERTS=1 vllm serve /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash-1017/snapshots/1bae0e1088993b9b82de08f59de5be5a34e42170 --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --tokenizer-mode deepseek_v41 --tensor-parallel-size 8 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 --mm-encoder-tp-mode data --block-size 64 --speculative-config '{"method": "dspark", "num_speculative_tokens": 5, "revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "enable_adaptive_verification": true, "use_local_argmax_reduction": true}' --enable-expert-parallel --gpu-memory-utilization 0.965 --max-num-batched-tokens 3840 --compilation-config '{"max_cudagraph_capture_size":768,"cudagraph_capture_sizes":[1,2,4,6,8,12,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184,192,200,208,216,224,232,240,248,256,272,288,304,320,336,352,368,384,400,416,432,448,464,480,496,512,640,768]}' --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 8 | 32 |
|---|---|---|---|
| 0 | 312.547 ms TTFT | 1,049.381 ms TTFT | 1,929.4357 ms TTFT |
| 16K | 281.7103 ms TTFT | 1,035.7142 ms TTFT | 1,879.0698 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sustained decode | 0 | 1 | Valid | 312.5 | 291.7 | 47 ms | 3.4 ms | 1.0 | 2 | 9,372 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,049.3 | 127.6 | 81 ms | 7.8 ms | 8.0 | 8 | 31,479 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,929.4 | 61.1 | 357 ms | 16.4 ms | 32.0 | 0 | 57,866 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 281.7 | 263.0 | 103 ms | 3.8 ms | 1.0 | 2 | 8,451 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,035.7 | 127.2 | 142 ms | 7.9 ms | 7.9 | 8 | 31,059 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,879.0 | 58.5 | 698 ms | 17.1 ms | 32.0 | 0 | 56,363 | 30.0 s |