Speed test matrix
DeepSeek-V4.1-Flash-1017 on H200 NVL 141GB ×2
vllm · 0.32.0 · MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- deepseek-ai/DeepSeek-V4.1-Flash-1017
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_RUST_FRONTEND=0 "$HOME/env_h200-deepseek-v41-1017-vllm-5df63a720/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash-1017 --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 262144 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true,"use_thp":false}' --offload-backend uva --cpu-offload-gb 23 --cpu-offload-params w13_weight w13_weight_scale w2_weight_scalePrefill
prompt tok/s by context length8K context2,7093,023 ms TTFT
16K context2,7595,875 ms TTFT
32K context2,82211,430 ms TTFT
64K context2,84422,626 ms TTFT
128K context2,83545,329 ms TTFT
251K context2,78890,353 ms TTFT
256K context2,77792,488 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 43.6741 ms TTFT | 56.61,122 ms TTFT | 71.21,677 ms TTFT | 89.71,853 ms TTFT | 118.82,023 ms TTFT | 163.42,943 ms TTFT | 224.53,746 ms TTFT | 303.76,541 ms TTFT |
| 16K | 43.5692 ms TTFT | 56.21,023 ms TTFT | 71.91,894 ms TTFT | 88.41,623 ms TTFT | 118.01,939 ms TTFT | 161.92,522 ms TTFT | 224.63,197 ms TTFT | Over capacity |
| 32K | 43.6793 ms TTFT | 56.61,165 ms TTFT | 69.11,721 ms TTFT | 88.91,955 ms TTFT | 118.32,200 ms TTFT | 161.22,852 ms TTFT | Over capacity | Over capacity |
| 64K | 43.5799 ms TTFT | 55.91,214 ms TTFT | 69.91,710 ms TTFT | 89.71,917 ms TTFT | 118.22,235 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 43.5843 ms TTFT | 55.71,182 ms TTFT | 69.52,053 ms TTFT | 90.22,525 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 43.31,258 ms TTFT | 57.11,624 ms TTFT | 71.32,334 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,709.0 | — | 3,023 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,759.0 | — | 5,875 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,822.0 | — | 11,430 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,844.0 | — | 22,626 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,835.0 | — | 45,329 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,788.0 | — | 90,353 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,777.0 | — | 92,488 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 43.6 | 43.6 | 741 ms | 22.9 ms | 1.0 | 0 | 1,308 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 56.6 | 28.0 | 1,122 ms | 35.7 ms | 2.0 | 0 | 1,696 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 71.2 | 17.9 | 1,677 ms | 55.8 ms | 4.0 | 0 | 2,136 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 89.7 | 11.5 | 1,853 ms | 87.1 ms | 8.0 | 0 | 2,688 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 118.8 | 7.7 | 2,023 ms | 129.2 ms | 16.0 | 0 | 3,551 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 163.4 | 5.4 | 2,943 ms | 185.6 ms | 32.0 | 0 | 4,894 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 224.5 | 3.5 | 3,746 ms | 285.8 ms | 64.0 | 0 | 6,668 | 29.7 s |
| Sustained decode | 8K | 128 | Valid | 303.7 | 2.3 | 6,541 ms | 439.6 ms | 128.0 | 0 | 9,086 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 43.5 | 43.6 | 692 ms | 22.9 ms | 1.0 | 0 | 1,306 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 56.2 | 28.0 | 1,023 ms | 35.7 ms | 2.0 | 0 | 1,683 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 71.9 | 18.2 | 1,894 ms | 55.1 ms | 4.0 | 0 | 2,155 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 88.4 | 11.4 | 1,623 ms | 88.0 ms | 8.0 | 0 | 2,648 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 118.0 | 7.7 | 1,939 ms | 129.8 ms | 16.0 | 0 | 3,536 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 161.9 | 5.3 | 2,522 ms | 188.1 ms | 32.0 | 0 | 4,833 | 29.9 s |
| Sustained decode | 16K | 64 | Valid | 224.6 | 3.5 | 3,197 ms | 287.5 ms | 64.0 | 0 | 6,727 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 43.6 | 43.6 | 793 ms | 23.0 ms | 1.0 | 0 | 1,306 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 56.6 | 28.0 | 1,165 ms | 35.7 ms | 2.0 | 0 | 1,696 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 69.1 | 17.7 | 1,721 ms | 56.7 ms | 4.0 | 0 | 2,069 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 88.9 | 11.4 | 1,955 ms | 87.7 ms | 8.0 | 0 | 2,664 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 118.3 | 7.7 | 2,200 ms | 130.0 ms | 16.0 | 0 | 3,536 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 161.2 | 5.0 | 2,852 ms | 198.5 ms | 32.0 | 0 | 4,832 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 43.5 | 43.5 | 799 ms | 23.0 ms | 1.0 | 0 | 1,305 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 55.9 | 27.8 | 1,214 ms | 36.0 ms | 2.0 | 0 | 1,676 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 69.9 | 17.6 | 1,710 ms | 56.8 ms | 4.0 | 0 | 2,097 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 89.7 | 11.6 | 1,917 ms | 86.5 ms | 8.0 | 0 | 2,687 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 118.2 | 7.5 | 2,235 ms | 134.0 ms | 16.0 | 0 | 3,536 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 43.5 | 43.5 | 843 ms | 23.0 ms | 1.0 | 0 | 1,303 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 55.7 | 27.7 | 1,182 ms | 36.0 ms | 2.0 | 0 | 1,672 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 69.5 | 17.7 | 2,053 ms | 56.6 ms | 4.0 | 0 | 2,084 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 90.2 | 11.3 | 2,525 ms | 88.2 ms | 8.0 | 0 | 2,704 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 43.3 | 43.4 | 1,258 ms | 23.0 ms | 1.0 | 0 | 1,299 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 57.1 | 28.4 | 1,624 ms | 35.2 ms | 2.0 | 0 | 1,713 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 71.3 | 17.7 | 2,334 ms | 56.6 ms | 4.0 | 0 | 2,140 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.