Speed test matrix
DeepSeek-V4-Flash-0731 on H200 NVL 141GB ×2
vllm · 0.25.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.25.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-vllm/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash-0731 --served-model-name deepseek --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.700892 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'Prefill
prompt tok/s by context length8K context11.5K712 ms TTFT
16K context11.7K1,395 ms TTFT
32K context11.4K2,831 ms TTFT
64K context11.4K5,646 ms TTFT
128K context11.1K11,602 ms TTFT
256K context10.2K25,230 ms TTFT
512K context8,53460,434 ms TTFT
1019K context6,305162,765 ms TTFT
1024K context6,256164,847 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 329.393 ms TTFT | 622.6103 ms TTFT | 559.2140 ms TTFT | 1,038.4267 ms TTFT | 1,535.7308 ms TTFT | 2,121.0758 ms TTFT | 3,266.61,120 ms TTFT | 3,440.91,994 ms TTFT |
| 16K | 417.2116 ms TTFT | 513.1145 ms TTFT | 748.9153 ms TTFT | 1,133.4233 ms TTFT | 1,694.4376 ms TTFT | 2,109.6622 ms TTFT | 3,372.9967 ms TTFT | 3,818.91,562 ms TTFT |
| 32K | 401.6141 ms TTFT | 411.9190 ms TTFT | 672.7190 ms TTFT | 1,226.5248 ms TTFT | 1,703.3451 ms TTFT | 2,203.7898 ms TTFT | 3,454.91,520 ms TTFT | 3,978.42,775 ms TTFT |
| 64K | 286.6209 ms TTFT | 647.6269 ms TTFT | 740.9281 ms TTFT | 1,198.6308 ms TTFT | 1,952.4562 ms TTFT | 2,558.01,458 ms TTFT | 3,591.12,767 ms TTFT | 3,806.35,314 ms TTFT |
| 128K | 196.0359 ms TTFT | 500.4407 ms TTFT | 705.4391 ms TTFT | 1,171.8601 ms TTFT | 1,341.11,526 ms TTFT | 2,052.23,024 ms TTFT | 3,031.95,842 ms TTFT | 3,791.411,774 ms TTFT |
| 256K | 259.9635 ms TTFT | 364.2792 ms TTFT | 640.7743 ms TTFT | 1,241.8921 ms TTFT | 1,638.12,088 ms TTFT | 2,369.36,033 ms TTFT | 2,964.711,420 ms TTFT | 3,262.224,816 ms TTFT |
| 512K | 406.51,305 ms TTFT | 436.11,592 ms TTFT | 540.51,799 ms TTFT | 1,031.02,226 ms TTFT | 1,408.35,351 ms TTFT | 1,517.513,491 ms TTFT | 2,088.424,601 ms TTFT | 2,281.448,482 ms TTFT |
| 1019K | 193.02,650 ms TTFT | 254.3Requests queued3,079 ms TTFT | 479.03,688 ms TTFT | 542.58,114 ms TTFT | 733.718,023 ms TTFT | 851.831,135 ms TTFT | 1,014.557,826 ms TTFT | 1,007.7Requests queued107,783 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.5K | — | 712 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.7K | — | 1,395 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.4K | — | 2,831 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.4K | — | 5,646 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.1K | — | 11,602 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.2K | — | 25,230 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 8,534.0 | — | 60,434 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,305.0 | — | 162,765 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,256.0 | — | 164,847 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 329.3 | 421.7 | 93 ms | 2.4 ms | 1.0 | 2 | 9,875 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 622.6 | 358.1 | 103 ms | 2.8 ms | 2.0 | 4 | 18,676 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 559.2 | 130.1 | 140 ms | 7.7 ms | 4.0 | 4 | 16,772 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,038.4 | 124.0 | 267 ms | 8.5 ms | 8.0 | 4 | 31,133 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,535.7 | 71.9 | 308 ms | 13.9 ms | 15.9 | 6 | 46,061 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,121.0 | 49.9 | 758 ms | 20.0 ms | 32.0 | 0 | 63,593 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,266.6 | 39.1 | 1,120 ms | 25.6 ms | 64.0 | 0 | 97,866 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,440.9 | 22.7 | 1,994 ms | 44.0 ms | 128.0 | 0 | 102,972 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 417.2 | 390.6 | 116 ms | 2.6 ms | 1.0 | 3 | 12,517 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 513.1 | 255.7 | 145 ms | 3.9 ms | 2.0 | 3 | 15,385 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 748.9 | 152.3 | 153 ms | 6.6 ms | 4.0 | 5 | 22,455 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,133.4 | 112.2 | 233 ms | 8.9 ms | 7.9 | 5 | 33,984 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,694.4 | 85.0 | 376 ms | 11.8 ms | 15.9 | 6 | 50,796 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,109.6 | 49.5 | 622 ms | 20.2 ms | 32.0 | 2 | 63,240 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,372.9 | 44.3 | 967 ms | 22.6 ms | 64.0 | 0 | 101,057 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,818.9 | 24.9 | 1,562 ms | 40.2 ms | 128.0 | 0 | 114,566 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 401.6 | 361.4 | 141 ms | 2.8 ms | 1.0 | 3 | 12,047 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 411.9 | 217.0 | 190 ms | 4.6 ms | 2.0 | 2 | 12,358 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 672.7 | 145.7 | 190 ms | 6.9 ms | 4.0 | 4 | 20,181 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,226.5 | 136.9 | 248 ms | 7.3 ms | 7.9 | 6 | 36,765 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,703.3 | 97.7 | 451 ms | 10.2 ms | 16.0 | 6 | 51,054 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,203.7 | 57.0 | 898 ms | 17.5 ms | 32.0 | 2 | 66,073 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,454.9 | 43.3 | 1,520 ms | 23.1 ms | 64.0 | 0 | 103,454 | 29.9 s |
| Sustained decode | 32K | 128 | Valid | 3,978.4 | 25.4 | 2,775 ms | 39.4 ms | 128.0 | 0 | 119,213 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 286.6 | 298.8 | 209 ms | 3.3 ms | 1.0 | 2 | 8,594 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 647.6 | 347.8 | 269 ms | 3.0 ms | 1.9 | 4 | 19,429 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 740.9 | 195.2 | 281 ms | 5.2 ms | 4.0 | 4 | 22,213 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,198.6 | 165.0 | 308 ms | 6.1 ms | 7.9 | 6 | 35,948 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,952.4 | 112.9 | 562 ms | 8.9 ms | 15.9 | 10 | 58,547 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,558.0 | 71.3 | 1,458 ms | 14.0 ms | 32.0 | 4 | 76,742 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,591.1 | 45.9 | 2,767 ms | 21.8 ms | 64.0 | 0 | 107,662 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,806.3 | 26.3 | 5,314 ms | 38.0 ms | 128.0 | 0 | 114,252 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 196.0 | 201.5 | 359 ms | 5.0 ms | 1.0 | 1 | 5,877 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 500.4 | 317.5 | 407 ms | 3.1 ms | 2.0 | 3 | 15,011 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 705.4 | 180.1 | 391 ms | 5.8 ms | 4.0 | 4 | 21,155 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,171.8 | 148.1 | 601 ms | 6.8 ms | 8.0 | 6 | 35,128 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,341.1 | 68.0 | 1,526 ms | 14.7 ms | 16.0 | 2 | 40,225 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,052.2 | 49.3 | 3,024 ms | 20.3 ms | 32.0 | 1 | 61,467 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,031.9 | 36.6 | 5,842 ms | 27.4 ms | 64.0 | 0 | 90,937 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,791.4 | 25.4 | 11,774 ms | 39.4 ms | 128.0 | 0 | 113,741 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 259.9 | 262.9 | 635 ms | 3.8 ms | 1.0 | 2 | 7,795 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 364.2 | 154.8 | 792 ms | 6.5 ms | 2.0 | 2 | 10,925 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 640.7 | 131.8 | 743 ms | 7.6 ms | 3.9 | 4 | 19,200 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,241.8 | 159.0 | 921 ms | 6.3 ms | 7.9 | 7 | 37,224 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,638.1 | 82.5 | 2,088 ms | 12.1 ms | 15.9 | 7 | 49,078 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 2,369.3 | 67.2 | 6,033 ms | 14.9 ms | 31.9 | 2 | 70,987 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 2,964.7 | 41.3 | 11,420 ms | 24.2 ms | 63.9 | 6 | 88,351 | 29.8 s |
| Sustained decode | 256K | 128 | Valid | 3,262.2 | 22.5 | 24,816 ms | 44.5 ms | 127.8 | 7 | 97,454 | 29.9 s |
| Sustained decode | 512K | 1 | Valid | 406.5 | 433.8 | 1,305 ms | 2.3 ms | 0.9 | 3 | 12,190 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 436.1 | 212.3 | 1,592 ms | 4.7 ms | 1.9 | 2 | 13,075 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 540.5 | 107.6 | 1,799 ms | 9.3 ms | 3.9 | 2 | 16,209 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 1,031.0 | 133.8 | 2,226 ms | 7.5 ms | 7.8 | 6 | 30,917 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,408.3 | 78.1 | 5,351 ms | 12.8 ms | 15.7 | 8 | 42,211 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,517.5 | 37.2 | 13,491 ms | 26.9 ms | 31.9 | 4 | 45,504 | 30.0 s |
| Sustained decode | 512K | 64 | Valid | 2,088.4 | 29.5 | 24,601 ms | 33.9 ms | 63.7 | 9 | 62,657 | 30.0 s |
| Sustained decode | 512K | 128 | Valid | 2,281.4 | 17.5 | 48,482 ms | 57.3 ms | 127.4 | 32 | 68,178 | 29.9 s |
| Sustained decode | 1019K | 1 | Valid | 193.0 | 245.8 | 2,650 ms | 4.4 ms | 0.9 | 1 | 5,789 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 254.3 | 148.0 | 3,079 ms | 6.8 ms | 1.8 | 2 | 7,629 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 479.0 | 98.0 | 3,688 ms | 10.2 ms | 3.9 | 2 | 14,362 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 542.5 | 64.4 | 8,114 ms | 15.5 ms | 7.8 | 2 | 16,262 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 733.7 | 44.0 | 18,023 ms | 22.7 ms | 15.9 | 1 | 21,978 | 30.0 s |
| Sustained decode | 1019K | 32 | Valid | 851.8 | 26.5 | 31,135 ms | 37.9 ms | 31.9 | 2 | 25,520 | 30.0 s |
| Sustained decode | 1019K | 64 | Valid | 1,014.5 | 17.3 | 57,826 ms | 57.7 ms | 63.8 | 7 | 30,397 | 30.0 s |
| Sustained decode | 1019K | 128 | Requests queued | 1,007.7 | 12.8 | 107,783 ms | 78.2 ms | 92.1 | 9 | 30,142 | 29.9 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.