Speed test matrix
DeepSeek-V4-Flash-Vision-Exp on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev317+g1356635d8 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev317+g1356635d8
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
1356635d837c
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-vision-vllm-pr-54566/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp --served-model-name deepseek --trust-remote-code --tensor-parallel-size 4 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.783310 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}'Prefill
prompt tok/s by context length8K context19.3K425 ms TTFT
16K context21.2K762 ms TTFT
32K context21.6K1,487 ms TTFT
64K context21.4K2,994 ms TTFT
128K context20K6,386 ms TTFT
256K context17K14,987 ms TTFT
512K context13.1K39,089 ms TTFT
1019K context8,773115,746 ms TTFT
1024K context8,680117,569 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 244.693 ms TTFT | 439.4106 ms TTFT | 750.3102 ms TTFT | 1,259.4130 ms TTFT | 2,087.2162 ms TTFT | 3,208.25,573 ms TTFT | Looped | 6,553.44,454 ms TTFT |
| 16K | 243.1740 ms TTFT | 435.6762 ms TTFT | 684.41,052 ms TTFT | 1,052.71,364 ms TTFT | 1,889.74,338 ms TTFT | 3,215.510,899 ms TTFT | 4,485.121,285 ms TTFT | 1,474.8Requests queued42,686 ms TTFT |
| 32K | 246.1188 ms TTFT | 447.5211 ms TTFT | 777.2212 ms TTFT | 1,249.5233 ms TTFT | 2,052.5342 ms TTFT | 3,164.51,169 ms TTFT | 4,429.52,062 ms TTFT | 6,252.83,932 ms TTFT |
| 64K | 243.0310 ms TTFT | Looped | 746.0299 ms TTFT | 1,230.3329 ms TTFT | 1,995.0471 ms TTFT | 3,129.21,967 ms TTFT | Looped | Looped |
| 128K | 209.36,366 ms TTFT | 296.56,467 ms TTFT | 463.5Requests queued12,562 ms TTFT | 945.0Requests queued24,416 ms TTFT | 1,655.751,280 ms TTFT | 1,066.6Requests queued99,334 ms TTFT | 173.6Requests queued102,591 ms TTFT | 181.6Requests queued102,792 ms TTFT |
| 256K | 116.614,868 ms TTFT | 221.122,212 ms TTFT | 431.2Requests queued36,128 ms TTFT | 716.464,466 ms TTFT | 58.4Requests queued99,856 ms TTFT | 57.7Requests queued100,030 ms TTFT | 58.1Requests queued100,136 ms TTFT | 59.4Requests queued100,653 ms TTFT |
| 512K | 214.11,868 ms TTFT | 401.51,921 ms TTFT | 684.51,955 ms TTFT | 1,013.1Requests queued2,726 ms TTFT | 1,577.56,512 ms TTFT | Looped | Looped | Looped |
| 1019K | 196.13,863 ms TTFT | 376.7Requests queued3,970 ms TTFT | 534.9Requests queued4,703 ms TTFT | 860.3Requests queued6,616 ms TTFT | 1,226.3Requests queued12,069 ms TTFT | Looped | Looped | Looped |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.3K | — | 425 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.2K | — | 762 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.6K | — | 1,487 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.4K | — | 2,994 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20K | — | 6,386 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17K | — | 14,987 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.1K | — | 39,089 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,773.0 | — | 115,746 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,680.0 | — | 117,569 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 244.6 | 245.6 | 93 ms | 4.1 ms | 1.0 | 1 | 7,337 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 439.4 | 217.8 | 106 ms | 4.6 ms | 2.0 | 2 | 13,183 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 750.3 | 187.5 | 102 ms | 5.3 ms | 4.0 | 4 | 22,503 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,259.4 | 157.1 | 130 ms | 6.4 ms | 8.0 | 8 | 37,784 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,087.2 | 129.1 | 162 ms | 7.7 ms | 16.0 | 16 | 62,604 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,208.2 | 97.7 | 5,573 ms | 10.2 ms | 32.0 | 0 | 96,258 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 6,553.4 | 50.7 | 4,454 ms | 19.7 ms | 128.0 | 0 | 196,574 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 243.1 | 247.8 | 740 ms | 4.0 ms | 1.0 | 2 | 7,279 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 435.6 | 220.6 | 762 ms | 4.5 ms | 2.0 | 2 | 13,067 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 684.4 | 173.6 | 1,052 ms | 5.8 ms | 4.0 | 4 | 20,533 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,052.7 | 129.6 | 1,364 ms | 7.7 ms | 7.9 | 8 | 31,573 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,889.7 | 105.7 | 4,338 ms | 9.5 ms | 16.0 | 5 | 56,574 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 3,215.5 | 79.4 | 10,899 ms | 12.6 ms | 32.0 | 0 | 96,456 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,485.1 | 46.3 | 21,285 ms | 21.6 ms | 64.0 | 0 | 134,572 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,474.8 | 10.6 | 42,686 ms | 94.3 ms | 111.7 | 0 | 44,251 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 246.1 | 252.6 | 188 ms | 4.0 ms | 1.0 | 2 | 7,388 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 447.5 | 225.2 | 211 ms | 4.4 ms | 2.0 | 2 | 13,423 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 777.2 | 188.0 | 212 ms | 5.3 ms | 4.0 | 4 | 23,309 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,249.5 | 155.9 | 233 ms | 6.4 ms | 8.0 | 8 | 37,487 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,052.5 | 132.0 | 342 ms | 7.6 ms | 16.0 | 15 | 61,563 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,164.5 | 98.0 | 1,169 ms | 10.2 ms | 32.0 | 1 | 94,919 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,429.5 | 69.2 | 2,062 ms | 14.5 ms | 64.0 | 0 | 132,864 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,252.8 | 49.4 | 3,932 ms | 20.2 ms | 128.0 | 0 | 187,655 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 243.0 | 244.9 | 310 ms | 4.1 ms | 1.0 | 1 | 7,290 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 746.0 | 188.4 | 299 ms | 5.3 ms | 4.0 | 4 | 22,381 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,230.3 | 158.0 | 329 ms | 6.3 ms | 7.9 | 8 | 36,899 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,995.0 | 125.4 | 471 ms | 8.0 ms | 15.9 | 14 | 59,824 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,129.2 | 96.0 | 1,967 ms | 10.4 ms | 32.0 | 1 | 93,873 | 30.0 s |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 1 | Valid | 209.3 | 264.3 | 6,366 ms | 3.8 ms | 1.0 | 1 | 6,279 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 296.5 | 176.5 | 6,467 ms | 6.0 ms | 2.0 | 2 | 8,895 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 463.5 | 122.1 | 12,562 ms | 8.2 ms | 3.8 | 3 | 13,788 | 29.7 s |
| Sustained decode | 128K | 8 | Requests queued | 945.0 | 76.0 | 24,416 ms | 13.2 ms | 7.9 | 3 | 28,251 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 1,655.7 | 47.6 | 51,280 ms | 21.1 ms | 16.0 | 1 | 49,539 | 29.9 s |
| Sustained decode | 128K | 32 | Requests queued | 1,066.6 | 14.3 | 99,334 ms | 70.0 ms | 31.3 | 0 | 31,996 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 173.6 | 5.8 | 102,591 ms | 172.5 ms | 31.6 | 0 | 5,158 | 29.7 s |
| Sustained decode | 128K | 128 | Requests queued | 181.6 | 5.9 | 102,792 ms | 170.2 ms | 31.7 | 0 | 5,427 | 29.9 s |
| Sustained decode | 256K | 1 | Valid | 116.6 | 247.6 | 14,868 ms | 4.0 ms | 1.0 | 1 | 3,499 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 221.1 | 124.1 | 22,212 ms | 8.1 ms | 2.0 | 1 | 6,446 | 29.2 s |
| Sustained decode | 256K | 4 | Requests queued | 431.2 | 82.9 | 36,128 ms | 12.1 ms | 3.6 | 2 | 12,924 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 716.4 | 42.4 | 64,466 ms | 23.7 ms | 8.0 | 1 | 21,306 | 29.7 s |
| Sustained decode | 256K | 16 | Requests queued | 58.4 | 4.8 | 99,856 ms | 206.3 ms | 13.1 | 0 | 1,743 | 29.8 s |
| Sustained decode | 256K | 32 | Requests queued | 57.7 | 5.1 | 100,030 ms | 196.8 ms | 13.2 | 0 | 1,705 | 29.5 s |
| Sustained decode | 256K | 64 | Requests queued | 58.1 | 4.8 | 100,136 ms | 206.3 ms | 13.2 | 0 | 1,721 | 29.6 s |
| Sustained decode | 256K | 128 | Requests queued | 59.4 | 4.9 | 100,653 ms | 202.9 ms | 13.1 | 0 | 1,766 | 29.7 s |
| Sustained decode | 512K | 1 | Valid | 214.1 | 229.7 | 1,868 ms | 4.4 ms | 0.9 | 1 | 6,421 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 401.5 | 205.1 | 1,921 ms | 4.9 ms | 1.9 | 2 | 12,042 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 684.5 | 176.2 | 1,955 ms | 5.7 ms | 3.8 | 4 | 20,532 | 30.0 s |
| Sustained decode | 512K | 8 | Requests queued | 1,013.1 | 134.3 | 2,726 ms | 7.4 ms | 7.5 | 8 | 30,391 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,577.5 | 101.5 | 6,512 ms | 9.9 ms | 15.4 | 11 | 47,317 | 30.0 s |
| Sustained decode | 512K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 1 | Valid | 196.1 | 227.9 | 3,863 ms | 4.4 ms | 0.9 | 1 | 5,881 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 376.7 | 202.5 | 3,970 ms | 4.9 ms | 1.8 | 2 | 11,302 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 534.9 | 159.9 | 4,703 ms | 6.3 ms | 3.5 | 4 | 16,043 | 30.0 s |
| Sustained decode | 1019K | 8 | Requests queued | 860.3 | 123.2 | 6,616 ms | 8.1 ms | 7.3 | 7 | 25,823 | 30.0 s |
| Sustained decode | 1019K | 16 | Requests queued | 1,226.3 | 83.5 | 12,069 ms | 12.0 ms | 14.9 | 10 | 36,785 | 30.0 s |
| Sustained decode | 1019K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement.