Speed test matrix
DeepSeek-V4-Flash on H200 NVL 141GB ×2
vllm · 0.25.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.25.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-vllm/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.840939 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --speculative_config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context14.6K562 ms TTFT
16K context14.8K1,097 ms TTFT
32K context14.7K2,198 ms TTFT
64K context14.7K4,397 ms TTFT
128K context14K9,249 ms TTFT
256K context12.4K20,870 ms TTFT
512K context9,99751,588 ms TTFT
1019K context7,088144,798 ms TTFT
1024K context7,038146,541 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 190.990 ms TTFT | 320.9192 ms TTFT | 542.2115 ms TTFT | 823.0329 ms TTFT | 1,329.51,169 ms TTFT | 1,857.5685 ms TTFT | 2,746.32,633 ms TTFT | 3,575.41,560 ms TTFT |
| 16K | 196.6117 ms TTFT | 333.6140 ms TTFT | 566.5140 ms TTFT | 855.3306 ms TTFT | 1,266.3439 ms TTFT | 1,810.1636 ms TTFT | 2,723.4968 ms TTFT | 3,477.51,597 ms TTFT |
| 32K | 197.1131 ms TTFT | 335.3174 ms TTFT | 555.0201 ms TTFT | 846.1355 ms TTFT | 1,249.6547 ms TTFT | 1,816.0853 ms TTFT | 2,720.31,540 ms TTFT | 3,475.12,612 ms TTFT |
| 64K | 183.4190 ms TTFT | 327.1247 ms TTFT | 538.6261 ms TTFT | 846.6514 ms TTFT | 1,212.5823 ms TTFT | 1,804.21,467 ms TTFT | 2,630.02,698 ms TTFT | 3,359.75,188 ms TTFT |
| 128K | 188.0346 ms TTFT | 327.9484 ms TTFT | 511.2445 ms TTFT | 834.11,023 ms TTFT | 1,231.31,812 ms TTFT | 1,791.53,031 ms TTFT | 2,614.55,784 ms TTFT | 3,393.111,043 ms TTFT |
| 256K | 162.7607 ms TTFT | 317.3772 ms TTFT | 520.0704 ms TTFT | 807.61,905 ms TTFT | 1,127.83,334 ms TTFT | 1,747.86,149 ms TTFT | 2,478.411,854 ms TTFT | 3,092.823,183 ms TTFT |
| 512K | 173.51,247 ms TTFT | 297.81,340 ms TTFT | 486.61,337 ms TTFT | 789.14,249 ms TTFT | 1,104.67,646 ms TTFT | 1,518.414,280 ms TTFT | 2,171.928,387 ms TTFT | 2,646.352,092 ms TTFT |
| 1019K | 151.72,658 ms TTFT | 262.5Requests queued3,532 ms TTFT | 438.24,855 ms TTFT | 674.08,917 ms TTFT | 985.514,493 ms TTFT | 1,268.127,664 ms TTFT | 1,689.642,901 ms TTFT | 1,669.0Requests queued105,464 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.6K | — | 562 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.8K | — | 1,097 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.7K | — | 2,198 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.7K | — | 4,397 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14K | — | 9,249 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.4K | — | 20,870 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,997.0 | — | 51,588 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,088.0 | — | 144,798 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,038.0 | — | 146,541 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 190.9 | 193.0 | 90 ms | 5.2 ms | 1.0 | 1 | 5,726 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 320.9 | 160.2 | 192 ms | 6.3 ms | 2.0 | 2 | 9,626 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 542.2 | 136.4 | 115 ms | 7.3 ms | 4.0 | 4 | 16,265 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 823.0 | 103.2 | 329 ms | 9.7 ms | 8.0 | 0 | 24,690 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,329.5 | 81.2 | 1,169 ms | 12.3 ms | 16.0 | 0 | 39,885 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,857.5 | 57.9 | 685 ms | 17.3 ms | 32.0 | 0 | 55,720 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,746.3 | 42.5 | 2,633 ms | 23.6 ms | 64.0 | 0 | 82,303 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,575.4 | 27.5 | 1,560 ms | 36.4 ms | 128.0 | 0 | 107,167 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 196.6 | 196.0 | 117 ms | 5.1 ms | 1.0 | 1 | 5,898 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 333.6 | 163.2 | 140 ms | 6.1 ms | 2.0 | 2 | 10,006 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 566.5 | 139.7 | 140 ms | 7.2 ms | 4.0 | 4 | 16,993 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 855.3 | 106.9 | 306 ms | 9.4 ms | 8.0 | 0 | 25,677 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,266.3 | 79.4 | 439 ms | 12.6 ms | 16.0 | 0 | 37,973 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,810.1 | 55.9 | 636 ms | 17.9 ms | 32.0 | 0 | 54,293 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,723.4 | 42.5 | 968 ms | 23.5 ms | 64.0 | 0 | 81,617 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,477.5 | 27.0 | 1,597 ms | 37.0 ms | 128.0 | 0 | 104,370 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 197.1 | 194.4 | 131 ms | 5.1 ms | 1.0 | 1 | 5,912 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 335.3 | 167.9 | 174 ms | 6.0 ms | 2.0 | 2 | 10,056 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 555.0 | 134.8 | 201 ms | 7.4 ms | 4.0 | 4 | 16,643 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 846.1 | 104.9 | 355 ms | 9.5 ms | 8.0 | 0 | 25,369 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,249.6 | 78.2 | 547 ms | 12.8 ms | 16.0 | 0 | 37,463 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,816.0 | 56.9 | 853 ms | 17.6 ms | 32.0 | 0 | 54,476 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,720.3 | 42.8 | 1,540 ms | 23.4 ms | 64.0 | 0 | 81,626 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,475.1 | 26.9 | 2,612 ms | 37.1 ms | 128.0 | 0 | 104,205 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 183.4 | 184.5 | 190 ms | 5.4 ms | 1.0 | 1 | 5,500 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 327.1 | 164.6 | 247 ms | 6.1 ms | 2.0 | 2 | 9,810 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 538.6 | 132.5 | 261 ms | 7.6 ms | 4.0 | 4 | 16,154 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 846.6 | 107.5 | 514 ms | 9.3 ms | 8.0 | 0 | 25,386 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,212.5 | 76.5 | 823 ms | 13.1 ms | 16.0 | 0 | 36,361 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,804.2 | 56.2 | 1,467 ms | 17.8 ms | 32.0 | 0 | 54,093 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,630.0 | 40.9 | 2,698 ms | 24.5 ms | 64.0 | 0 | 78,916 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,359.7 | 26.3 | 5,188 ms | 38.0 ms | 128.0 | 0 | 100,753 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 188.0 | 188.4 | 346 ms | 5.3 ms | 1.0 | 1 | 5,638 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 327.9 | 161.2 | 484 ms | 6.2 ms | 2.0 | 2 | 9,835 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 511.2 | 127.6 | 445 ms | 7.8 ms | 4.0 | 4 | 15,336 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 834.1 | 101.4 | 1,023 ms | 9.9 ms | 8.0 | 0 | 25,020 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,231.3 | 77.8 | 1,812 ms | 12.8 ms | 16.0 | 0 | 36,913 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,791.5 | 55.9 | 3,031 ms | 17.9 ms | 32.0 | 0 | 53,747 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,614.5 | 40.7 | 5,784 ms | 24.6 ms | 64.0 | 0 | 78,448 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,393.1 | 26.7 | 11,043 ms | 37.4 ms | 128.0 | 0 | 101,698 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 162.7 | 173.2 | 607 ms | 5.8 ms | 1.0 | 1 | 4,881 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 317.3 | 161.3 | 772 ms | 6.2 ms | 2.0 | 2 | 9,516 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 520.0 | 131.4 | 704 ms | 7.6 ms | 3.9 | 4 | 15,597 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 807.6 | 100.0 | 1,905 ms | 10.0 ms | 8.0 | 0 | 24,224 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,127.8 | 71.0 | 3,334 ms | 14.1 ms | 16.0 | 0 | 33,821 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,747.8 | 55.0 | 6,149 ms | 18.2 ms | 32.0 | 0 | 52,429 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 2,478.4 | 39.7 | 11,854 ms | 25.2 ms | 64.0 | 0 | 74,337 | 30.0 s |
| Sustained decode | 256K | 128 | Valid | 3,092.8 | 25.5 | 23,183 ms | 39.2 ms | 128.0 | 0 | 92,757 | 30.0 s |
| Sustained decode | 512K | 1 | Valid | 173.5 | 180.7 | 1,247 ms | 5.5 ms | 1.0 | 1 | 5,205 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 297.8 | 156.3 | 1,340 ms | 6.4 ms | 1.9 | 2 | 8,931 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 486.6 | 123.3 | 1,337 ms | 8.1 ms | 3.9 | 3 | 14,596 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 789.1 | 98.4 | 4,249 ms | 10.2 ms | 8.0 | 0 | 23,665 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,104.6 | 69.6 | 7,646 ms | 14.4 ms | 16.0 | 0 | 33,139 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,518.4 | 49.2 | 14,280 ms | 20.3 ms | 32.0 | 0 | 45,522 | 30.0 s |
| Sustained decode | 512K | 64 | Valid | 2,171.9 | 34.7 | 28,387 ms | 28.8 ms | 64.0 | 1 | 65,104 | 30.0 s |
| Sustained decode | 512K | 128 | Valid | 2,646.3 | 22.3 | 52,092 ms | 44.8 ms | 127.8 | 6 | 79,425 | 30.0 s |
| Sustained decode | 1019K | 1 | Valid | 151.7 | 168.8 | 2,658 ms | 5.9 ms | 0.9 | 1 | 4,550 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 262.5 | 150.3 | 3,532 ms | 6.7 ms | 1.8 | 2 | 7,873 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 438.2 | 118.3 | 4,855 ms | 8.5 ms | 3.8 | 3 | 13,142 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 674.0 | 88.5 | 8,917 ms | 11.3 ms | 7.9 | 1 | 20,212 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 985.5 | 62.7 | 14,493 ms | 15.9 ms | 15.8 | 2 | 29,563 | 30.0 s |
| Sustained decode | 1019K | 32 | Valid | 1,268.1 | 41.1 | 27,664 ms | 24.3 ms | 31.6 | 4 | 38,002 | 30.0 s |
| Sustained decode | 1019K | 64 | Valid | 1,689.6 | 28.0 | 42,901 ms | 35.7 ms | 63.1 | 24 | 50,648 | 30.0 s |
| Sustained decode | 1019K | 128 | Requests queued | 1,669.0 | 25.4 | 105,464 ms | 39.4 ms | 75.1 | 31 | 50,057 | 30.0 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.