Speed test matrix
DeepSeek-V4-Flash-DSpark on H200 NVL 141GB ×2
vllm · 0.25.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.25.0
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-vllm/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash-DSpark --served-model-name deepseek --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.700892 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'Prefill
prompt tok/s by context length8K context11.3K726 ms TTFT
16K context11.6K1,402 ms TTFT
32K context11.4K2,834 ms TTFT
64K context11.6K5,580 ms TTFT
128K context11.2K11,505 ms TTFT
256K context10.2K25,181 ms TTFT
512K context8,53160,458 ms TTFT
1019K context6,296163,008 ms TTFT
1024K context6,256164,864 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 223.592 ms TTFT | 373.8119 ms TTFT | 854.7143 ms TTFT | 1,276.9157 ms TTFT | 1,540.5336 ms TTFT | 2,447.8614 ms TTFT | 3,747.81,102 ms TTFT | 4,047.82,023 ms TTFT |
| 16K | 381.9115 ms TTFT | 679.3139 ms TTFT | 583.2151 ms TTFT | 1,098.4244 ms TTFT | 1,729.7373 ms TTFT | 2,487.9660 ms TTFT | 3,518.11,069 ms TTFT | 3,784.11,600 ms TTFT |
| 32K | 383.7126 ms TTFT | 467.4187 ms TTFT | 817.3193 ms TTFT | 1,026.7281 ms TTFT | 1,799.2439 ms TTFT | 2,178.3971 ms TTFT | 3,465.21,479 ms TTFT | 3,980.72,697 ms TTFT |
| 64K | 269.9186 ms TTFT | 574.3274 ms TTFT | 719.4281 ms TTFT | 1,168.1331 ms TTFT | 1,615.2745 ms TTFT | 2,592.51,499 ms TTFT | 3,159.02,917 ms TTFT | 4,201.15,597 ms TTFT |
| 128K | 222.0366 ms TTFT | 486.2460 ms TTFT | 641.0513 ms TTFT | 957.1911 ms TTFT | 1,655.01,671 ms TTFT | 1,945.33,604 ms TTFT | 3,038.36,866 ms TTFT | 3,453.413,166 ms TTFT |
| 256K | 315.7616 ms TTFT | 527.91,021 ms TTFT | 660.4820 ms TTFT | 822.62,077 ms TTFT | 1,487.63,420 ms TTFT | 1,981.57,193 ms TTFT | 2,987.211,942 ms TTFT | 3,171.828,525 ms TTFT |
| 512K | 219.01,340 ms TTFT | 410.81,493 ms TTFT | 553.91,801 ms TTFT | 876.92,713 ms TTFT | 1,264.96,554 ms TTFT | 1,681.813,538 ms TTFT | 2,066.626,581 ms TTFT | 2,250.653,032 ms TTFT |
| 1019K | 180.52,670 ms TTFT | 320.2Requests queued2,954 ms TTFT | 441.24,799 ms TTFT | 679.68,529 ms TTFT | 759.215,807 ms TTFT | 999.630,677 ms TTFT | 1,108.555,517 ms TTFT | 1,179.4Requests queued94,031 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.3K | — | 726 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.6K | — | 1,402 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.4K | — | 2,834 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.6K | — | 5,580 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.2K | — | 11,505 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.2K | — | 25,181 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 8,531.0 | — | 60,458 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,296.0 | — | 163,008 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,256.0 | — | 164,864 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 223.5 | 222.3 | 92 ms | 4.5 ms | 1.0 | 1 | 6,702 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 373.8 | 146.9 | 119 ms | 6.8 ms | 2.0 | 3 | 11,210 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 854.7 | 209.7 | 143 ms | 4.8 ms | 4.0 | 4 | 25,636 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,276.9 | 141.9 | 157 ms | 7.1 ms | 8.0 | 9 | 38,284 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,540.5 | 81.2 | 336 ms | 12.3 ms | 16.0 | 4 | 46,213 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,447.8 | 64.1 | 614 ms | 15.6 ms | 32.0 | 1 | 73,422 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,747.8 | 47.8 | 1,102 ms | 20.9 ms | 64.0 | 0 | 112,385 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,047.8 | 26.2 | 2,023 ms | 38.2 ms | 128.0 | 0 | 121,237 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 381.9 | 363.4 | 115 ms | 2.9 ms | 1.0 | 3 | 11,456 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 679.3 | 345.2 | 139 ms | 2.9 ms | 2.0 | 5 | 20,371 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 583.2 | 143.7 | 151 ms | 7.0 ms | 4.0 | 4 | 17,495 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,098.4 | 113.9 | 244 ms | 8.8 ms | 8.0 | 4 | 32,933 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,729.7 | 97.7 | 373 ms | 10.2 ms | 16.0 | 4 | 51,870 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,487.9 | 69.8 | 660 ms | 14.3 ms | 32.0 | 0 | 74,645 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,518.1 | 48.6 | 1,069 ms | 20.6 ms | 64.0 | 0 | 105,367 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,784.1 | 26.1 | 1,600 ms | 38.3 ms | 128.0 | 0 | 113,489 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 383.7 | 366.7 | 126 ms | 2.7 ms | 1.0 | 3 | 11,510 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 467.4 | 217.9 | 187 ms | 4.6 ms | 2.0 | 3 | 14,015 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 817.3 | 194.3 | 193 ms | 5.2 ms | 4.0 | 4 | 24,504 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,026.7 | 119.3 | 281 ms | 8.4 ms | 8.0 | 4 | 30,786 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,799.2 | 91.5 | 439 ms | 10.9 ms | 16.0 | 7 | 53,949 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,178.3 | 56.3 | 971 ms | 17.8 ms | 32.0 | 1 | 65,289 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,465.2 | 45.8 | 1,479 ms | 21.8 ms | 64.0 | 0 | 103,802 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,980.7 | 26.4 | 2,697 ms | 37.8 ms | 128.0 | 0 | 119,259 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 269.9 | 226.2 | 186 ms | 4.4 ms | 1.0 | 2 | 8,088 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 574.3 | 331.6 | 274 ms | 3.0 ms | 2.0 | 3 | 17,227 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 719.4 | 151.5 | 281 ms | 6.6 ms | 3.9 | 4 | 21,569 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,168.1 | 140.1 | 331 ms | 7.1 ms | 8.0 | 5 | 35,045 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,615.2 | 84.0 | 745 ms | 11.9 ms | 16.0 | 4 | 48,459 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,592.5 | 69.0 | 1,499 ms | 14.5 ms | 32.0 | 2 | 77,760 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,159.0 | 40.3 | 2,917 ms | 24.8 ms | 64.0 | 0 | 94,718 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,201.1 | 28.3 | 5,597 ms | 35.3 ms | 128.0 | 0 | 126,007 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 222.0 | 223.8 | 366 ms | 4.5 ms | 1.0 | 1 | 6,659 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 486.2 | 239.8 | 460 ms | 4.2 ms | 1.9 | 2 | 14,581 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 641.0 | 173.7 | 513 ms | 5.8 ms | 4.0 | 4 | 19,230 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 957.1 | 113.4 | 911 ms | 8.8 ms | 8.0 | 3 | 28,697 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,655.0 | 88.5 | 1,671 ms | 11.3 ms | 15.9 | 5 | 49,596 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,945.3 | 52.9 | 3,604 ms | 18.9 ms | 32.0 | 1 | 58,259 | 29.9 s |
| Sustained decode | 128K | 64 | Valid | 3,038.3 | 39.7 | 6,866 ms | 25.2 ms | 64.0 | 2 | 91,069 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,453.4 | 22.5 | 13,166 ms | 44.5 ms | 128.0 | 0 | 103,611 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 315.7 | 353.2 | 616 ms | 2.8 ms | 1.0 | 2 | 9,470 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 527.9 | 227.0 | 1,021 ms | 4.4 ms | 1.9 | 4 | 15,830 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 660.4 | 140.6 | 820 ms | 7.1 ms | 3.9 | 4 | 19,809 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 822.6 | 100.5 | 2,077 ms | 10.0 ms | 8.0 | 2 | 24,659 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,487.6 | 88.6 | 3,420 ms | 11.3 ms | 15.9 | 4 | 44,567 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,981.5 | 50.7 | 7,193 ms | 19.7 ms | 31.9 | 3 | 59,354 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 2,987.2 | 40.0 | 11,942 ms | 25.0 ms | 63.9 | 3 | 89,441 | 29.9 s |
| Sustained decode | 256K | 128 | Valid | 3,171.8 | 23.2 | 28,525 ms | 43.1 ms | 127.9 | 4 | 95,157 | 30.0 s |
| Sustained decode | 512K | 1 | Valid | 219.0 | 228.6 | 1,340 ms | 4.4 ms | 1.0 | 1 | 6,567 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 410.8 | 242.4 | 1,493 ms | 4.5 ms | 1.9 | 2 | 12,323 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 553.9 | 132.0 | 1,801 ms | 7.6 ms | 3.9 | 3 | 16,615 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 876.9 | 96.8 | 2,713 ms | 10.5 ms | 7.9 | 4 | 26,297 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,264.9 | 65.9 | 6,554 ms | 15.2 ms | 15.9 | 3 | 37,945 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,681.8 | 42.8 | 13,538 ms | 23.4 ms | 31.9 | 2 | 50,439 | 30.0 s |
| Sustained decode | 512K | 64 | Valid | 2,066.6 | 30.8 | 26,581 ms | 32.5 ms | 63.9 | 1 | 61,979 | 30.0 s |
| Sustained decode | 512K | 128 | Valid | 2,250.6 | 18.2 | 53,032 ms | 54.9 ms | 127.4 | 15 | 67,033 | 29.8 s |
| Sustained decode | 1019K | 1 | Valid | 180.5 | 199.3 | 2,670 ms | 5.0 ms | 0.9 | 1 | 5,413 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 320.2 | 161.5 | 2,954 ms | 6.2 ms | 1.8 | 2 | 9,602 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 441.2 | 106.1 | 4,799 ms | 9.4 ms | 3.9 | 1 | 13,233 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 679.6 | 94.4 | 8,529 ms | 10.6 ms | 7.9 | 2 | 20,299 | 29.9 s |
| Sustained decode | 1019K | 16 | Valid | 759.2 | 47.8 | 15,807 ms | 20.9 ms | 15.9 | 1 | 22,763 | 30.0 s |
| Sustained decode | 1019K | 32 | Valid | 999.6 | 32.2 | 30,677 ms | 31.1 ms | 31.9 | 2 | 29,921 | 29.9 s |
| Sustained decode | 1019K | 64 | Valid | 1,108.5 | 18.8 | 55,517 ms | 53.3 ms | 63.6 | 11 | 33,195 | 29.9 s |
| Sustained decode | 1019K | 128 | Requests queued | 1,179.4 | 16.6 | 94,031 ms | 60.1 ms | 73.4 | 14 | 35,146 | 29.8 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.