Speed test matrix
DeepSeek-V4.1-Flash on H200 SXM 141GB ×8
vllm · 0.1.1.dev5+ge77daef89 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":false}' --speculative-config '{"method":"dspark","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context19.5K420 ms TTFT
16K context20.1K808 ms TTFT
32K context20.1K1,607 ms TTFT
64K context19.6K3,280 ms TTFT
128K context18.5K6,931 ms TTFT
256K context16.4K15,701 ms TTFT
512K context13.1K39,069 ms TTFT
1019K context7,738132,082 ms TTFT
1024K context7,672133,863 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 254.449 ms TTFT | 438.176 ms TTFT | 698.968 ms TTFT | 1,118.591 ms TTFT | 1,530.6226 ms TTFT | 2,320.3357 ms TTFT | 3,016.1676 ms TTFT | 2,804.41,569 ms TTFT |
| 16K | 247.272 ms TTFT | 425.784 ms TTFT | 686.2114 ms TTFT | 1,127.1105 ms TTFT | 1,549.4328 ms TTFT | 2,334.4587 ms TTFT | 2,994.61,066 ms TTFT | 2,793.62,281 ms TTFT |
| 32K | 239.6110 ms TTFT | 435.7121 ms TTFT | 692.8122 ms TTFT | 1,101.5146 ms TTFT | 1,463.3535 ms TTFT | 2,281.2994 ms TTFT | 2,904.91,907 ms TTFT | 2,775.43,697 ms TTFT |
| 64K | 227.2194 ms TTFT | 410.2213 ms TTFT | 662.8214 ms TTFT | 1,087.7270 ms TTFT | 1,452.61,047 ms TTFT | 2,231.92,028 ms TTFT | 2,860.14,016 ms TTFT | Over capacity |
| 128K | 241.0366 ms TTFT | 414.2392 ms TTFT | 699.1548 ms TTFT | 1,055.7615 ms TTFT | 1,499.02,274 ms TTFT | 2,138.44,318 ms TTFT | 2,695.38,188 ms TTFT | Over capacity |
| 256K | 234.1749 ms TTFT | 403.9788 ms TTFT | 649.2907 ms TTFT | 1,016.51,352 ms TTFT | 1,394.54,930 ms TTFT | 1,955.59,315 ms TTFT | Over capacity | Over capacity |
| 512K | 220.81,712 ms TTFT | 381.41,803 ms TTFT | 587.0Requests queued2,638 ms TTFT | 899.83,111 ms TTFT | 1,167.19,089 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 193.23,939 ms TTFT | 330.1Requests queued4,138 ms TTFT | 524.0Requests queued4,098 ms TTFT | 762.9Requests queued8,621 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.5K | — | 420 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.1K | — | 808 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.1K | — | 1,607 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.6K | — | 3,280 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,931 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.4K | — | 15,701 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.1K | — | 39,069 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,738.0 | — | 132,082 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,672.0 | — | 133,863 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 254.4 | 249.4 | 49 ms | 4.0 ms | 1.0 | 2 | 7,631 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 438.1 | 218.4 | 76 ms | 4.6 ms | 2.0 | 2 | 13,141 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 698.9 | 172.8 | 68 ms | 5.8 ms | 4.0 | 4 | 20,959 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,118.5 | 136.3 | 91 ms | 7.3 ms | 8.0 | 8 | 33,541 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,530.6 | 94.8 | 226 ms | 10.6 ms | 16.0 | 0 | 45,901 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,320.3 | 72.5 | 357 ms | 13.8 ms | 32.0 | 0 | 69,555 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,016.1 | 47.7 | 676 ms | 21.0 ms | 64.0 | 0 | 90,372 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,804.4 | 22.5 | 1,569 ms | 44.5 ms | 128.0 | 0 | 84,075 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 247.2 | 247.9 | 72 ms | 4.0 ms | 1.0 | 2 | 7,414 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 425.7 | 213.8 | 84 ms | 4.7 ms | 2.0 | 2 | 12,770 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 686.2 | 167.3 | 114 ms | 6.0 ms | 4.0 | 4 | 20,579 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,127.1 | 140.5 | 105 ms | 7.1 ms | 8.0 | 8 | 33,796 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,549.4 | 96.1 | 328 ms | 10.4 ms | 16.0 | 0 | 46,448 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,334.4 | 73.8 | 587 ms | 13.5 ms | 32.0 | 0 | 69,985 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,994.6 | 47.1 | 1,066 ms | 21.2 ms | 64.0 | 0 | 89,817 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,793.6 | 22.4 | 2,281 ms | 44.6 ms | 128.0 | 0 | 83,742 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 239.6 | 240.1 | 110 ms | 4.2 ms | 1.0 | 1 | 7,187 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 435.7 | 220.3 | 121 ms | 4.5 ms | 2.0 | 2 | 13,071 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 692.8 | 170.9 | 122 ms | 5.9 ms | 4.0 | 4 | 20,781 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,101.5 | 132.6 | 146 ms | 7.5 ms | 8.0 | 8 | 33,046 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,463.3 | 92.6 | 535 ms | 10.8 ms | 16.0 | 0 | 43,897 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,281.2 | 71.5 | 994 ms | 14.0 ms | 32.0 | 0 | 68,408 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,904.9 | 45.7 | 1,907 ms | 21.9 ms | 64.0 | 0 | 87,072 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,775.4 | 22.6 | 3,697 ms | 44.3 ms | 128.0 | 0 | 83,133 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 227.2 | 229.5 | 194 ms | 4.4 ms | 1.0 | 1 | 6,816 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 410.2 | 207.9 | 213 ms | 4.8 ms | 2.0 | 2 | 12,302 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 662.8 | 168.4 | 214 ms | 5.9 ms | 4.0 | 4 | 19,882 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,087.7 | 138.4 | 270 ms | 7.2 ms | 8.0 | 8 | 32,614 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,452.6 | 89.4 | 1,047 ms | 11.2 ms | 16.0 | 0 | 43,577 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,231.9 | 70.1 | 2,028 ms | 14.3 ms | 32.0 | 0 | 66,932 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,860.1 | 46.0 | 4,016 ms | 21.7 ms | 64.0 | 0 | 85,774 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 241.0 | 243.0 | 366 ms | 4.1 ms | 1.0 | 1 | 7,228 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 414.2 | 205.7 | 392 ms | 4.9 ms | 2.0 | 2 | 12,424 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 699.1 | 175.7 | 548 ms | 5.7 ms | 3.9 | 4 | 20,972 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,055.7 | 132.0 | 615 ms | 7.6 ms | 7.9 | 8 | 31,672 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,499.0 | 91.0 | 2,274 ms | 11.0 ms | 16.0 | 0 | 44,951 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,138.4 | 68.0 | 4,318 ms | 14.7 ms | 32.0 | 0 | 64,137 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,695.3 | 43.8 | 8,188 ms | 22.8 ms | 64.0 | 0 | 80,756 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 234.1 | 237.8 | 749 ms | 4.2 ms | 1.0 | 1 | 7,022 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 403.9 | 208.5 | 788 ms | 4.8 ms | 1.9 | 2 | 12,115 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 649.2 | 166.8 | 907 ms | 6.0 ms | 3.9 | 4 | 19,474 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,016.5 | 127.2 | 1,352 ms | 7.9 ms | 7.8 | 7 | 30,481 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,394.5 | 88.3 | 4,930 ms | 11.3 ms | 16.0 | 0 | 41,810 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,955.5 | 62.1 | 9,315 ms | 16.1 ms | 32.0 | 0 | 58,599 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 220.8 | 235.8 | 1,712 ms | 4.2 ms | 0.9 | 1 | 6,623 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 381.4 | 204.0 | 1,803 ms | 4.9 ms | 1.9 | 2 | 11,441 | 30.0 s |
| Sustained decode | 512K | 4 | Requests queued | 587.0 | 155.7 | 2,638 ms | 6.4 ms | 3.8 | 4 | 17,602 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 899.8 | 121.8 | 3,111 ms | 8.2 ms | 7.7 | 6 | 26,992 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,167.1 | 75.6 | 9,089 ms | 13.2 ms | 15.7 | 4 | 35,013 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 193.2 | 222.0 | 3,939 ms | 4.5 ms | 0.9 | 1 | 5,796 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 330.1 | 189.5 | 4,138 ms | 5.3 ms | 1.8 | 2 | 9,900 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 524.0 | 154.3 | 4,098 ms | 6.5 ms | 3.5 | 4 | 15,577 | 29.7 s |
| Sustained decode | 1019K | 8 | Requests queued | 762.9 | 101.9 | 8,621 ms | 9.8 ms | 7.4 | 6 | 22,882 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.