Speed test matrix
DeepSeek-V4.1-Flash on H200 SXM 141GB ×8
vllm · 0.1.1.dev5+ge77daef89 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --speculative-config '{"method":"dspark","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context18.3K448 ms TTFT
16K context20.1K808 ms TTFT
32K context20K1,613 ms TTFT
64K context19.6K3,288 ms TTFT
128K context18.5K6,961 ms TTFT
256K context16.3K15,756 ms TTFT
512K context13.1K39,177 ms TTFT
1019K context7,739132,058 ms TTFT
1024K context7,702133,347 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 256.750 ms TTFT | 445.373 ms TTFT | 674.568 ms TTFT | 1,081.984 ms TTFT | 1,489.5223 ms TTFT | 2,311.5361 ms TTFT | 2,980.0666 ms TTFT | 2,799.81,555 ms TTFT |
| 16K | 252.273 ms TTFT | 459.287 ms TTFT | 694.489 ms TTFT | 1,094.4104 ms TTFT | 1,511.5328 ms TTFT | 2,267.5587 ms TTFT | 2,962.11,108 ms TTFT | 2,812.22,087 ms TTFT |
| 32K | 234.8115 ms TTFT | 401.3122 ms TTFT | 685.4136 ms TTFT | 1,116.4158 ms TTFT | 1,502.3561 ms TTFT | 2,236.41,017 ms TTFT | 2,936.01,938 ms TTFT | 2,776.23,762 ms TTFT |
| 64K | 231.4199 ms TTFT | 417.7212 ms TTFT | 691.3214 ms TTFT | 1,114.4266 ms TTFT | Looped | 2,209.12,027 ms TTFT | 2,812.23,931 ms TTFT | 2,774.07,598 ms TTFT |
| 128K | 249.4379 ms TTFT | 413.7391 ms TTFT | 670.8392 ms TTFT | 1,038.51,081 ms TTFT | 1,440.92,254 ms TTFT | 2,126.14,256 ms TTFT | 2,717.68,190 ms TTFT | Over capacity |
| 256K | 229.9786 ms TTFT | 392.01,031 ms TTFT | 665.51,174 ms TTFT | 1,002.51,091 ms TTFT | 1,353.24,772 ms TTFT | 1,930.89,154 ms TTFT | Over capacity | Over capacity |
| 512K | 212.41,752 ms TTFT | 385.8Requests queued1,766 ms TTFT | 568.9Requests queued2,527 ms TTFT | 901.2Requests queued3,878 ms TTFT | 1,181.710,245 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 207.93,903 ms TTFT | 336.4Requests queued3,971 ms TTFT | 484.7Requests queued4,002 ms TTFT | 727.4Requests queued8,792 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.3K | — | 448 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.1K | — | 808 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20K | — | 1,613 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.6K | — | 3,288 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,961 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.3K | — | 15,756 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.1K | — | 39,177 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,739.0 | — | 132,058 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,702.0 | — | 133,347 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 256.7 | 244.1 | 50 ms | 4.1 ms | 1.0 | 2 | 7,699 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 445.3 | 225.4 | 73 ms | 4.4 ms | 2.0 | 2 | 13,356 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 674.5 | 172.4 | 68 ms | 5.8 ms | 4.0 | 4 | 20,236 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,081.9 | 134.9 | 84 ms | 7.4 ms | 8.0 | 8 | 32,456 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,489.5 | 93.6 | 223 ms | 10.7 ms | 16.0 | 0 | 44,671 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,311.5 | 72.5 | 361 ms | 13.8 ms | 32.0 | 0 | 69,295 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,980.0 | 46.8 | 666 ms | 21.4 ms | 64.0 | 0 | 89,409 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,799.8 | 20.8 | 1,555 ms | 48.0 ms | 128.0 | 0 | 83,729 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 252.2 | 258.0 | 73 ms | 3.9 ms | 1.0 | 2 | 7,564 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 459.2 | 229.1 | 87 ms | 4.4 ms | 2.0 | 2 | 13,772 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 694.4 | 167.9 | 89 ms | 6.0 ms | 4.0 | 4 | 20,832 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,094.4 | 140.7 | 104 ms | 7.1 ms | 8.0 | 8 | 32,828 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,511.5 | 95.0 | 328 ms | 10.5 ms | 16.0 | 0 | 45,342 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,267.5 | 71.6 | 587 ms | 14.0 ms | 32.0 | 0 | 68,015 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,962.1 | 46.7 | 1,108 ms | 21.4 ms | 64.0 | 0 | 88,821 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,812.2 | 21.9 | 2,087 ms | 45.7 ms | 128.0 | 0 | 84,184 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 234.8 | 236.2 | 115 ms | 4.2 ms | 1.0 | 1 | 7,042 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 401.3 | 201.7 | 122 ms | 5.0 ms | 2.0 | 2 | 12,036 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 685.4 | 173.6 | 136 ms | 5.8 ms | 4.0 | 4 | 20,559 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,116.4 | 135.6 | 158 ms | 7.4 ms | 8.0 | 8 | 33,477 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,502.3 | 95.0 | 561 ms | 10.5 ms | 16.0 | 0 | 45,037 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,236.4 | 69.9 | 1,017 ms | 14.3 ms | 32.0 | 0 | 67,071 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,936.0 | 46.6 | 1,938 ms | 21.4 ms | 64.0 | 0 | 87,952 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,776.2 | 22.7 | 3,762 ms | 44.1 ms | 128.0 | 0 | 83,155 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 231.4 | 231.9 | 199 ms | 4.3 ms | 1.0 | 1 | 6,941 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 417.7 | 204.4 | 212 ms | 4.9 ms | 2.0 | 2 | 12,529 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 691.3 | 173.3 | 214 ms | 5.8 ms | 4.0 | 4 | 20,734 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,114.4 | 135.7 | 266 ms | 7.4 ms | 8.0 | 8 | 33,426 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 2,209.1 | 69.4 | 2,027 ms | 14.4 ms | 32.0 | 0 | 66,252 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,812.2 | 45.3 | 3,931 ms | 22.1 ms | 64.0 | 0 | 84,322 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 2,774.0 | 22.9 | 7,598 ms | 43.8 ms | 128.0 | 0 | 83,159 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 249.4 | 261.1 | 379 ms | 3.8 ms | 1.0 | 2 | 7,481 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 413.7 | 209.5 | 391 ms | 4.8 ms | 2.0 | 2 | 12,410 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 670.8 | 165.3 | 392 ms | 6.0 ms | 4.0 | 4 | 20,125 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,038.5 | 135.6 | 1,081 ms | 7.4 ms | 7.8 | 8 | 31,148 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,440.9 | 91.5 | 2,254 ms | 10.9 ms | 16.0 | 0 | 43,209 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,126.1 | 67.3 | 4,256 ms | 14.9 ms | 32.0 | 0 | 63,744 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,717.6 | 44.0 | 8,190 ms | 22.7 ms | 64.0 | 0 | 81,474 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 229.9 | 236.7 | 786 ms | 4.2 ms | 1.0 | 1 | 6,897 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 392.0 | 207.8 | 1,031 ms | 4.8 ms | 1.9 | 2 | 11,760 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 665.5 | 169.2 | 1,174 ms | 5.9 ms | 3.9 | 4 | 19,961 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,002.5 | 128.7 | 1,091 ms | 7.8 ms | 7.8 | 7 | 30,075 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,353.2 | 85.5 | 4,772 ms | 11.7 ms | 16.0 | 0 | 40,568 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,930.8 | 61.3 | 9,154 ms | 16.3 ms | 32.0 | 0 | 57,877 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 212.4 | 223.5 | 1,752 ms | 4.5 ms | 1.0 | 1 | 6,371 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 385.8 | 203.6 | 1,766 ms | 4.9 ms | 1.9 | 2 | 11,572 | 30.0 s |
| Sustained decode | 512K | 4 | Requests queued | 568.9 | 154.5 | 2,527 ms | 6.5 ms | 3.7 | 4 | 17,058 | 30.0 s |
| Sustained decode | 512K | 8 | Requests queued | 901.2 | 121.3 | 3,878 ms | 8.2 ms | 7.6 | 6 | 27,028 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,181.7 | 77.5 | 10,245 ms | 12.9 ms | 15.8 | 4 | 35,391 | 29.9 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 207.9 | 237.6 | 3,903 ms | 4.2 ms | 0.9 | 1 | 6,237 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 336.4 | 194.7 | 3,971 ms | 5.1 ms | 1.7 | 2 | 10,088 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 484.7 | 142.8 | 4,002 ms | 7.0 ms | 3.6 | 4 | 14,535 | 30.0 s |
| Sustained decode | 1019K | 8 | Requests queued | 727.4 | 101.7 | 8,792 ms | 9.8 ms | 7.4 | 5 | 21,811 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.