Speed test matrix
DeepSeek-V4.1-Flash on H200 SXM 141GB ×8
vllm · 0.1.1.dev5+ge77daef89 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":false}'Prefill
prompt tok/s by context length8K context18.7K437 ms TTFT
16K context20.5K790 ms TTFT
32K context20.6K1,568 ms TTFT
64K context20.2K3,193 ms TTFT
128K context19K6,759 ms TTFT
256K context16.7K15,340 ms TTFT
512K context13.4K38,366 ms TTFT
1019K context7,827130,577 ms TTFT
1024K context7,755132,438 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 130.950 ms TTFT | 249.858 ms TTFT | 464.995 ms TTFT | 840.0129 ms TTFT | 1,435.2222 ms TTFT | 2,272.0333 ms TTFT | 3,373.1597 ms TTFT | 5,277.21,148 ms TTFT |
| 16K | 130.664 ms TTFT | 248.988 ms TTFT | 463.8116 ms TTFT | 840.7188 ms TTFT | 1,433.4308 ms TTFT | 2,255.8571 ms TTFT | 3,361.2990 ms TTFT | 5,234.21,872 ms TTFT |
| 32K | 130.2107 ms TTFT | 248.1127 ms TTFT | 462.2201 ms TTFT | 836.3327 ms TTFT | 1,429.2547 ms TTFT | 2,239.9995 ms TTFT | 3,331.41,794 ms TTFT | 5,166.03,403 ms TTFT |
| 64K | 129.7184 ms TTFT | 246.9223 ms TTFT | 460.1363 ms TTFT | 835.0594 ms TTFT | 1,417.31,044 ms TTFT | 2,223.41,974 ms TTFT | 3,283.63,844 ms TTFT | 5,044.07,493 ms TTFT |
| 128K | 128.4354 ms TTFT | 244.1449 ms TTFT | 458.4749 ms TTFT | 828.11,263 ms TTFT | 1,394.42,281 ms TTFT | 2,173.44,316 ms TTFT | 3,176.88,326 ms TTFT | Over capacity |
| 256K | 126.0732 ms TTFT | 243.61,102 ms TTFT | 452.51,573 ms TTFT | 811.42,711 ms TTFT | 1,350.05,014 ms TTFT | 2,063.79,529 ms TTFT | Over capacity | Over capacity |
| 512K | 126.91,640 ms TTFT | 237.02,340 ms TTFT | 441.83,715 ms TTFT | 775.66,367 ms TTFT | 1,238.411,163 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 123.33,652 ms TTFT | 225.75,413 ms TTFT | 398.57,068 ms TTFT | 665.311,788 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.7K | — | 437 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.5K | — | 790 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.6K | — | 1,568 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.2K | — | 3,193 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19K | — | 6,759 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.7K | — | 15,340 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.4K | — | 38,366 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,827.0 | — | 130,577 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,755.0 | — | 132,438 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 130.9 | 131.2 | 50 ms | 7.6 ms | 1.0 | 1 | 3,926 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 249.8 | 125.1 | 58 ms | 8.0 ms | 1.9 | 2 | 7,495 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 464.9 | 116.3 | 95 ms | 8.6 ms | 4.0 | 0 | 13,948 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 840.0 | 105.2 | 129 ms | 9.5 ms | 8.0 | 0 | 25,200 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,435.2 | 89.9 | 222 ms | 11.1 ms | 16.0 | 0 | 43,056 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,272.0 | 71.2 | 333 ms | 14.0 ms | 32.0 | 0 | 68,160 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,373.1 | 52.8 | 597 ms | 18.9 ms | 64.0 | 0 | 101,186 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,277.2 | 41.1 | 1,148 ms | 24.3 ms | 128.0 | 0 | 158,336 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 130.6 | 130.9 | 64 ms | 7.6 ms | 1.0 | 1 | 3,917 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 248.9 | 124.8 | 88 ms | 8.0 ms | 2.0 | 2 | 7,467 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 463.8 | 116.0 | 116 ms | 8.6 ms | 4.0 | 0 | 13,912 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 840.7 | 105.2 | 188 ms | 9.5 ms | 8.0 | 0 | 25,216 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,433.4 | 89.8 | 308 ms | 11.1 ms | 16.0 | 0 | 42,992 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,255.8 | 70.6 | 571 ms | 14.2 ms | 32.0 | 0 | 67,648 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,361.2 | 52.5 | 990 ms | 19.0 ms | 64.0 | 0 | 100,797 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,234.2 | 41.0 | 1,872 ms | 24.4 ms | 128.0 | 0 | 157,055 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 130.2 | 131.1 | 107 ms | 7.6 ms | 1.0 | 1 | 3,906 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 248.1 | 124.6 | 127 ms | 8.0 ms | 2.0 | 2 | 7,442 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 462.2 | 115.6 | 201 ms | 8.7 ms | 4.0 | 0 | 13,864 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 836.3 | 104.6 | 327 ms | 9.6 ms | 8.0 | 0 | 25,090 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,429.2 | 89.3 | 547 ms | 11.2 ms | 16.0 | 0 | 42,865 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,239.9 | 70.1 | 995 ms | 14.3 ms | 32.0 | 0 | 67,200 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,331.4 | 52.1 | 1,794 ms | 19.2 ms | 64.0 | 0 | 99,900 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,166.0 | 40.2 | 3,403 ms | 24.9 ms | 128.0 | 0 | 155,009 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 129.7 | 131.3 | 184 ms | 7.6 ms | 1.0 | 1 | 3,890 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 246.9 | 125.0 | 223 ms | 8.0 ms | 2.0 | 2 | 7,404 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 460.1 | 115.1 | 363 ms | 8.7 ms | 4.0 | 0 | 13,800 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 835.0 | 104.4 | 594 ms | 9.6 ms | 8.0 | 0 | 25,049 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,417.3 | 88.5 | 1,044 ms | 11.3 ms | 16.0 | 0 | 42,513 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,223.4 | 69.5 | 1,974 ms | 14.4 ms | 32.0 | 0 | 66,687 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,283.6 | 51.4 | 3,844 ms | 19.5 ms | 64.0 | 0 | 98,495 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 5,044.0 | 39.4 | 7,493 ms | 25.4 ms | 128.0 | 0 | 151,301 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 128.4 | 132.1 | 354 ms | 7.6 ms | 1.0 | 1 | 3,853 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.1 | 126.2 | 449 ms | 7.9 ms | 1.9 | 2 | 7,323 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 458.4 | 114.7 | 749 ms | 8.7 ms | 4.0 | 0 | 13,749 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 828.1 | 103.4 | 1,263 ms | 9.7 ms | 8.0 | 0 | 24,840 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,394.4 | 87.0 | 2,281 ms | 11.5 ms | 16.0 | 0 | 41,834 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,173.4 | 67.8 | 4,316 ms | 14.7 ms | 32.0 | 0 | 65,185 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,176.8 | 49.9 | 8,326 ms | 20.1 ms | 64.0 | 0 | 95,295 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 126.0 | 132.9 | 732 ms | 7.5 ms | 1.0 | 1 | 3,780 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 243.6 | 122.7 | 1,102 ms | 8.1 ms | 2.0 | 1 | 7,309 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 452.5 | 113.1 | 1,573 ms | 8.8 ms | 4.0 | 0 | 13,573 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 811.4 | 101.2 | 2,711 ms | 9.9 ms | 8.0 | 0 | 24,338 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,350.0 | 84.1 | 5,014 ms | 11.9 ms | 16.0 | 0 | 40,496 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 2,063.7 | 64.5 | 9,529 ms | 15.5 ms | 32.0 | 0 | 61,888 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 126.9 | 127.5 | 1,640 ms | 7.8 ms | 1.0 | 1 | 3,691 | 29.1 s |
| Sustained decode | 512K | 2 | Valid | 237.0 | 120.6 | 2,340 ms | 8.3 ms | 2.0 | 1 | 7,110 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 441.8 | 110.5 | 3,715 ms | 9.0 ms | 4.0 | 0 | 13,255 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 775.6 | 97.3 | 6,367 ms | 10.3 ms | 8.0 | 1 | 23,268 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,238.4 | 77.8 | 11,163 ms | 12.9 ms | 15.9 | 2 | 37,147 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 123.3 | 124.4 | 3,652 ms | 8.0 ms | 1.0 | 1 | 3,687 | 29.9 s |
| Sustained decode | 1019K | 2 | Valid | 225.7 | 116.3 | 5,413 ms | 8.6 ms | 2.0 | 1 | 6,772 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 398.5 | 104.1 | 7,068 ms | 9.6 ms | 3.9 | 2 | 11,956 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 665.3 | 87.5 | 11,788 ms | 11.4 ms | 7.7 | 3 | 19,955 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.