Speed test matrix
DeepSeek-V4.1-Flash-NVFP4 on H200 SXM 141GB ×8
vllm · 0.1.1.dev5+ge77daef89 · NVFP4+MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- NVFP4+MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- nvidia/DeepSeek-V4.1-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-NVFP4 --served-model-name nvidia/DeepSeek-V4.1-Flash-NVFP4 --tensor-parallel-size 8 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":false}' --enable-chunked-prefill --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --attention-backend FLASHMLA_SPARSE_DSV41 --linear-backend marlin --moe-backend marlin --no-enable-prefix-cachingPrefill
prompt tok/s by context length8K context20.2K406 ms TTFT
16K context21.4K757 ms TTFT
32K context21.6K1,496 ms TTFT
64K context21.1K3,049 ms TTFT
128K context19.8K6,489 ms TTFT
256K context17.3K14,833 ms TTFT
512K context13.7K37,469 ms TTFT
1019K context7,978128,108 ms TTFT
1024K context7,906129,912 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 130.0396 ms TTFT | 248.9575 ms TTFT | 472.1935 ms TTFT | 860.31,646 ms TTFT | 1,487.53,427 ms TTFT | 2,376.16,286 ms TTFT | 3,523.812,015 ms TTFT | 5,533.823,427 ms TTFT |
| 16K | 128.1750 ms TTFT | 247.8764 ms TTFT | 472.21,819 ms TTFT | 857.43,242 ms TTFT | 1,481.46,076 ms TTFT | 2,353.511,765 ms TTFT | 3,493.123,117 ms TTFT | 292.4Requests queued44,924 ms TTFT |
| 32K | 124.71,489 ms TTFT | 253.7Requests queued2,227 ms TTFT | 471.13,657 ms TTFT | 855.86,519 ms TTFT | 1,469.912,244 ms TTFT | 2,323.323,554 ms TTFT | 145.0Requests queued44,965 ms TTFT | 144.2Requests queued45,049 ms TTFT |
| 64K | 130.83,040 ms TTFT | 252.84,582 ms TTFT | 468.07,510 ms TTFT | 848.213,426 ms TTFT | 1,445.624,863 ms TTFT | 2,283.648,215 ms TTFT | 120.3Requests queued74,612 ms TTFT | 120.5Requests queued74,707 ms TTFT |
| 128K | 130.26,476 ms TTFT | 251.09,752 ms TTFT | 463.616,065 ms TTFT | 834.028,274 ms TTFT | 1,414.653,205 ms TTFT | 362.5Requests queued103,028 ms TTFT | 75.1Requests queued103,063 ms TTFT | Over capacity |
| 256K | 129.314,847 ms TTFT | 247.722,182 ms TTFT | 455.336,388 ms TTFT | 811.564,821 ms TTFT | 26.6Requests queued100,548 ms TTFT | 26.8Requests queued100,743 ms TTFT | Over capacity | Over capacity |
| 512K | Looped | 241.455,633 ms TTFT | 395.4Requests queued91,820 ms TTFT | 7.3Requests queued92,004 ms TTFT | 7.3Requests queued92,132 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.2K | — | 406 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.4K | — | 757 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.6K | — | 1,496 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.1K | — | 3,049 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.8K | — | 6,489 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.3K | — | 14,833 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.7K | — | 37,469 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,978.0 | — | 128,108 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,906.0 | — | 129,912 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 130.0 | 131.8 | 396 ms | 7.6 ms | 1.0 | 1 | 3,898 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 248.9 | 126.9 | 575 ms | 7.9 ms | 1.9 | 2 | 7,466 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 472.1 | 116.3 | 935 ms | 8.6 ms | 4.0 | 0 | 14,160 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 860.3 | 104.0 | 1,646 ms | 9.6 ms | 8.0 | 0 | 25,807 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,487.5 | 87.5 | 3,427 ms | 11.4 ms | 16.0 | 0 | 44,626 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,376.1 | 64.9 | 6,286 ms | 15.4 ms | 32.0 | 0 | 71,263 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,523.8 | 42.6 | 12,015 ms | 23.5 ms | 64.0 | 0 | 105,723 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,533.8 | 27.2 | 23,427 ms | 36.7 ms | 128.0 | 0 | 166,016 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 128.1 | 131.7 | 750 ms | 7.6 ms | 1.0 | 1 | 3,844 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.8 | 125.8 | 764 ms | 8.0 ms | 1.9 | 2 | 7,403 | 29.9 s |
| Sustained decode | 16K | 4 | Valid | 472.2 | 114.7 | 1,819 ms | 8.7 ms | 4.0 | 0 | 14,164 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 857.4 | 100.4 | 3,242 ms | 10.0 ms | 8.0 | 0 | 25,720 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,481.4 | 80.7 | 6,076 ms | 12.4 ms | 16.0 | 0 | 44,432 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,353.5 | 56.4 | 11,765 ms | 17.7 ms | 32.0 | 0 | 70,592 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,493.1 | 34.3 | 23,117 ms | 29.2 ms | 64.0 | 0 | 104,774 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 292.4 | 2.8 | 44,924 ms | 356.5 ms | 104.8 | 0 | 8,736 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 124.7 | 133.9 | 1,489 ms | 7.5 ms | 1.0 | 1 | 3,741 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 253.7 | 124.4 | 2,227 ms | 8.0 ms | 2.0 | 2 | 7,256 | 28.6 s |
| Sustained decode | 32K | 4 | Valid | 471.1 | 111.3 | 3,657 ms | 9.0 ms | 4.0 | 0 | 14,133 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 855.8 | 93.9 | 6,519 ms | 10.7 ms | 8.0 | 0 | 25,672 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,469.9 | 70.8 | 12,244 ms | 14.1 ms | 16.0 | 0 | 44,099 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,323.3 | 45.4 | 23,554 ms | 22.0 ms | 32.0 | 0 | 69,695 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 145.0 | 2.8 | 44,965 ms | 359.7 ms | 52.7 | 0 | 4,342 | 29.9 s |
| Sustained decode | 32K | 128 | Requests queued | 144.2 | 2.8 | 45,049 ms | 359.7 ms | 52.6 | 0 | 4,285 | 29.7 s |
| Sustained decode | 64K | 1 | Valid | 130.8 | 130.9 | 3,040 ms | 7.6 ms | 1.0 | 1 | 3,629 | 27.7 s |
| Sustained decode | 64K | 2 | Valid | 252.8 | 121.6 | 4,582 ms | 8.2 ms | 2.0 | 2 | 7,262 | 28.7 s |
| Sustained decode | 64K | 4 | Valid | 468.0 | 104.6 | 7,510 ms | 9.6 ms | 4.0 | 0 | 14,036 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 848.2 | 82.6 | 13,426 ms | 12.1 ms | 8.0 | 0 | 25,440 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,445.6 | 56.0 | 24,863 ms | 17.9 ms | 16.0 | 0 | 43,358 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,283.6 | 32.0 | 48,215 ms | 31.3 ms | 32.0 | 0 | 68,510 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 120.3 | 2.7 | 74,612 ms | 371.6 ms | 45.6 | 0 | 3,571 | 29.7 s |
| Sustained decode | 64K | 128 | Requests queued | 120.5 | 2.7 | 74,707 ms | 371.2 ms | 45.6 | 0 | 3,573 | 29.7 s |
| Sustained decode | 128K | 1 | Valid | 130.2 | 130.4 | 6,476 ms | 7.7 ms | 1.0 | 1 | 3,681 | 28.3 s |
| Sustained decode | 128K | 2 | Valid | 251.0 | 116.2 | 9,752 ms | 8.7 ms | 2.0 | 2 | 7,182 | 28.6 s |
| Sustained decode | 128K | 4 | Valid | 463.6 | 92.6 | 16,065 ms | 10.8 ms | 4.0 | 0 | 13,904 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 834.0 | 64.7 | 28,274 ms | 15.5 ms | 8.0 | 0 | 25,016 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,414.6 | 38.5 | 53,205 ms | 26.0 ms | 16.0 | 0 | 42,432 | 30.0 s |
| Sustained decode | 128K | 32 | Requests queued | 362.5 | 5.2 | 103,028 ms | 192.4 ms | 30.6 | 0 | 10,875 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 75.1 | 2.5 | 103,063 ms | 396.3 ms | 30.7 | 0 | 2,234 | 29.8 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 129.3 | 129.6 | 14,847 ms | 7.7 ms | 1.0 | 1 | 3,585 | 27.7 s |
| Sustained decode | 256K | 2 | Valid | 247.7 | 106.3 | 22,182 ms | 9.7 ms | 1.9 | 2 | 7,207 | 29.1 s |
| Sustained decode | 256K | 4 | Valid | 455.3 | 72.2 | 36,388 ms | 14.1 ms | 4.0 | 0 | 13,657 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 811.5 | 42.5 | 64,821 ms | 23.7 ms | 8.0 | 0 | 24,344 | 30.0 s |
| Sustained decode | 256K | 16 | Requests queued | 26.6 | 2.2 | 100,548 ms | 450.7 ms | 13.1 | 0 | 788 | 29.6 s |
| Sustained decode | 256K | 32 | Requests queued | 26.8 | 2.2 | 100,743 ms | 450.9 ms | 13.1 | 0 | 799 | 29.9 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 2 | Valid | 241.4 | 90.8 | 55,633 ms | 12.5 ms | 2.0 | 2 | 7,151 | 29.6 s |
| Sustained decode | 512K | 4 | Requests queued | 395.4 | 39.9 | 91,820 ms | 26.3 ms | 4.0 | 0 | 11,861 | 30.0 s |
| Sustained decode | 512K | 8 | Requests queued | 7.3 | 1.8 | 92,004 ms | 555.6 ms | 4.9 | 0 | 217 | 29.9 s |
| Sustained decode | 512K | 16 | Requests queued | 7.3 | 1.8 | 92,132 ms | 554.5 ms | 4.9 | 0 | 216 | 29.4 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.