Speed test matrix
DeepSeek-V4.1-Flash-NVFP4 on H200 SXM 141GB ×4
vllm · 0.1.1.dev5+ge77daef89 · NVFP4+MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- NVFP4+MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- nvidia/DeepSeek-V4.1-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-deepseek-v41-vllm-e77daef89/bin/vllm" serve nvidia/DeepSeek-V4.1-Flash-NVFP4 --served-model-name nvidia/DeepSeek-V4.1-Flash-NVFP4 --tensor-parallel-size 4 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true}' --enable-chunked-prefill --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --attention-backend FLASHMLA_SPARSE_DSV41 --linear-backend marlin --moe-backend marlin --no-enable-prefix-cachingPrefill
prompt tok/s by context length8K context15.8K517 ms TTFT
16K context16.7K968 ms TTFT
32K context16.9K1,906 ms TTFT
64K context16.6K3,866 ms TTFT
128K context15.9K8,101 ms TTFT
256K context14.2K18,069 ms TTFT
512K context11.8K43,708 ms TTFT
1019K context7,264140,706 ms TTFT
1024K context7,204142,567 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 125.5498 ms TTFT | 240.2741 ms TTFT | 439.61,206 ms TTFT | 772.22,131 ms TTFT | 1,262.04,453 ms TTFT | 1,994.78,173 ms TTFT | 2,927.515,635 ms TTFT | 4,472.3Requests queued30,420 ms TTFT |
| 16K | 125.2961 ms TTFT | 240.01,444 ms TTFT | 437.32,349 ms TTFT | 772.04,203 ms TTFT | 1,262.87,892 ms TTFT | 1,990.015,275 ms TTFT | 2,944.8Requests queued30,048 ms TTFT | 173.9Requests queued44,861 ms TTFT |
| 32K | 125.11,902 ms TTFT | 239.32,867 ms TTFT | 437.24,708 ms TTFT | 769.28,399 ms TTFT | 1,256.315,815 ms TTFT | 1,953.1Requests queued30,448 ms TTFT | 84.9Requests queued44,215 ms TTFT | 84.8Requests queued44,299 ms TTFT |
| 64K | 124.83,857 ms TTFT | 238.65,826 ms TTFT | Looped | 762.717,149 ms TTFT | 1,235.131,856 ms TTFT | 1,721.3Requests queued61,795 ms TTFT | 72.7Requests queued73,218 ms TTFT | 72.5Requests queued73,358 ms TTFT |
| 128K | 124.38,098 ms TTFT | 237.412,200 ms TTFT | 430.820,154 ms TTFT | 752.435,636 ms TTFT | 1,211.167,034 ms TTFT | 47.2Requests queued102,547 ms TTFT | 47.1Requests queued102,694 ms TTFT | Over capacity |
| 256K | 123.418,082 ms TTFT | 234.727,006 ms TTFT | 423.444,503 ms TTFT | 734.679,393 ms TTFT | 17.1Requests queued96,939 ms TTFT | 17.0Requests queued97,064 ms TTFT | Over capacity | Over capacity |
| 512K | 121.843,693 ms TTFT | 229.965,122 ms TTFT | 4.3Requests queued86,476 ms TTFT | 4.3Requests queued86,577 ms TTFT | 4.3Requests queued86,653 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.8K | — | 517 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16.7K | — | 968 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.9K | — | 1,906 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.6K | — | 3,866 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.9K | — | 8,101 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.2K | — | 18,069 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.8K | — | 43,708 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,264.0 | — | 140,706 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,204.0 | — | 142,567 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 125.5 | 125.5 | 498 ms | 8.0 ms | 1.0 | 1 | 3,711 | 29.6 s |
| Sustained decode | 8K | 2 | Valid | 240.2 | 119.4 | 741 ms | 8.4 ms | 2.0 | 0 | 7,207 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 439.6 | 107.9 | 1,206 ms | 9.3 ms | 4.0 | 0 | 13,188 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 772.2 | 92.5 | 2,131 ms | 10.8 ms | 8.0 | 0 | 23,160 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,262.0 | 72.8 | 4,453 ms | 13.7 ms | 16.0 | 0 | 37,857 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,994.7 | 52.7 | 8,173 ms | 19.0 ms | 32.0 | 0 | 59,842 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,927.5 | 33.1 | 15,635 ms | 30.2 ms | 64.0 | 0 | 87,813 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 4,472.3 | 18.8 | 30,420 ms | 53.2 ms | 128.0 | 0 | 134,142 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 125.2 | 125.2 | 961 ms | 8.0 ms | 1.0 | 1 | 3,645 | 29.1 s |
| Sustained decode | 16K | 2 | Valid | 240.0 | 118.5 | 1,444 ms | 8.4 ms | 2.0 | 0 | 7,198 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 437.3 | 105.3 | 2,349 ms | 9.5 ms | 4.0 | 0 | 13,117 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 772.0 | 88.6 | 4,203 ms | 11.3 ms | 8.0 | 0 | 23,161 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,262.8 | 66.2 | 7,892 ms | 15.1 ms | 16.0 | 0 | 37,889 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,990.0 | 44.8 | 15,275 ms | 22.3 ms | 32.0 | 0 | 59,681 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 2,944.8 | 24.8 | 30,048 ms | 40.4 ms | 64.0 | 0 | 88,320 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 173.9 | 2.2 | 44,861 ms | 462.5 ms | 81.0 | 0 | 5,212 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 125.1 | 125.1 | 1,902 ms | 8.0 ms | 1.0 | 1 | 3,636 | 29.1 s |
| Sustained decode | 32K | 2 | Valid | 239.3 | 116.7 | 2,867 ms | 8.6 ms | 2.0 | 0 | 7,178 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 437.2 | 101.5 | 4,708 ms | 9.9 ms | 4.0 | 0 | 13,116 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 769.2 | 81.4 | 8,399 ms | 12.3 ms | 8.0 | 0 | 23,072 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,256.3 | 56.9 | 15,815 ms | 17.6 ms | 16.0 | 0 | 37,679 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 1,953.1 | 32.7 | 30,448 ms | 30.6 ms | 32.0 | 0 | 58,588 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 84.9 | 2.2 | 44,215 ms | 465.0 ms | 40.4 | 0 | 2,522 | 29.7 s |
| Sustained decode | 32K | 128 | Requests queued | 84.8 | 2.2 | 44,299 ms | 465.0 ms | 40.4 | 0 | 2,522 | 29.8 s |
| Sustained decode | 64K | 1 | Valid | 124.8 | 124.9 | 3,857 ms | 8.0 ms | 1.0 | 1 | 3,626 | 29.0 s |
| Sustained decode | 64K | 2 | Valid | 238.6 | 113.7 | 5,826 ms | 8.8 ms | 2.0 | 0 | 7,156 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 762.7 | 69.7 | 17,149 ms | 14.4 ms | 8.0 | 0 | 22,882 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,235.1 | 43.3 | 31,856 ms | 23.1 ms | 16.0 | 0 | 37,055 | 30.0 s |
| Sustained decode | 64K | 32 | Requests queued | 1,721.3 | 20.5 | 61,795 ms | 48.7 ms | 32.0 | 0 | 51,641 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 72.7 | 2.1 | 73,218 ms | 475.5 ms | 35.5 | 0 | 2,176 | 29.9 s |
| Sustained decode | 64K | 128 | Requests queued | 72.5 | 2.1 | 73,358 ms | 475.7 ms | 35.5 | 0 | 2,174 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 124.3 | 124.4 | 8,098 ms | 8.0 ms | 1.0 | 1 | 3,650 | 29.4 s |
| Sustained decode | 128K | 2 | Valid | 237.4 | 108.0 | 12,200 ms | 9.4 ms | 2.0 | 0 | 7,120 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 430.8 | 81.6 | 20,154 ms | 12.3 ms | 4.0 | 0 | 12,924 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 752.4 | 53.2 | 35,636 ms | 18.9 ms | 8.0 | 0 | 22,568 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,211.1 | 29.0 | 67,034 ms | 34.6 ms | 16.0 | 0 | 36,336 | 30.0 s |
| Sustained decode | 128K | 32 | Requests queued | 47.2 | 2.0 | 102,547 ms | 499.5 ms | 24.4 | 0 | 1,403 | 29.7 s |
| Sustained decode | 128K | 64 | Requests queued | 47.1 | 2.0 | 102,694 ms | 499.3 ms | 24.3 | 0 | 1,396 | 29.7 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 123.4 | 123.7 | 18,082 ms | 8.1 ms | 1.0 | 1 | 3,632 | 29.4 s |
| Sustained decode | 256K | 2 | Valid | 234.7 | 98.0 | 27,006 ms | 10.6 ms | 2.0 | 0 | 7,040 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 423.4 | 61.6 | 44,503 ms | 16.6 ms | 4.0 | 0 | 12,701 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 734.6 | 33.8 | 79,393 ms | 29.8 ms | 8.0 | 0 | 22,033 | 30.0 s |
| Sustained decode | 256K | 16 | Requests queued | 17.1 | 1.8 | 96,939 ms | 555.9 ms | 10.5 | 0 | 508 | 29.8 s |
| Sustained decode | 256K | 32 | Requests queued | 17.0 | 1.8 | 97,064 ms | 555.9 ms | 10.5 | 0 | 506 | 29.8 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 121.8 | 122.3 | 43,693 ms | 8.2 ms | 1.0 | 0 | 3,654 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 229.9 | 83.6 | 65,122 ms | 14.0 ms | 2.0 | 0 | 6,896 | 30.0 s |
| Sustained decode | 512K | 4 | Requests queued | 4.3 | 1.5 | 86,476 ms | 664.0 ms | 4.0 | 0 | 129 | 29.9 s |
| Sustained decode | 512K | 8 | Requests queued | 4.3 | 1.5 | 86,577 ms | 664.3 ms | 4.0 | 0 | 129 | 30.0 s |
| Sustained decode | 512K | 16 | Requests queued | 4.3 | 1.5 | 86,653 ms | 663.6 ms | 4.0 | 0 | 129 | 29.9 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.