Speed test matrix
DeepSeek-V4.1-Flash-1017-lossless-CSF on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · MXFP4+FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Quantization
- MXFP4+FP8
- Backend
- cuda
Run
- Served model
- local-inference-lab/DeepSeek-V4.1-Flash-1017-lossless-CSF
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Prefill
prompt tok/s by context length8K context17.5K467 ms TTFT
16K context17.9K904 ms TTFT
32K context18.3K1,763 ms TTFT
64K context17.8K3,617 ms TTFT
128K context17.3K7,431 ms TTFT
256K context16K16,093 ms TTFT
507K context13.4K38,095 ms TTFT
512K context13.3K38,548 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 162.8202 ms TTFT | 260.3218 ms TTFT | 402.9297 ms TTFT | 639.5415 ms TTFT | 1,040.4569 ms TTFT | 1,583.5594 ms TTFT | 1,464.3Requests queued1,702 ms TTFT | 1,475.5Requests queued1,763 ms TTFT |
| 16K | 153.7202 ms TTFT | 251.8232 ms TTFT | 397.9406 ms TTFT | 627.8447 ms TTFT | 1,037.2604 ms TTFT | 1,547.21,250 ms TTFT | 1,460.6Requests queued1,968 ms TTFT | 1,470.4Requests queued2,108 ms TTFT |
| 32K | 156.8211 ms TTFT | 264.9248 ms TTFT | 410.7341 ms TTFT | 624.9514 ms TTFT | 1,031.8714 ms TTFT | 1,574.0818 ms TTFT | 1,448.0Requests queued1,709 ms TTFT | 1,444.9Requests queued1,808 ms TTFT |
| 64K | 152.4228 ms TTFT | 258.8264 ms TTFT | 386.6519 ms TTFT | 625.9678 ms TTFT | 1,027.9863 ms TTFT | 1,537.71,612 ms TTFT | 1,504.9Requests queued42,732 ms TTFT | 1,496.6Requests queued40,751 ms TTFT |
| 128K | 154.9267 ms TTFT | 248.3378 ms TTFT | 384.9627 ms TTFT | 598.0929 ms TTFT | 1,002.91,616 ms TTFT | 1,517.21,898 ms TTFT | 1,389.4Requests queued88,301 ms TTFT | Over capacity |
| 256K | 153.8332 ms TTFT | 237.9428 ms TTFT | 375.4820 ms TTFT | 601.61,172 ms TTFT | 972.82,278 ms TTFT | 1,443.53,191 ms TTFT | Over capacity | Over capacity |
| 507K | 151.8451 ms TTFT | 243.6Requests queued609 ms TTFT | 359.71,117 ms TTFT | 566.81,853 ms TTFT | 907.12,974 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.5K | — | 467 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.9K | — | 904 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18.3K | — | 1,763 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.8K | — | 3,617 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.3K | — | 7,431 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16K | — | 16,093 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 13.4K | — | 38,095 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.3K | — | 38,548 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 162.8 | 163.7 | 202 ms | 6.1 ms | 1.0 | 1 | 4,883 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 260.3 | 131.2 | 218 ms | 7.6 ms | 2.0 | 2 | 7,807 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 402.9 | 101.1 | 297 ms | 9.9 ms | 4.0 | 0 | 12,087 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 639.5 | 81.5 | 415 ms | 12.3 ms | 8.0 | 0 | 19,184 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,040.4 | 66.1 | 569 ms | 15.1 ms | 16.0 | 0 | 31,192 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,583.5 | 50.2 | 594 ms | 19.9 ms | 32.0 | 0 | 47,485 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,464.3 | 46.7 | 1,702 ms | 21.4 ms | 32.0 | 25 | 43,932 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,475.5 | 47.0 | 1,763 ms | 21.3 ms | 31.9 | 28 | 44,151 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 153.7 | 155.7 | 202 ms | 6.4 ms | 1.0 | 1 | 4,609 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 251.8 | 127.7 | 232 ms | 7.8 ms | 2.0 | 2 | 7,554 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 397.9 | 101.0 | 406 ms | 9.9 ms | 4.0 | 0 | 11,928 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 627.8 | 79.2 | 447 ms | 12.6 ms | 8.0 | 0 | 18,819 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,037.2 | 65.4 | 604 ms | 15.3 ms | 16.0 | 0 | 31,106 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,547.2 | 48.9 | 1,250 ms | 20.5 ms | 32.0 | 0 | 46,361 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,460.6 | 46.3 | 1,968 ms | 21.6 ms | 32.0 | 24 | 43,807 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,470.4 | 45.9 | 2,108 ms | 21.8 ms | 31.9 | 22 | 44,091 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 156.8 | 159.1 | 211 ms | 6.3 ms | 1.0 | 1 | 4,705 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 264.9 | 127.8 | 248 ms | 7.8 ms | 2.0 | 2 | 7,944 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 410.7 | 103.3 | 341 ms | 9.7 ms | 4.0 | 0 | 12,313 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 624.9 | 79.0 | 514 ms | 12.7 ms | 8.0 | 0 | 18,728 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,031.8 | 65.9 | 714 ms | 15.2 ms | 16.0 | 0 | 30,939 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,574.0 | 50.1 | 818 ms | 20.0 ms | 32.0 | 0 | 47,153 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,448.0 | 46.4 | 1,709 ms | 21.6 ms | 31.9 | 22 | 43,404 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,444.9 | 46.5 | 1,808 ms | 21.5 ms | 31.9 | 25 | 43,278 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 152.4 | 154.0 | 228 ms | 6.5 ms | 1.0 | 1 | 4,570 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 258.8 | 132.2 | 264 ms | 7.6 ms | 2.0 | 2 | 7,763 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 386.6 | 97.7 | 519 ms | 10.2 ms | 4.0 | 0 | 11,590 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 625.9 | 79.3 | 678 ms | 12.6 ms | 8.0 | 0 | 18,768 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,027.9 | 64.7 | 863 ms | 15.5 ms | 16.0 | 0 | 30,820 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,537.7 | 48.8 | 1,612 ms | 20.5 ms | 32.0 | 0 | 46,127 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,504.9 | 46.9 | 42,732 ms | 21.3 ms | 32.0 | 32 | 45,101 | 30.0 s |
| Sustained decode | 64K | 128 | Requests queued | 1,496.6 | 46.7 | 40,751 ms | 21.4 ms | 32.0 | 32 | 44,869 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 154.9 | 158.4 | 267 ms | 6.3 ms | 1.0 | 1 | 4,644 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 248.3 | 132.0 | 378 ms | 7.6 ms | 2.0 | 1 | 7,448 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 384.9 | 96.8 | 627 ms | 10.3 ms | 4.0 | 0 | 11,540 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 598.0 | 76.0 | 929 ms | 13.2 ms | 8.0 | 0 | 17,924 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,002.9 | 63.5 | 1,616 ms | 15.8 ms | 16.0 | 0 | 30,069 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,517.2 | 48.0 | 1,898 ms | 20.8 ms | 32.0 | 0 | 45,481 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 1,389.4 | 44.9 | 88,301 ms | 22.3 ms | 32.0 | 64 | 41,656 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 153.8 | 160.1 | 332 ms | 6.3 ms | 1.0 | 1 | 4,614 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 237.9 | 125.3 | 428 ms | 8.0 ms | 2.0 | 1 | 7,132 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 375.4 | 94.8 | 820 ms | 10.6 ms | 4.0 | 0 | 11,256 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 601.6 | 75.9 | 1,172 ms | 13.2 ms | 8.0 | 0 | 18,044 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 972.8 | 61.8 | 2,278 ms | 16.2 ms | 16.0 | 0 | 29,149 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,443.5 | 45.9 | 3,191 ms | 21.8 ms | 32.0 | 0 | 43,267 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 151.8 | 151.6 | 451 ms | 6.6 ms | 1.0 | 1 | 4,555 | 30.0 s |
| Sustained decode | 507K | 2 | Requests queued | 243.6 | 127.7 | 609 ms | 7.8 ms | 2.0 | 1 | 7,307 | 30.0 s |
| Sustained decode | 507K | 4 | Valid | 359.7 | 91.5 | 1,117 ms | 10.9 ms | 4.0 | 0 | 10,784 | 30.0 s |
| Sustained decode | 507K | 8 | Valid | 566.8 | 72.3 | 1,853 ms | 13.8 ms | 8.0 | 0 | 17,003 | 30.0 s |
| Sustained decode | 507K | 16 | Valid | 907.1 | 57.6 | 2,974 ms | 17.4 ms | 16.0 | 0 | 27,212 | 30.0 s |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.