Speed test matrix
Ministral-3-8B-Instruct-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-8B-Instruct-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.873694 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context22.4K366 ms TTFT
16K context26.2K620 ms TTFT
32K context24.7K1,309 ms TTFT
64K context18.6K3,475 ms TTFT
128K context11.7K11,020 ms TTFT
251K context6,62538,191 ms TTFT
256K context6,55439,372 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 236.4196 ms TTFT | 437.8308 ms TTFT | 823.7436 ms TTFT | 1,401.3525 ms TTFT | 2,017.11,066 ms TTFT | 2,397.83,784 ms TTFT | 3,440.27,577 ms TTFT | Over capacity |
| 16K | 221.3208 ms TTFT | 408.0326 ms TTFT | 731.0453 ms TTFT | 1,118.7765 ms TTFT | 1,234.81,913 ms TTFT | 1,898.54,065 ms TTFT | Over capacity | Over capacity |
| 32K | 197.2233 ms TTFT | 351.5363 ms TTFT | 579.2501 ms TTFT | 651.71,111 ms TTFT | 1,020.62,004 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 161.5260 ms TTFT | 272.5371 ms TTFT | 334.9640 ms TTFT | 514.11,340 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 119.6341 ms TTFT | 187.9513 ms TTFT | 255.8721 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 79.2439 ms TTFT | 118.6739 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 22.4K | — | 366 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.2K | — | 620 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.7K | — | 1,309 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.6K | — | 3,475 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.7K | — | 11,020 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,625.0 | — | 38,191 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,554.0 | — | 39,372 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 236.4 | 238.4 | 196 ms | 4.2 ms | 1.0 | 1 | 7,091 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 437.8 | 229.1 | 308 ms | 4.4 ms | 2.0 | 2 | 11,061 | 25.3 s |
| Sustained decode | 8K | 4 | Valid | 823.7 | 212.8 | 436 ms | 4.7 ms | 3.9 | 4 | 24,712 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,401.3 | 181.0 | 525 ms | 5.5 ms | 7.9 | 8 | 42,034 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,017.1 | 133.4 | 1,066 ms | 7.5 ms | 15.8 | 16 | 60,508 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,397.8 | 87.1 | 3,784 ms | 11.5 ms | 32.0 | 0 | 71,926 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,440.2 | 58.8 | 7,577 ms | 17.0 ms | 64.0 | 0 | 103,212 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 221.3 | 223.9 | 208 ms | 4.5 ms | 1.0 | 1 | 5,547 | 25.1 s |
| Sustained decode | 16K | 2 | Valid | 408.0 | 209.1 | 326 ms | 4.8 ms | 2.0 | 2 | 12,239 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 731.0 | 186.4 | 453 ms | 5.4 ms | 3.9 | 4 | 21,930 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,118.7 | 145.8 | 765 ms | 6.9 ms | 7.9 | 8 | 33,556 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,234.8 | 100.6 | 1,913 ms | 9.9 ms | 16.0 | 0 | 37,037 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,898.5 | 61.3 | 4,065 ms | 16.3 ms | 32.0 | 0 | 56,935 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 197.2 | 199.1 | 233 ms | 5.0 ms | 1.0 | 1 | 5,917 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 351.5 | 178.6 | 363 ms | 5.6 ms | 2.0 | 2 | 10,544 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 579.2 | 148.6 | 501 ms | 6.7 ms | 4.0 | 4 | 17,376 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 651.7 | 104.5 | 1,111 ms | 9.6 ms | 8.0 | 0 | 19,547 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,020.6 | 66.9 | 2,004 ms | 14.9 ms | 16.0 | 0 | 30,609 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 161.5 | 163.4 | 260 ms | 6.1 ms | 1.0 | 1 | 4,845 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 272.5 | 138.8 | 371 ms | 7.2 ms | 2.0 | 2 | 8,174 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 334.9 | 104.6 | 640 ms | 9.6 ms | 4.0 | 0 | 10,045 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 514.1 | 66.2 | 1,340 ms | 15.1 ms | 8.0 | 0 | 15,419 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 119.6 | 119.7 | 341 ms | 8.4 ms | 1.0 | 0 | 3,587 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 187.9 | 95.3 | 513 ms | 10.5 ms | 2.0 | 0 | 5,638 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 255.8 | 64.1 | 721 ms | 15.6 ms | 4.0 | 0 | 7,672 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 79.2 | 79.3 | 439 ms | 12.6 ms | 1.0 | 0 | 1,736 | 21.9 s |
| Sustained decode | 251K | 2 | Valid | 118.6 | 59.3 | 739 ms | 16.9 ms | 2.0 | 0 | 3,556 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.