Speed test matrix
Ministral-3-14B-Base-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-14B-Base-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.872104 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context20.3K403 ms TTFT
16K context18.5K878 ms TTFT
32K context15.8K2,052 ms TTFT
64K context12.1K5,332 ms TTFT
128K context8,25115,649 ms TTFT
251K context5,01850,427 ms TTFT
256K context4,97551,868 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 112.230 ms TTFT | 219.048 ms TTFT | 417.685 ms TTFT | 754.4135 ms TTFT | 1,247.9189 ms TTFT | 1,905.1286 ms TTFT | Over capacity | Over capacity |
| 16K | 108.035 ms TTFT | 208.358 ms TTFT | 385.3114 ms TTFT | 657.5178 ms TTFT | 1,017.6245 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 100.859 ms TTFT | 189.085 ms TTFT | 332.9151 ms TTFT | 522.5262 ms TTFT | 722.2399 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 88.978 ms TTFT | 161.4133 ms TTFT | 260.0215 ms TTFT | 359.9325 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 72.2121 ms TTFT | 123.5232 ms TTFT | 174.2395 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 53.0261 ms TTFT | 82.6319 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.3K | — | 403 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.5K | — | 878 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.8K | — | 2,052 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.1K | — | 5,332 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,251.0 | — | 15,649 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,018.0 | — | 50,427 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,975.0 | — | 51,868 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 112.2 | 112.3 | 30 ms | 8.9 ms | 1.0 | 0 | 3,365 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 219.0 | 109.8 | 48 ms | 9.1 ms | 2.0 | 0 | 6,570 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 417.6 | 104.7 | 85 ms | 9.5 ms | 4.0 | 0 | 12,524 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 754.4 | 94.8 | 135 ms | 10.5 ms | 8.0 | 0 | 22,633 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,247.9 | 78.7 | 189 ms | 12.7 ms | 16.0 | 0 | 37,424 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,905.1 | 60.3 | 286 ms | 16.6 ms | 32.0 | 0 | 57,152 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 108.0 | 108.1 | 35 ms | 9.3 ms | 1.0 | 0 | 3,240 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 208.3 | 104.4 | 58 ms | 9.6 ms | 2.0 | 0 | 6,250 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 385.3 | 96.6 | 114 ms | 10.4 ms | 4.0 | 0 | 11,564 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 657.5 | 82.4 | 178 ms | 12.1 ms | 8.0 | 0 | 19,720 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,017.6 | 64.0 | 245 ms | 15.6 ms | 16.0 | 0 | 30,528 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 100.8 | 100.9 | 59 ms | 9.9 ms | 1.0 | 0 | 3,023 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 189.0 | 94.6 | 85 ms | 10.6 ms | 2.0 | 0 | 5,670 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 332.9 | 83.4 | 151 ms | 12.0 ms | 4.0 | 0 | 9,992 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 522.5 | 65.4 | 262 ms | 15.3 ms | 8.0 | 0 | 15,672 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 722.2 | 45.3 | 399 ms | 22.1 ms | 16.0 | 0 | 21,664 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 88.9 | 88.9 | 78 ms | 11.2 ms | 1.0 | 0 | 2,667 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 161.4 | 80.8 | 133 ms | 12.4 ms | 2.0 | 0 | 4,840 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 260.0 | 65.1 | 215 ms | 15.4 ms | 4.0 | 0 | 7,800 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 359.9 | 45.1 | 325 ms | 22.2 ms | 8.0 | 0 | 10,800 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 72.2 | 72.3 | 121 ms | 13.8 ms | 1.0 | 0 | 2,167 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 123.5 | 61.8 | 232 ms | 16.2 ms | 2.0 | 0 | 3,704 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 174.2 | 43.6 | 395 ms | 22.9 ms | 4.0 | 0 | 5,224 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 53.0 | 53.0 | 261 ms | 18.9 ms | 1.0 | 0 | 1,589 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 82.6 | 41.3 | 319 ms | 24.2 ms | 2.0 | 0 | 2,478 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.