Speed test matrix
Ministral-3-8B-Base-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-8B-Base-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.873164 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context30.6K268 ms TTFT
16K context27.2K599 ms TTFT
32K context22.2K1,456 ms TTFT
64K context16.5K3,922 ms TTFT
128K context10.7K12,062 ms TTFT
251K context6,26340,399 ms TTFT
256K context6,19941,629 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 164.639 ms TTFT | 321.147 ms TTFT | 606.581 ms TTFT | 1,063.889 ms TTFT | 1,663.0177 ms TTFT | 2,419.8286 ms TTFT | 3,161.1506 ms TTFT | Over capacity |
| 16K | 156.945 ms TTFT | 301.365 ms TTFT | 548.991 ms TTFT | 897.7143 ms TTFT | 1,327.9265 ms TTFT | 1,730.1415 ms TTFT | Over capacity | Over capacity |
| 32K | 144.267 ms TTFT | 266.097 ms TTFT | 459.3151 ms TTFT | 691.0214 ms TTFT | 914.3401 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 123.6100 ms TTFT | 221.2137 ms TTFT | 343.9237 ms TTFT | 457.9378 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 97.2152 ms TTFT | 162.8224 ms TTFT | 221.7308 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 68.6249 ms TTFT | 104.3328 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.6K | — | 268 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 27.2K | — | 599 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.2K | — | 1,456 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.5K | — | 3,922 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.7K | — | 12,062 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,263.0 | — | 40,399 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,199.0 | — | 41,629 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 164.6 | 165.3 | 39 ms | 6.0 ms | 1.0 | 1 | 4,938 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 321.1 | 161.8 | 47 ms | 6.2 ms | 2.0 | 2 | 9,634 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 606.5 | 153.5 | 81 ms | 6.5 ms | 4.0 | 4 | 18,200 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,063.8 | 136.4 | 89 ms | 7.3 ms | 8.0 | 8 | 31,914 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,663.0 | 105.5 | 177 ms | 9.5 ms | 16.0 | 0 | 49,888 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,419.8 | 77.0 | 286 ms | 13.0 ms | 32.0 | 0 | 72,576 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,161.1 | 50.2 | 506 ms | 19.9 ms | 64.0 | 0 | 94,848 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 156.9 | 157.7 | 45 ms | 6.3 ms | 1.0 | 1 | 4,707 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 301.3 | 152.0 | 65 ms | 6.6 ms | 2.0 | 2 | 9,044 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 548.9 | 138.9 | 91 ms | 7.2 ms | 4.0 | 4 | 16,466 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 897.7 | 112.9 | 143 ms | 8.9 ms | 8.0 | 0 | 26,928 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,327.9 | 83.8 | 265 ms | 11.9 ms | 16.0 | 0 | 39,824 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,730.1 | 54.6 | 415 ms | 18.3 ms | 32.0 | 0 | 51,905 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 144.2 | 145.1 | 67 ms | 6.9 ms | 1.0 | 1 | 4,325 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 266.0 | 134.7 | 97 ms | 7.4 ms | 2.0 | 2 | 7,980 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 459.3 | 115.1 | 151 ms | 8.7 ms | 4.0 | 0 | 13,780 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 691.0 | 86.7 | 214 ms | 11.5 ms | 8.0 | 0 | 20,728 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 914.3 | 57.5 | 401 ms | 17.4 ms | 16.0 | 0 | 27,424 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 123.6 | 125.0 | 100 ms | 8.0 ms | 1.0 | 1 | 3,707 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 221.2 | 110.7 | 137 ms | 9.0 ms | 2.0 | 0 | 6,636 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 343.9 | 86.2 | 237 ms | 11.6 ms | 4.0 | 0 | 10,320 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 457.9 | 57.4 | 378 ms | 17.4 ms | 8.0 | 0 | 13,744 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 97.2 | 97.3 | 152 ms | 10.3 ms | 1.0 | 0 | 2,917 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 162.8 | 81.5 | 224 ms | 12.3 ms | 2.0 | 0 | 4,884 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 221.7 | 55.4 | 308 ms | 18.0 ms | 4.0 | 0 | 6,648 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 68.6 | 68.6 | 249 ms | 14.6 ms | 1.0 | 0 | 2,058 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 104.3 | 52.2 | 328 ms | 19.2 ms | 2.0 | 0 | 3,128 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.