Speed test matrix
Ministral-3-3B-Base-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-3B-Base-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.873914 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context58.9K139 ms TTFT
16K context50.6K322 ms TTFT
32K context38.4K843 ms TTFT
64K context26.3K2,454 ms TTFT
128K context15.9K8,135 ms TTFT
251K context8,79928,755 ms TTFT
256K context8,73329,551 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 288.332 ms TTFT | 554.748 ms TTFT | 1,029.274 ms TTFT | 1,746.1100 ms TTFT | 2,613.0122 ms TTFT | 3,402.9266 ms TTFT | 4,346.0541 ms TTFT | Over capacity |
| 16K | 270.757 ms TTFT | 509.866 ms TTFT | 906.6102 ms TTFT | 1,426.6120 ms TTFT | 1,910.1190 ms TTFT | 2,414.2389 ms TTFT | Over capacity | Over capacity |
| 32K | 242.065 ms TTFT | 438.474 ms TTFT | 730.794 ms TTFT | 1,024.7173 ms TTFT | 1,292.2381 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 201.094 ms TTFT | 347.2133 ms TTFT | 512.1226 ms TTFT | 653.7384 ms TTFT | 762.1686 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 149.2156 ms TTFT | 242.3240 ms TTFT | 315.5356 ms TTFT | 372.3645 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 100.0258 ms TTFT | 147.6312 ms TTFT | 179.4468 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 58.9K | — | 139 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 50.6K | — | 322 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 38.4K | — | 843 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.3K | — | 2,454 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.9K | — | 8,135 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,799.0 | — | 28,755 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,733.0 | — | 29,551 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 288.3 | 289.0 | 32 ms | 3.5 ms | 1.0 | 2 | 8,648 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 554.7 | 278.5 | 48 ms | 3.6 ms | 2.0 | 4 | 16,645 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,029.2 | 259.0 | 74 ms | 3.9 ms | 4.0 | 8 | 30,875 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,746.1 | 220.7 | 100 ms | 4.5 ms | 7.8 | 8 | 52,376 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,613.0 | 169.5 | 122 ms | 5.9 ms | 16.0 | 16 | 78,388 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,402.9 | 109.3 | 266 ms | 9.2 ms | 32.0 | 0 | 102,080 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,346.0 | 69.5 | 541 ms | 14.4 ms | 64.0 | 0 | 130,368 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 270.7 | 271.9 | 57 ms | 3.7 ms | 1.0 | 2 | 8,121 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 509.8 | 256.5 | 66 ms | 3.9 ms | 2.0 | 4 | 15,292 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 906.6 | 228.1 | 102 ms | 4.4 ms | 4.0 | 4 | 27,197 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,426.6 | 180.5 | 120 ms | 5.5 ms | 8.0 | 8 | 42,799 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,910.1 | 121.7 | 190 ms | 8.2 ms | 16.0 | 16 | 57,300 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,414.2 | 76.6 | 389 ms | 13.1 ms | 32.0 | 0 | 72,416 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 242.0 | 242.8 | 65 ms | 4.1 ms | 1.0 | 1 | 7,259 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 438.4 | 220.3 | 74 ms | 4.5 ms | 2.0 | 2 | 13,152 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 730.7 | 183.8 | 94 ms | 5.4 ms | 4.0 | 4 | 21,921 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,024.7 | 130.1 | 173 ms | 7.7 ms | 7.8 | 8 | 30,740 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,292.2 | 81.5 | 381 ms | 12.3 ms | 16.0 | 0 | 38,752 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 201.0 | 202.0 | 94 ms | 5.0 ms | 1.0 | 1 | 6,029 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 347.2 | 175.0 | 133 ms | 5.7 ms | 2.0 | 2 | 10,418 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 512.1 | 130.8 | 226 ms | 7.6 ms | 3.9 | 4 | 15,368 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 653.7 | 82.0 | 384 ms | 12.2 ms | 8.0 | 0 | 19,616 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 762.1 | 47.8 | 686 ms | 20.9 ms | 16.0 | 0 | 22,848 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 149.2 | 150.7 | 156 ms | 6.6 ms | 1.0 | 1 | 4,476 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 242.3 | 121.3 | 240 ms | 8.2 ms | 2.0 | 0 | 7,268 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 315.5 | 79.0 | 356 ms | 12.7 ms | 4.0 | 0 | 9,464 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 372.3 | 46.6 | 645 ms | 21.5 ms | 8.0 | 0 | 11,168 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 100.0 | 100.1 | 258 ms | 10.0 ms | 1.0 | 0 | 3,001 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 147.6 | 73.8 | 312 ms | 13.5 ms | 2.0 | 0 | 4,426 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 179.4 | 44.9 | 468 ms | 22.3 ms | 4.0 | 0 | 5,380 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.