Speed test matrix
Ministral-3-8B-Reasoning-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-8B-Reasoning-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.873164 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context14.8K556 ms TTFT
16K context18.4K885 ms TTFT
32K context18.8K1,725 ms TTFT
64K context15.3K4,216 ms TTFT
128K context10.4K12,406 ms TTFT
251K context6,20940,753 ms TTFT
256K context6,15141,951 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 162.6387 ms TTFT | 315.3544 ms TTFT | 566.4868 ms TTFT | 1,017.71,429 ms TTFT | 1,530.54,110 ms TTFT | 2,342.68,737 ms TTFT | 3,125.918,583 ms TTFT | Over capacity |
| 16K | 155.3385 ms TTFT | 295.5Requests queued585 ms TTFT | 540.6920 ms TTFT | 752.71,832 ms TTFT | 1,306.24,364 ms TTFT | 1,677.98,606 ms TTFT | Over capacity | Over capacity |
| 32K | 142.3406 ms TTFT | 261.4594 ms TTFT | 351.51,141 ms TTFT | 636.52,208 ms TTFT | 912.34,367 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 124.0448 ms TTFT | 211.4670 ms TTFT | 318.11,157 ms TTFT | 449.02,068 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 97.2523 ms TTFT | 81.3740 ms TTFT | 221.71,282 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 68.6591 ms TTFT | 104.3886 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.8K | — | 556 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.4K | — | 885 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18.8K | — | 1,725 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.3K | — | 4,216 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.4K | — | 12,406 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,209.0 | — | 40,753 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,151.0 | — | 41,951 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 162.6 | 165.3 | 387 ms | 6.1 ms | 1.0 | 1 | 4,877 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 315.3 | 161.7 | 544 ms | 6.2 ms | 2.0 | 2 | 9,458 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 566.4 | 152.3 | 868 ms | 6.6 ms | 3.9 | 4 | 16,999 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,017.7 | 133.6 | 1,429 ms | 7.5 ms | 7.9 | 8 | 30,527 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,530.5 | 107.2 | 4,110 ms | 9.3 ms | 16.0 | 0 | 45,913 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,342.6 | 77.9 | 8,737 ms | 12.8 ms | 32.0 | 0 | 70,260 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,125.9 | 53.2 | 18,583 ms | 18.8 ms | 64.0 | 0 | 93,760 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 155.3 | 157.7 | 385 ms | 6.3 ms | 1.0 | 1 | 4,658 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 295.5 | 151.9 | 585 ms | 6.6 ms | 1.9 | 2 | 8,864 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 540.6 | 139.7 | 920 ms | 7.2 ms | 3.9 | 4 | 16,214 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 752.7 | 113.4 | 1,832 ms | 8.8 ms | 8.0 | 0 | 22,579 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,306.2 | 84.3 | 4,364 ms | 11.9 ms | 16.0 | 0 | 39,180 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,677.9 | 54.8 | 8,606 ms | 18.3 ms | 32.0 | 0 | 50,330 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 142.3 | 145.0 | 406 ms | 6.9 ms | 1.0 | 1 | 4,270 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 261.4 | 134.6 | 594 ms | 7.4 ms | 2.0 | 2 | 7,842 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 351.5 | 115.7 | 1,141 ms | 8.6 ms | 4.0 | 0 | 10,546 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 636.5 | 86.9 | 2,208 ms | 11.5 ms | 8.0 | 0 | 19,091 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 912.3 | 57.7 | 4,367 ms | 17.3 ms | 16.0 | 0 | 27,360 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 124.0 | 124.1 | 448 ms | 8.1 ms | 1.0 | 1 | 3,707 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 211.4 | 110.7 | 670 ms | 9.0 ms | 2.0 | 0 | 6,346 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 318.1 | 86.4 | 1,157 ms | 11.6 ms | 4.0 | 0 | 9,542 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 449.0 | 57.3 | 2,068 ms | 17.5 ms | 8.0 | 0 | 13,469 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 97.2 | 97.4 | 523 ms | 10.3 ms | 1.0 | 0 | 2,703 | 27.8 s |
| Sustained decode | 128K | 2 | Valid | 81.3 | 81.1 | 740 ms | 12.3 ms | 2.0 | 0 | 2,438 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 221.7 | 55.5 | 1,282 ms | 18.0 ms | 4.0 | 0 | 6,648 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 68.6 | 68.7 | 591 ms | 14.6 ms | 1.0 | 0 | 2,058 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 104.3 | 52.2 | 886 ms | 19.2 ms | 2.0 | 0 | 3,130 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.