Speed test matrix
Ministral-3-3B-Reasoning-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-3B-Reasoning-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.873914 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context19K431 ms TTFT
16K context27K602 ms TTFT
32K context28.4K1,140 ms TTFT
64K context23.5K2,749 ms TTFT
128K context15.3K8,435 ms TTFT
251K context8,70229,078 ms TTFT
256K context8,64129,866 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 282.4378 ms TTFT | 534.5548 ms TTFT | 978.2691 ms TTFT | 1,604.11,599 ms TTFT | 2,435.12,917 ms TTFT | 2,741.78,777 ms TTFT | 3,953.318,540 ms TTFT | Over capacity |
| 16K | 264.3414 ms TTFT | 489.9Requests queued557 ms TTFT | 825.3861 ms TTFT | 1,380.11,241 ms TTFT | 1,489.23,156 ms TTFT | 2,309.09,102 ms TTFT | Over capacity | Over capacity |
| 32K | 239.1425 ms TTFT | 428.2Requests queued606 ms TTFT | 707.91,035 ms TTFT | 984.11,684 ms TTFT | 1,262.13,707 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 198.4448 ms TTFT | 340.3605 ms TTFT | 502.8858 ms TTFT | 653.92,391 ms TTFT | 760.24,664 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 147.7477 ms TTFT | 241.3797 ms TTFT | 315.11,400 ms TTFT | 372.32,654 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 100.1621 ms TTFT | 147.5854 ms TTFT | 179.61,513 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19K | — | 431 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 27K | — | 602 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.4K | — | 1,140 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 23.5K | — | 2,749 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.3K | — | 8,435 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,702.0 | — | 29,078 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,641.0 | — | 29,866 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 282.4 | 289.5 | 378 ms | 3.5 ms | 1.0 | 2 | 8,470 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 534.5 | 277.6 | 548 ms | 3.6 ms | 1.9 | 4 | 16,035 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 978.2 | 256.8 | 691 ms | 3.9 ms | 3.9 | 8 | 29,344 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,604.1 | 221.3 | 1,599 ms | 4.5 ms | 7.7 | 8 | 48,124 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,435.1 | 165.4 | 2,917 ms | 6.0 ms | 15.6 | 16 | 73,053 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,741.7 | 112.5 | 8,777 ms | 8.9 ms | 31.7 | 9 | 83,248 | 30.4 s |
| Sustained decode | 8K | 64 | Valid | 3,953.3 | 73.1 | 18,540 ms | 13.7 ms | 64.0 | 0 | 118,614 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 264.3 | 272.0 | 414 ms | 3.7 ms | 0.9 | 2 | 7,929 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 489.9 | 255.2 | 557 ms | 3.9 ms | 1.9 | 4 | 14,697 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 825.3 | 229.2 | 861 ms | 4.4 ms | 3.9 | 4 | 24,759 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,380.1 | 180.0 | 1,241 ms | 5.6 ms | 7.9 | 8 | 41,400 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,489.2 | 122.9 | 3,156 ms | 8.1 ms | 15.8 | 6 | 44,680 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,309.0 | 77.9 | 9,102 ms | 12.8 ms | 32.0 | 0 | 69,257 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 239.1 | 242.9 | 425 ms | 4.1 ms | 1.0 | 1 | 7,174 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 428.2 | 219.5 | 606 ms | 4.6 ms | 2.0 | 2 | 12,846 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 707.9 | 183.8 | 1,035 ms | 5.4 ms | 3.9 | 4 | 21,237 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 984.1 | 130.1 | 1,684 ms | 7.7 ms | 7.7 | 8 | 29,524 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,262.1 | 81.5 | 3,707 ms | 12.3 ms | 16.0 | 0 | 37,857 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 198.4 | 201.9 | 448 ms | 5.0 ms | 1.0 | 1 | 5,952 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 340.3 | 174.3 | 605 ms | 5.7 ms | 2.0 | 2 | 10,209 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 502.8 | 131.0 | 858 ms | 7.6 ms | 3.9 | 4 | 15,084 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 653.9 | 82.2 | 2,391 ms | 12.2 ms | 8.0 | 0 | 19,616 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 760.2 | 48.1 | 4,664 ms | 20.8 ms | 16.0 | 0 | 22,800 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 147.7 | 150.7 | 477 ms | 6.6 ms | 1.0 | 1 | 4,431 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 241.3 | 121.1 | 797 ms | 8.3 ms | 2.0 | 2 | 7,217 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 315.1 | 79.0 | 1,400 ms | 12.7 ms | 4.0 | 0 | 9,452 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 372.3 | 46.6 | 2,654 ms | 21.4 ms | 8.0 | 0 | 11,160 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 100.1 | 100.2 | 621 ms | 10.0 ms | 1.0 | 0 | 3,003 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 147.5 | 73.8 | 854 ms | 13.6 ms | 2.0 | 0 | 4,426 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 179.6 | 45.0 | 1,513 ms | 22.2 ms | 4.0 | 0 | 5,388 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.