Speed test matrix
Ministral-3-3B-Instruct-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-3B-Instruct-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.873048 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context29.7K276 ms TTFT
16K context38.5K422 ms TTFT
32K context36.7K883 ms TTFT
64K context27.6K2,341 ms TTFT
128K context16.6K7,791 ms TTFT
251K context9,05227,953 ms TTFT
256K context8,98128,733 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 382.2183 ms TTFT | 713.8Requests queued310 ms TTFT | 1,283.0322 ms TTFT | 2,102.0422 ms TTFT | 2,955.6867 ms TTFT | 3,347.83,535 ms TTFT | 4,312.97,962 ms TTFT | Over capacity |
| 16K | 354.1204 ms TTFT | 619.6329 ms TTFT | 1,113.0376 ms TTFT | 1,653.3640 ms TTFT | 2,135.51,061 ms TTFT | 2,543.14,513 ms TTFT | Over capacity | Over capacity |
| 32K | 308.9221 ms TTFT | 531.8331 ms TTFT | 795.3451 ms TTFT | 1,153.9634 ms TTFT | 1,374.52,341 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 246.9267 ms TTFT | 404.7351 ms TTFT | 588.3Requests queued580 ms TTFT | 703.21,313 ms TTFT | 840.12,314 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 176.4335 ms TTFT | 266.2490 ms TTFT | 352.0742 ms TTFT | 405.11,467 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 114.0440 ms TTFT | 162.4731 ms TTFT | 195.7988 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.7K | — | 276 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.5K | — | 422 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.7K | — | 883 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27.6K | — | 2,341 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.6K | — | 7,791 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,052.0 | — | 27,953 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,981.0 | — | 28,733 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 382.2 | 390.1 | 183 ms | 2.6 ms | 1.0 | 3 | 11,467 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 713.8 | 369.3 | 310 ms | 2.7 ms | 1.9 | 6 | 21,415 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,283.0 | 337.3 | 322 ms | 3.0 ms | 3.8 | 8 | 38,487 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,102.0 | 270.8 | 422 ms | 3.7 ms | 7.8 | 16 | 63,057 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,955.6 | 195.7 | 867 ms | 5.1 ms | 15.9 | 16 | 88,660 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,347.8 | 119.7 | 3,535 ms | 8.4 ms | 31.8 | 13 | 100,406 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,312.9 | 77.7 | 7,962 ms | 12.9 ms | 64.0 | 0 | 129,413 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 354.1 | 359.3 | 204 ms | 2.8 ms | 1.0 | 2 | 10,317 | 29.1 s |
| Sustained decode | 16K | 2 | Valid | 619.6 | 331.0 | 329 ms | 3.0 ms | 2.0 | 4 | 18,108 | 29.2 s |
| Sustained decode | 16K | 4 | Valid | 1,113.0 | 285.4 | 376 ms | 3.5 ms | 3.9 | 8 | 33,388 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,653.3 | 214.0 | 640 ms | 4.7 ms | 7.9 | 8 | 49,599 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,135.5 | 141.0 | 1,061 ms | 7.1 ms | 15.8 | 16 | 64,077 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,543.1 | 84.3 | 4,513 ms | 11.9 ms | 32.0 | 0 | 76,296 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 308.9 | 313.2 | 221 ms | 3.2 ms | 1.0 | 2 | 9,267 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 531.8 | 272.1 | 331 ms | 3.7 ms | 1.9 | 4 | 15,954 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 795.3 | 219.2 | 451 ms | 4.6 ms | 4.0 | 4 | 23,858 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,153.9 | 148.3 | 634 ms | 6.7 ms | 7.8 | 8 | 34,616 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,374.5 | 91.0 | 2,341 ms | 11.0 ms | 16.0 | 0 | 41,231 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 246.9 | 249.6 | 267 ms | 4.0 ms | 1.0 | 2 | 7,359 | 29.8 s |
| Sustained decode | 64K | 2 | Valid | 404.7 | 205.5 | 351 ms | 4.9 ms | 2.0 | 2 | 12,140 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 588.3 | 150.6 | 580 ms | 6.6 ms | 3.9 | 4 | 17,645 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 703.2 | 90.9 | 1,313 ms | 11.0 ms | 8.0 | 0 | 21,099 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 840.1 | 53.0 | 2,314 ms | 18.9 ms | 16.0 | 0 | 25,200 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 176.4 | 178.7 | 335 ms | 5.6 ms | 1.0 | 1 | 5,294 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 266.2 | 136.2 | 490 ms | 7.3 ms | 1.9 | 2 | 7,986 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 352.0 | 88.2 | 742 ms | 11.3 ms | 4.0 | 0 | 10,560 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 405.1 | 50.7 | 1,467 ms | 19.7 ms | 8.0 | 0 | 12,154 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 114.0 | 114.1 | 440 ms | 8.8 ms | 1.0 | 0 | 3,419 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 162.4 | 81.3 | 731 ms | 12.3 ms | 2.0 | 0 | 4,872 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 195.7 | 49.1 | 988 ms | 20.4 ms | 4.0 | 0 | 5,872 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.