Speed test matrix
Mistral-Medium-3.5-128B on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Mistral-Medium-3.5-128B --served-model-name mistral --trust-remote-code --tensor-parallel-size 2 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.855653 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral
Prefill
prompt tok/s by context length8K context4,5781,794 ms TTFT
16K context4,8373,366 ms TTFT
32K context4,4497,282 ms TTFT
64K context3,49818,478 ms TTFT
128K context2,39054,035 ms TTFT
251K context1,489169,946 ms TTFT
256K context1,470175,562 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 47.7361 ms TTFT | 95.0587 ms TTFT | 186.0984 ms TTFT | 356.71,760 ms TTFT | 647.54,284 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 46.6371 ms TTFT | 92.5616 ms TTFT | 178.41,123 ms TTFT | 330.81,927 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 45.0424 ms TTFT | 87.8658 ms TTFT | 165.51,130 ms TTFT | 287.62,174 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 42.2448 ms TTFT | 80.1673 ms TTFT | 144.01,186 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 37.8487 ms TTFT | 56.1672 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 31.4 | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,578.0 | — | 1,794 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,837.0 | — | 3,366 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,449.0 | — | 7,282 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,498.0 | — | 18,478 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,390.0 | — | 54,035 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,489.0 | — | 169,946 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,470.0 | — | 175,562 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 47.7 | 47.7 | 361 ms | 21.0 ms | 1.0 | 0 | 1,432 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 95.0 | 47.6 | 587 ms | 21.0 ms | 2.0 | 0 | 2,851 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 186.0 | 46.5 | 984 ms | 21.5 ms | 4.0 | 0 | 5,576 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 356.7 | 44.8 | 1,760 ms | 22.3 ms | 8.0 | 0 | 10,704 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 647.5 | 40.7 | 4,284 ms | 24.6 ms | 16.0 | 0 | 19,417 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 46.6 | 46.6 | 371 ms | 21.4 ms | 1.0 | 0 | 1,399 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 92.5 | 46.3 | 616 ms | 21.6 ms | 2.0 | 0 | 2,774 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 178.4 | 44.7 | 1,123 ms | 22.4 ms | 4.0 | 0 | 5,352 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 330.8 | 41.4 | 1,927 ms | 24.1 ms | 8.0 | 0 | 9,920 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 45.0 | 45.0 | 424 ms | 22.2 ms | 1.0 | 0 | 1,350 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 87.8 | 43.8 | 658 ms | 22.8 ms | 2.0 | 0 | 2,634 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 165.5 | 41.4 | 1,130 ms | 24.1 ms | 4.0 | 0 | 4,964 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 287.6 | 36.0 | 2,174 ms | 27.7 ms | 8.0 | 0 | 8,632 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 42.2 | 42.2 | 448 ms | 23.7 ms | 1.0 | 0 | 1,267 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 80.1 | 40.1 | 673 ms | 25.0 ms | 2.0 | 0 | 2,402 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 144.0 | 36.1 | 1,186 ms | 27.7 ms | 4.0 | 0 | 4,320 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 37.8 | 37.8 | 487 ms | 26.5 ms | 1.0 | 0 | 1,133 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 56.1 | 34.4 | 672 ms | 29.1 ms | 2.0 | 0 | 1,683 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 31.4 | — | — | — | 1.0 | 0 | 91 | 2.9 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.