Speed test matrix
Mistral-Medium-3.5-128B on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Mistral-Medium-3.5-128B --served-model-name mistral --trust-remote-code --tensor-parallel-size 2 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.835177 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral --speculative-config '{"model":"mistralai/Mistral-Medium-3.5-128B-EAGLE","num_speculative_tokens":3,"method":"eagle","max_model_len":65536}'Prefill
prompt tok/s by context length8K context4,6331,772 ms TTFT
16K context4,7553,424 ms TTFT
32K context4,3977,367 ms TTFT
64K context3,43418,824 ms TTFT
128K context2,34355,104 ms TTFT
251K context1,458173,574 ms TTFT
256K context1,437179,574 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 97.5363 ms TTFT | 205.2614 ms TTFT | 394.91,088 ms TTFT | 627.61,813 ms TTFT | 1,215.44,276 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 103.8387 ms TTFT | 192.9612 ms TTFT | 379.51,090 ms TTFT | 594.12,032 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 104.5433 ms TTFT | 189.6695 ms TTFT | 343.61,069 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 90.7427 ms TTFT | 169.0660 ms TTFT | 298.21,180 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 89.2481 ms TTFT | 144.8730 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 68.6Requests queued603 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,633.0 | — | 1,772 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,755.0 | — | 3,424 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,397.0 | — | 7,367 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,434.0 | — | 18,824 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,343.0 | — | 55,104 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,458.0 | — | 173,574 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,437.0 | — | 179,574 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 97.5 | 98.2 | 363 ms | 10.2 ms | 1.0 | 0 | 2,925 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 205.2 | 102.1 | 614 ms | 9.8 ms | 2.0 | 0 | 6,153 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 394.9 | 99.1 | 1,088 ms | 10.1 ms | 4.0 | 0 | 11,841 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 627.6 | 77.9 | 1,813 ms | 12.8 ms | 8.0 | 0 | 18,811 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,215.4 | 76.6 | 4,276 ms | 13.1 ms | 16.0 | 0 | 36,448 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 103.8 | 103.2 | 387 ms | 9.7 ms | 1.0 | 0 | 3,113 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 192.9 | 96.4 | 612 ms | 10.4 ms | 2.0 | 0 | 5,789 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 379.5 | 95.0 | 1,090 ms | 10.5 ms | 4.0 | 0 | 11,385 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 594.1 | 73.5 | 2,032 ms | 13.6 ms | 8.0 | 0 | 17,816 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 104.5 | 103.3 | 433 ms | 9.7 ms | 1.0 | 0 | 3,134 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 189.6 | 94.6 | 695 ms | 10.6 ms | 2.0 | 0 | 5,686 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 343.6 | 85.3 | 1,069 ms | 11.7 ms | 4.0 | 0 | 10,304 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 90.7 | 91.3 | 427 ms | 11.0 ms | 1.0 | 0 | 2,720 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 169.0 | 84.2 | 660 ms | 11.9 ms | 2.0 | 0 | 5,069 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 298.2 | 74.5 | 1,180 ms | 13.4 ms | 4.0 | 0 | 8,944 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 89.2 | 87.9 | 481 ms | 11.4 ms | 1.0 | 0 | 2,675 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 144.8 | 72.2 | 730 ms | 13.8 ms | 2.0 | 0 | 4,343 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 68.6 | 66.0 | 603 ms | 15.3 ms | 1.0 | 1 | 2,057 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.