Speed test matrix
Mistral-Small-4-119B-2603 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Mistral-Small-4-119B-2603 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.877289 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.8K490 ms TTFT
16K context21.4K762 ms TTFT
32K context21.6K1,499 ms TTFT
64K context17.3K3,726 ms TTFT
128K context11.2K11,542 ms TTFT
251K context6,50338,909 ms TTFT
256K context6,40140,318 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 178.2390 ms TTFT | 308.8Requests queued545 ms TTFT | 491.3786 ms TTFT | 646.32,135 ms TTFT | 1,058.04,051 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 177.5370 ms TTFT | 276.8Requests queued582 ms TTFT | 422.41,072 ms TTFT | 459.62,155 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 156.7390 ms TTFT | 259.5Requests queued600 ms TTFT | 138.31,153 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 141.6435 ms TTFT | 244.8634 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 107.8484 ms TTFT | 97.9742 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 76.6580 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.8K | — | 490 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.4K | — | 762 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.6K | — | 1,499 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.3K | — | 3,726 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.2K | — | 11,542 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,503.0 | — | 38,909 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,401.0 | — | 40,318 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 178.2 | 181.1 | 390 ms | 5.5 ms | 1.0 | 1 | 5,348 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 308.8 | 158.0 | 545 ms | 6.3 ms | 1.9 | 2 | 9,264 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 491.3 | 125.5 | 786 ms | 8.0 ms | 4.0 | 4 | 14,740 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 646.3 | 93.0 | 2,135 ms | 10.8 ms | 8.0 | 0 | 19,390 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,058.0 | 66.9 | 4,051 ms | 15.0 ms | 16.0 | 0 | 31,728 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 177.5 | 172.8 | 370 ms | 5.8 ms | 1.0 | 1 | 5,327 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 276.8 | 151.3 | 582 ms | 6.6 ms | 1.9 | 2 | 8,304 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 422.4 | 120.7 | 1,072 ms | 8.3 ms | 4.0 | 2 | 12,645 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 459.6 | 89.1 | 2,155 ms | 11.2 ms | 8.0 | 0 | 13,795 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 156.7 | 159.5 | 390 ms | 6.3 ms | 1.0 | 1 | 4,701 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 259.5 | 140.5 | 600 ms | 7.1 ms | 1.9 | 2 | 7,783 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 138.3 | 113.2 | 1,153 ms | 8.8 ms | 4.0 | 0 | 4,151 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 141.6 | 136.8 | 435 ms | 7.3 ms | 1.0 | 1 | 4,250 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 244.8 | 121.9 | 634 ms | 8.2 ms | 2.0 | 2 | 7,257 | 29.6 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 107.8 | 107.8 | 484 ms | 9.3 ms | 1.0 | 0 | 3,233 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 97.9 | 99.2 | 742 ms | 10.1 ms | 2.0 | 0 | 2,938 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 76.6 | 76.5 | 580 ms | 13.1 ms | 1.0 | 0 | 2,298 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.