Speed test matrix
Devstral-Small-2-24B-Instruct-2512 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- devstral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Devstral-Small-2-24B-Instruct-2512 --served-model-name devstral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.873026 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context12.8K639 ms TTFT
16K context14K1,164 ms TTFT
32K context13.3K2,439 ms TTFT
64K context10.9K5,909 ms TTFT
128K context7,74116,679 ms TTFT
251K context4,82052,492 ms TTFT
256K context4,79353,835 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 119.5198 ms TTFT | 221.2304 ms TTFT | 419.3569 ms TTFT | 758.11,092 ms TTFT | 1,291.82,207 ms TTFT | 1,750.24,399 ms TTFT | Over capacity | Over capacity |
| 16K | 114.9208 ms TTFT | 221.1354 ms TTFT | 402.0600 ms TTFT | 690.61,011 ms TTFT | 1,047.81,990 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 106.9247 ms TTFT | 199.3389 ms TTFT | 338.7600 ms TTFT | 544.41,033 ms TTFT | 761.02,017 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 93.7269 ms TTFT | 168.5392 ms TTFT | 272.9620 ms TTFT | 376.91,248 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 75.5334 ms TTFT | 127.4506 ms TTFT | 183.8715 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 54.8444 ms TTFT | 86.5604 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.8K | — | 639 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,164 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.3K | — | 2,439 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.9K | — | 5,909 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,741.0 | — | 16,679 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,820.0 | — | 52,492 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,793.0 | — | 53,835 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 119.5 | 119.6 | 198 ms | 8.4 ms | 1.0 | 0 | 3,586 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 221.2 | 117.1 | 304 ms | 8.5 ms | 2.0 | 0 | 6,637 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 419.3 | 111.3 | 569 ms | 9.0 ms | 4.0 | 0 | 12,578 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 758.1 | 100.7 | 1,092 ms | 9.9 ms | 8.0 | 0 | 22,752 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,291.8 | 82.8 | 2,207 ms | 12.1 ms | 16.0 | 0 | 38,751 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,750.2 | 56.1 | 4,399 ms | 17.8 ms | 32.0 | 0 | 52,512 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 114.9 | 115.1 | 208 ms | 8.7 ms | 1.0 | 0 | 3,449 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 221.1 | 110.8 | 354 ms | 9.0 ms | 2.0 | 0 | 6,632 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 402.0 | 102.0 | 600 ms | 9.8 ms | 4.0 | 0 | 12,060 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 690.6 | 86.7 | 1,011 ms | 11.5 ms | 8.0 | 0 | 20,712 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,047.8 | 66.9 | 1,990 ms | 15.0 ms | 16.0 | 0 | 31,423 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 106.9 | 107.0 | 247 ms | 9.3 ms | 1.0 | 0 | 2,875 | 26.9 s |
| Sustained decode | 32K | 2 | Valid | 199.3 | 99.9 | 389 ms | 10.0 ms | 2.0 | 0 | 5,981 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 338.7 | 87.6 | 600 ms | 11.4 ms | 4.0 | 0 | 10,160 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 544.4 | 68.2 | 1,033 ms | 14.7 ms | 8.0 | 0 | 16,328 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 761.0 | 47.9 | 2,017 ms | 20.9 ms | 16.0 | 0 | 22,832 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 93.7 | 93.8 | 269 ms | 10.7 ms | 1.0 | 0 | 2,813 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 168.5 | 84.3 | 392 ms | 11.9 ms | 2.0 | 0 | 5,052 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 272.9 | 68.4 | 620 ms | 14.6 ms | 4.0 | 0 | 8,188 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 376.9 | 47.3 | 1,248 ms | 21.2 ms | 8.0 | 0 | 11,304 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 75.5 | 75.5 | 334 ms | 13.2 ms | 1.0 | 0 | 2,264 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 127.4 | 63.7 | 506 ms | 15.7 ms | 2.0 | 0 | 3,822 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 183.8 | 46.0 | 715 ms | 21.7 ms | 4.0 | 0 | 5,512 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 54.8 | 54.8 | 444 ms | 18.2 ms | 1.0 | 0 | 1,644 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 86.5 | 43.3 | 604 ms | 23.1 ms | 2.0 | 0 | 2,594 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.