Speed test matrix
Mistral-Small-4-119B-2603-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- mistral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Mistral-Small-4-119B-2603-NVFP4 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --reasoning-parser mistral --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 16384 --max-num-seqs 128 --attention-backend TRITON_MLA
Prefill
prompt tok/s by context length8K context11.2K734 ms TTFT
16K context14K1,166 ms TTFT
32K context14.2K2,280 ms TTFT
64K context12.1K5,343 ms TTFT
128K context8,76114,738 ms TTFT
251K context5,65044,783 ms TTFT
256K context5,60446,052 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 122.4388 ms TTFT | 143.0736 ms TTFT | 236.41,078 ms TTFT | 402.02,105 ms TTFT | 540.44,194 ms TTFT | 785.88,565 ms TTFT | 1,085.517,942 ms TTFT | 1,290.927,652 ms TTFT |
| 16K | 120.0376 ms TTFT | 104.9747 ms TTFT | 297.71,088 ms TTFT | 312.32,195 ms TTFT | 583.54,391 ms TTFT | 660.38,781 ms TTFT | 957.413,817 ms TTFT | Over capacity |
| 32K | 115.2403 ms TTFT | 96.2727 ms TTFT | 172.61,077 ms TTFT | 313.42,153 ms TTFT | 426.34,491 ms TTFT | 584.07,904 ms TTFT | Over capacity | Over capacity |
| 64K | 108.6426 ms TTFT | 164.3843 ms TTFT | 238.81,204 ms TTFT | 323.42,337 ms TTFT | 377.84,536 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 95.1528 ms TTFT | 136.0939 ms TTFT | 134.71,442 ms TTFT | 205.92,667 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 77.1712 ms TTFT | 79.2956 ms TTFT | 97.81,720 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.2K | — | 734 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,166 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.2K | — | 2,280 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.1K | — | 5,343 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,761.0 | — | 14,738 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,650.0 | — | 44,783 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,604.0 | — | 46,052 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 122.4 | 122.5 | 388 ms | 8.2 ms | 1.0 | 0 | 3,670 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 143.0 | 106.2 | 736 ms | 9.4 ms | 2.0 | 0 | 4,290 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 236.4 | 79.3 | 1,078 ms | 12.6 ms | 4.0 | 0 | 7,095 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 402.0 | 57.8 | 2,105 ms | 17.3 ms | 8.0 | 0 | 12,054 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 540.4 | 42.1 | 4,194 ms | 23.7 ms | 16.0 | 0 | 16,207 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 785.8 | 30.0 | 8,565 ms | 33.3 ms | 32.0 | 0 | 23,575 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,085.5 | 20.5 | 17,942 ms | 48.8 ms | 64.0 | 0 | 32,560 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,290.9 | 13.3 | 27,652 ms | 75.3 ms | 128.0 | 0 | 38,700 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 120.0 | 120.5 | 376 ms | 8.3 ms | 1.0 | 0 | 1,058 | 8.8 s |
| Sustained decode | 16K | 2 | Valid | 104.9 | 104.5 | 747 ms | 9.6 ms | 2.0 | 0 | 3,147 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 297.7 | 75.1 | 1,088 ms | 13.3 ms | 4.0 | 0 | 8,932 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 312.3 | 55.2 | 2,195 ms | 18.1 ms | 8.0 | 0 | 9,370 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 583.5 | 39.7 | 4,391 ms | 25.2 ms | 16.0 | 0 | 17,490 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 660.3 | 27.8 | 8,781 ms | 35.9 ms | 32.0 | 0 | 19,805 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 957.4 | 19.0 | 13,817 ms | 52.5 ms | 64.0 | 0 | 28,678 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 115.2 | 115.4 | 403 ms | 8.7 ms | 1.0 | 0 | 3,457 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 96.2 | 97.5 | 727 ms | 10.3 ms | 2.0 | 0 | 2,887 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 172.6 | 69.6 | 1,077 ms | 14.4 ms | 4.0 | 0 | 5,179 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 313.4 | 50.0 | 2,153 ms | 20.0 ms | 8.0 | 0 | 9,404 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 426.3 | 34.6 | 4,491 ms | 28.9 ms | 16.0 | 0 | 12,781 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 584.0 | 22.9 | 7,904 ms | 43.7 ms | 32.0 | 0 | 17,509 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 108.6 | 108.6 | 426 ms | 9.2 ms | 1.0 | 0 | 1,918 | 17.7 s |
| Sustained decode | 64K | 2 | Valid | 164.3 | 81.8 | 843 ms | 12.2 ms | 2.0 | 0 | 4,932 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 238.8 | 59.9 | 1,204 ms | 16.7 ms | 4.0 | 0 | 7,168 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 323.4 | 40.6 | 2,337 ms | 24.6 ms | 8.0 | 0 | 9,704 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 377.8 | 26.3 | 4,536 ms | 38.0 ms | 16.0 | 0 | 11,325 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 95.1 | 95.1 | 528 ms | 10.5 ms | 1.0 | 0 | 1,525 | 16.0 s |
| Sustained decode | 128K | 2 | Valid | 136.0 | 67.6 | 939 ms | 14.8 ms | 2.0 | 0 | 4,080 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 134.7 | 46.4 | 1,442 ms | 21.5 ms | 4.0 | 0 | 4,042 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 205.9 | 29.4 | 2,667 ms | 34.0 ms | 8.0 | 0 | 6,174 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 77.1 | 77.4 | 712 ms | 12.9 ms | 1.0 | 0 | 605 | 7.8 s |
| Sustained decode | 251K | 2 | Valid | 79.2 | 51.2 | 956 ms | 19.5 ms | 2.0 | 0 | 2,376 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 97.8 | 32.5 | 1,720 ms | 30.8 ms | 4.0 | 0 | 2,934 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.