Speed test matrix
Fara1.5-4B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- fara
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-microsoft-vllm/bin/vllm" serve microsoft/Fara1.5-4B --served-model-name fara --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.882964 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --limit-mm-per-prompt '{"image":10}'Prefill
prompt tok/s by context length8K context48.3K170 ms TTFT
16K context48.5K335 ms TTFT
32K context45.9K704 ms TTFT
64K context41.2K1,562 ms TTFT
128K context32.3K3,980 ms TTFT
251K context22.6K11,166 ms TTFT
256K context22.2K11,566 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 237.870 ms TTFT | 465.796 ms TTFT | 887.3129 ms TTFT | 1,622.0166 ms TTFT | 2,732.7229 ms TTFT | 4,261.3199 ms TTFT | 5,991.8534 ms TTFT | 7,437.9830 ms TTFT |
| 16K | 234.2100 ms TTFT | 454.2129 ms TTFT | 847.2182 ms TTFT | 1,498.4187 ms TTFT | 2,412.6264 ms TTFT | 3,532.2502 ms TTFT | 4,550.7718 ms TTFT | 5,284.91,304 ms TTFT |
| 32K | 226.6116 ms TTFT | 432.0165 ms TTFT | 779.6222 ms TTFT | 1,306.1246 ms TTFT | 1,934.2264 ms TTFT | 2,581.9693 ms TTFT | 3,037.91,201 ms TTFT | Over capacity |
| 64K | 214.4170 ms TTFT | 394.1197 ms TTFT | 674.6197 ms TTFT | 1,027.3227 ms TTFT | 1,389.1774 ms TTFT | 1,633.11,469 ms TTFT | Over capacity | Over capacity |
| 128K | 192.9286 ms TTFT | 334.1354 ms TTFT | 515.4332 ms TTFT | 712.2868 ms TTFT | 847.71,493 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 160.1570 ms TTFT | 248.7635 ms TTFT | 353.51,132 ms TTFT | 432.71,785 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 48.3K | — | 170 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 48.5K | — | 335 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 45.9K | — | 704 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.2K | — | 1,562 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 32.3K | — | 3,980 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 22.6K | — | 11,166 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.2K | — | 11,566 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 237.8 | 238.5 | 70 ms | 4.2 ms | 1.0 | 1 | 7,132 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 465.7 | 233.9 | 96 ms | 4.3 ms | 2.0 | 2 | 13,978 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 887.3 | 223.0 | 129 ms | 4.5 ms | 4.0 | 4 | 26,617 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,622.0 | 204.1 | 166 ms | 4.9 ms | 8.0 | 8 | 48,661 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,732.7 | 173.2 | 229 ms | 5.8 ms | 16.0 | 16 | 81,983 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,261.3 | 135.3 | 199 ms | 7.4 ms | 31.4 | 32 | 127,843 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,991.8 | 94.4 | 534 ms | 10.6 ms | 64.0 | 0 | 179,776 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,437.9 | 58.2 | 830 ms | 17.2 ms | 128.0 | 0 | 223,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 234.2 | 235.2 | 100 ms | 4.3 ms | 1.0 | 1 | 7,025 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 454.2 | 228.5 | 129 ms | 4.4 ms | 2.0 | 2 | 13,624 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 847.2 | 213.5 | 182 ms | 4.7 ms | 4.0 | 4 | 25,416 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,498.4 | 188.9 | 187 ms | 5.3 ms | 8.0 | 8 | 44,951 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,412.6 | 152.8 | 264 ms | 6.5 ms | 16.0 | 16 | 72,375 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,532.2 | 111.0 | 502 ms | 9.0 ms | 32.0 | 0 | 105,952 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,550.7 | 71.1 | 718 ms | 14.1 ms | 64.0 | 0 | 136,512 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,284.9 | 41.0 | 1,304 ms | 24.4 ms | 128.0 | 0 | 158,464 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 226.6 | 227.9 | 116 ms | 4.4 ms | 1.0 | 1 | 6,798 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 432.0 | 217.5 | 165 ms | 4.6 ms | 2.0 | 2 | 12,958 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 779.6 | 196.7 | 222 ms | 5.1 ms | 4.0 | 4 | 23,388 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,306.1 | 165.2 | 246 ms | 6.1 ms | 8.0 | 8 | 39,178 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,934.2 | 122.2 | 264 ms | 8.2 ms | 15.8 | 16 | 58,020 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,581.9 | 81.0 | 693 ms | 12.3 ms | 32.0 | 0 | 77,440 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,037.9 | 47.7 | 1,201 ms | 21.0 ms | 64.0 | 0 | 91,136 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 214.4 | 216.0 | 170 ms | 4.6 ms | 1.0 | 1 | 6,431 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 394.1 | 198.7 | 197 ms | 5.0 ms | 2.0 | 2 | 11,825 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 674.6 | 170.2 | 197 ms | 5.9 ms | 4.0 | 4 | 20,237 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,027.3 | 129.6 | 227 ms | 7.7 ms | 8.0 | 8 | 30,820 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,389.1 | 87.0 | 774 ms | 11.5 ms | 16.0 | 0 | 41,664 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,633.1 | 51.2 | 1,469 ms | 19.5 ms | 32.0 | 0 | 48,992 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 192.9 | 195.3 | 286 ms | 5.1 ms | 1.0 | 1 | 5,785 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 334.1 | 169.5 | 354 ms | 5.9 ms | 2.0 | 2 | 10,029 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 515.4 | 130.5 | 332 ms | 7.7 ms | 3.9 | 4 | 15,459 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 712.2 | 89.1 | 868 ms | 11.2 ms | 8.0 | 0 | 21,360 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 847.7 | 53.0 | 1,493 ms | 18.9 ms | 16.0 | 0 | 25,424 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 160.1 | 164.3 | 570 ms | 6.1 ms | 1.0 | 1 | 4,803 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 248.7 | 132.1 | 635 ms | 7.6 ms | 2.0 | 2 | 7,464 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 353.5 | 88.7 | 1,132 ms | 11.3 ms | 4.0 | 0 | 10,604 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 432.7 | 54.3 | 1,785 ms | 18.4 ms | 8.0 | 0 | 12,976 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.