Speed test matrix
Fara1.5-9B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- fara
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-microsoft-vllm/bin/vllm" serve microsoft/Fara1.5-9B --served-model-name fara --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.882500 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --limit-mm-per-prompt '{"image":10}'Prefill
prompt tok/s by context length8K context32.4K253 ms TTFT
16K context31.3K519 ms TTFT
32K context30.4K1,063 ms TTFT
64K context28.4K2,266 ms TTFT
128K context24K5,371 ms TTFT
251K context18.1K13,929 ms TTFT
256K context18K14,319 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 160.679 ms TTFT | 317.1115 ms TTFT | 612.4907 ms TTFT | 1,146.2150 ms TTFT | 2,083.0225 ms TTFT | 3,451.3435 ms TTFT | 5,085.9600 ms TTFT | 6,566.41,020 ms TTFT |
| 16K | 158.7101 ms TTFT | 312.0123 ms TTFT | 593.2164 ms TTFT | 1,081.0218 ms TTFT | 1,898.8299 ms TTFT | 2,946.8480 ms TTFT | 3,976.1784 ms TTFT | 4,804.71,416 ms TTFT |
| 32K | 155.2117 ms TTFT | 301.8134 ms TTFT | 558.8193 ms TTFT | 977.1181 ms TTFT | 1,601.0458 ms TTFT | 2,241.7682 ms TTFT | 2,761.91,176 ms TTFT | Over capacity |
| 64K | 149.1169 ms TTFT | 281.7229 ms TTFT | 501.4211 ms TTFT | 821.4519 ms TTFT | 1,187.3822 ms TTFT | 1,491.91,395 ms TTFT | Over capacity | Over capacity |
| 128K | 138.3287 ms TTFT | 249.6323 ms TTFT | 415.1551 ms TTFT | 602.6938 ms TTFT | 765.91,579 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 122.3568 ms TTFT | 203.0843 ms TTFT | 301.51,092 ms TTFT | 388.91,896 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.4K | — | 253 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.3K | — | 519 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.4K | — | 1,063 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,266 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24K | — | 5,371 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.1K | — | 13,929 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18K | — | 14,319 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 160.6 | 161.3 | 79 ms | 6.2 ms | 1.0 | 1 | 4,819 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 317.1 | 159.4 | 115 ms | 6.3 ms | 1.9 | 2 | 9,512 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 612.4 | 154.2 | 907 ms | 6.5 ms | 4.0 | 4 | 18,380 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,146.2 | 144.4 | 150 ms | 6.9 ms | 8.0 | 8 | 34,387 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,083.0 | 131.4 | 225 ms | 7.6 ms | 16.0 | 16 | 62,494 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,451.3 | 108.6 | 435 ms | 9.2 ms | 32.0 | 0 | 103,520 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,085.9 | 79.9 | 600 ms | 12.5 ms | 64.0 | 0 | 152,576 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,566.4 | 51.2 | 1,020 ms | 19.5 ms | 128.0 | 0 | 196,994 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 158.7 | 159.5 | 101 ms | 6.3 ms | 1.0 | 1 | 4,760 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 312.0 | 157.0 | 123 ms | 6.4 ms | 2.0 | 2 | 9,360 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 593.2 | 149.7 | 164 ms | 6.7 ms | 4.0 | 4 | 17,802 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,081.0 | 136.6 | 218 ms | 7.3 ms | 8.0 | 8 | 32,430 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,898.8 | 118.9 | 299 ms | 8.4 ms | 16.0 | 0 | 56,960 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,946.8 | 92.2 | 480 ms | 10.8 ms | 32.0 | 0 | 88,384 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,976.1 | 61.9 | 784 ms | 16.2 ms | 64.0 | 0 | 119,296 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,804.7 | 37.0 | 1,416 ms | 27.0 ms | 128.0 | 0 | 144,128 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.2 | 156.2 | 117 ms | 6.4 ms | 1.0 | 1 | 4,656 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 301.8 | 152.0 | 134 ms | 6.6 ms | 2.0 | 2 | 9,054 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 558.8 | 140.7 | 193 ms | 7.1 ms | 4.0 | 4 | 16,765 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 977.1 | 123.0 | 181 ms | 8.1 ms | 8.0 | 8 | 29,309 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,601.0 | 100.3 | 458 ms | 10.0 ms | 16.0 | 0 | 48,032 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,241.7 | 70.3 | 682 ms | 14.2 ms | 32.0 | 0 | 67,232 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,761.9 | 43.3 | 1,176 ms | 23.1 ms | 64.0 | 0 | 82,816 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 149.1 | 150.6 | 169 ms | 6.6 ms | 1.0 | 1 | 4,472 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 281.7 | 142.4 | 229 ms | 7.0 ms | 2.0 | 2 | 8,452 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 501.4 | 126.5 | 211 ms | 7.9 ms | 4.0 | 4 | 15,040 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 821.4 | 102.8 | 519 ms | 9.7 ms | 8.0 | 0 | 24,648 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,187.3 | 74.4 | 822 ms | 13.4 ms | 16.0 | 0 | 35,616 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,491.9 | 46.7 | 1,395 ms | 21.4 ms | 32.0 | 0 | 44,736 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.3 | 140.8 | 287 ms | 7.1 ms | 1.0 | 1 | 4,148 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 249.6 | 130.6 | 323 ms | 7.7 ms | 2.0 | 2 | 7,486 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 415.1 | 103.9 | 551 ms | 9.6 ms | 4.0 | 0 | 12,452 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 602.6 | 75.4 | 938 ms | 13.3 ms | 8.0 | 0 | 18,080 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 765.9 | 47.9 | 1,579 ms | 20.9 ms | 16.0 | 0 | 22,976 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 122.3 | 122.6 | 568 ms | 8.2 ms | 1.0 | 0 | 3,669 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 203.0 | 101.9 | 843 ms | 9.8 ms | 2.0 | 0 | 6,090 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 301.5 | 75.6 | 1,092 ms | 13.2 ms | 4.0 | 0 | 9,040 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 388.9 | 48.8 | 1,896 ms | 20.5 ms | 8.0 | 0 | 11,664 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.