Speed test matrix
Fara1.5-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- fara
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-microsoft-vllm/bin/vllm" serve microsoft/Fara1.5-27B --served-model-name fara --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.882500 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --limit-mm-per-prompt '{"image":10}'Prefill
prompt tok/s by context length8K context10.4K790 ms TTFT
16K context10.1K1,601 ms TTFT
32K context9,6273,354 ms TTFT
64K context8,8477,282 ms TTFT
128K context7,45717,260 ms TTFT
251K context5,66844,499 ms TTFT
256K context5,64545,564 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 59.772 ms TTFT | 118.2116 ms TTFT | 229.61,777 ms TTFT | 433.1331 ms TTFT | 790.5681 ms TTFT | 1,370.71,251 ms TTFT | 2,070.32,427 ms TTFT | Over capacity |
| 16K | 59.3128 ms TTFT | 116.7160 ms TTFT | 224.6326 ms TTFT | 415.5484 ms TTFT | 735.2932 ms TTFT | 1,203.61,810 ms TTFT | Over capacity | Over capacity |
| 32K | 58.5118 ms TTFT | 113.8157 ms TTFT | 214.4249 ms TTFT | 384.4295 ms TTFT | 642.3478 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 56.8161 ms TTFT | 108.2255 ms TTFT | 197.5390 ms TTFT | 333.0506 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 53.8275 ms TTFT | 98.7368 ms TTFT | 168.9571 ms TTFT | 256.9905 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 48.7658 ms TTFT | 83.3892 ms TTFT | 129.41,271 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 790 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.1K | — | 1,601 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,627.0 | — | 3,354 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,847.0 | — | 7,282 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,457.0 | — | 17,260 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,668.0 | — | 44,499 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,645.0 | — | 45,564 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 59.7 | 59.7 | 72 ms | 16.7 ms | 1.0 | 0 | 1,790 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 118.2 | 59.1 | 116 ms | 16.9 ms | 2.0 | 0 | 3,546 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 229.6 | 57.4 | 1,777 ms | 17.4 ms | 4.0 | 0 | 6,884 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 433.1 | 54.2 | 331 ms | 18.5 ms | 8.0 | 0 | 12,993 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 790.5 | 49.5 | 681 ms | 20.2 ms | 16.0 | 0 | 23,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,370.7 | 43.0 | 1,251 ms | 23.3 ms | 32.0 | 0 | 41,120 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,070.3 | 32.5 | 2,427 ms | 30.8 ms | 64.0 | 0 | 62,080 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.3 | 59.4 | 128 ms | 16.8 ms | 1.0 | 0 | 1,780 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 116.7 | 58.4 | 160 ms | 17.1 ms | 2.0 | 0 | 3,502 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 224.6 | 56.2 | 326 ms | 17.8 ms | 4.0 | 0 | 6,736 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 415.5 | 51.9 | 484 ms | 19.3 ms | 8.0 | 0 | 12,464 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 735.2 | 45.9 | 932 ms | 21.8 ms | 16.0 | 0 | 22,064 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,203.6 | 37.6 | 1,810 ms | 26.6 ms | 32.0 | 0 | 36,096 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 58.5 | 58.5 | 118 ms | 17.1 ms | 1.0 | 0 | 1,754 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 113.8 | 57.0 | 157 ms | 17.6 ms | 2.0 | 0 | 3,412 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 214.4 | 53.7 | 249 ms | 18.6 ms | 4.0 | 0 | 6,433 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 384.4 | 48.0 | 295 ms | 20.8 ms | 8.0 | 0 | 11,528 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 642.3 | 40.1 | 478 ms | 25.0 ms | 16.0 | 0 | 19,264 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 56.8 | 56.8 | 161 ms | 17.6 ms | 1.0 | 0 | 1,703 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 108.2 | 54.2 | 255 ms | 18.5 ms | 2.0 | 0 | 3,244 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 197.5 | 49.4 | 390 ms | 20.2 ms | 4.0 | 0 | 5,924 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 333.0 | 41.6 | 506 ms | 24.0 ms | 8.0 | 0 | 9,984 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 53.8 | 53.8 | 275 ms | 18.6 ms | 1.0 | 0 | 1,612 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 98.7 | 49.4 | 368 ms | 20.2 ms | 2.0 | 0 | 2,960 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 168.9 | 42.3 | 571 ms | 23.7 ms | 4.0 | 0 | 5,064 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 256.9 | 32.1 | 905 ms | 31.1 ms | 8.0 | 0 | 7,704 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 48.7 | 48.8 | 658 ms | 20.5 ms | 1.0 | 0 | 1,461 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 83.3 | 41.7 | 892 ms | 24.0 ms | 2.0 | 0 | 2,500 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 129.4 | 32.2 | 1,271 ms | 31.0 ms | 4.0 | 0 | 3,880 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.