Speed test matrix
Fara1.5-9B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- fara
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-microsoft-vllm-73a5831127a9/bin/vllm" serve microsoft/Fara1.5-9B --served-model-name fara --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --limit-mm-per-prompt '{"image":10}'Prefill
prompt tok/s by context length8K context21.5K381 ms TTFT
16K context21.4K758 ms TTFT
32K context19.8K1,631 ms TTFT
64K context17.5K3,675 ms TTFT
128K context14.2K9,083 ms TTFT
251K context10.3K24,412 ms TTFT
256K context10.2K25,181 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.243 ms TTFT | 154.961 ms TTFT | 307.479 ms TTFT | Looped | 1,125.2180 ms TTFT | 1,887.0351 ms TTFT | Looped | Looped |
| 16K | 82.159 ms TTFT | 152.274 ms TTFT | 300.8103 ms TTFT | 596.1171 ms TTFT | 1,054.7303 ms TTFT | Looped | Looped | Over capacity |
| 32K | 79.879 ms TTFT | 147.1100 ms TTFT | Looped | 556.3216 ms TTFT | Looped | 1,419.9671 ms TTFT | Over capacity | Over capacity |
| 64K | 75.7137 ms TTFT | 131.3185 ms TTFT | Looped | 490.4430 ms TTFT | 768.9753 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 68.6281 ms TTFT | 113.3368 ms TTFT | 219.5624 ms TTFT | 396.9844 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 58.1571 ms TTFT | 89.1773 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.5K | — | 381 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.4K | — | 758 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,631 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.5K | — | 3,675 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.2K | — | 9,083 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.3K | — | 24,412 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.2K | — | 25,181 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.2 | 83.2 | 43 ms | 12.0 ms | 1.0 | 0 | 2,496 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.9 | 77.5 | 61 ms | 12.9 ms | 2.0 | 0 | 4,648 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 307.4 | 76.9 | 79 ms | 13.0 ms | 4.0 | 0 | 9,221 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,125.2 | 70.3 | 180 ms | 14.2 ms | 16.0 | 0 | 33,744 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,887.0 | 58.9 | 351 ms | 17.0 ms | 32.0 | 0 | 56,609 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 82.1 | 82.1 | 59 ms | 12.2 ms | 1.0 | 0 | 2,461 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 152.2 | 76.1 | 74 ms | 13.1 ms | 2.0 | 0 | 4,567 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 300.8 | 75.2 | 103 ms | 13.3 ms | 4.0 | 0 | 9,024 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 596.1 | 74.5 | 171 ms | 13.4 ms | 8.0 | 0 | 17,880 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,054.7 | 65.8 | 303 ms | 15.2 ms | 16.0 | 0 | 31,648 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 79.8 | 79.8 | 79 ms | 12.5 ms | 1.0 | 0 | 2,394 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 147.1 | 73.6 | 100 ms | 13.6 ms | 2.0 | 0 | 4,413 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 556.3 | 69.6 | 216 ms | 14.4 ms | 8.0 | 0 | 16,689 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,419.9 | 44.1 | 671 ms | 22.7 ms | 32.0 | 0 | 42,592 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.7 | 75.7 | 137 ms | 13.2 ms | 1.0 | 0 | 2,270 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 131.3 | 65.7 | 185 ms | 15.2 ms | 2.0 | 0 | 3,938 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 490.4 | 61.3 | 430 ms | 16.3 ms | 8.0 | 0 | 14,704 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 768.9 | 47.9 | 753 ms | 20.9 ms | 16.0 | 0 | 23,056 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 68.6 | 68.6 | 281 ms | 14.6 ms | 1.0 | 0 | 2,057 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 113.3 | 56.6 | 368 ms | 17.7 ms | 2.0 | 0 | 3,398 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 219.5 | 54.8 | 624 ms | 18.3 ms | 4.0 | 0 | 6,585 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 396.9 | 49.2 | 844 ms | 20.3 ms | 8.0 | 0 | 11,904 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 58.1 | 58.2 | 571 ms | 17.2 ms | 1.0 | 0 | 1,743 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 89.1 | 44.6 | 773 ms | 22.4 ms | 2.0 | 0 | 2,672 | 30.0 s |
| Sustained decode | 251K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.