Speed test matrix
Fara1.5-4B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- fara
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-microsoft-vllm-73a5831127a9/bin/vllm" serve microsoft/Fara1.5-4B --served-model-name fara --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --limit-mm-per-prompt '{"image":10}'Prefill
prompt tok/s by context length8K context34.6K236 ms TTFT
16K context34.2K474 ms TTFT
32K context30.7K1,053 ms TTFT
64K context25.4K2,533 ms TTFT
128K context18.9K6,794 ms TTFT
251K context12.7K19,920 ms TTFT
256K context12.5K20,533 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 141.437 ms TTFT | 256.352 ms TTFT | 490.968 ms TTFT | 915.3106 ms TTFT | Looped | 2,646.7247 ms TTFT | 3,910.4463 ms TTFT | 4,786.2862 ms TTFT |
| 16K | 138.151 ms TTFT | 248.566 ms TTFT | Looped | 872.0148 ms TTFT | 1,506.5216 ms TTFT | 2,316.3411 ms TTFT | Looped | Over capacity |
| 32K | 131.573 ms TTFT | 234.993 ms TTFT | Looped | Looped | 1,282.2371 ms TTFT | 1,819.3658 ms TTFT | Over capacity | Over capacity |
| 64K | 120.7132 ms TTFT | Looped | 376.9245 ms TTFT | 665.5403 ms TTFT | 984.3706 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 103.7272 ms TTFT | 159.5352 ms TTFT | 300.9519 ms TTFT | 505.7847 ms TTFT | 671.91,558 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 81.4569 ms TTFT | 115.2768 ms TTFT | 214.11,131 ms TTFT | 345.41,833 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.6K | — | 236 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.2K | — | 474 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.7K | — | 1,053 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.4K | — | 2,533 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.9K | — | 6,794 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.7K | — | 19,920 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.5K | — | 20,533 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 141.4 | 142.0 | 37 ms | 7.0 ms | 1.0 | 1 | 4,243 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 256.3 | 129.0 | 52 ms | 7.8 ms | 2.0 | 2 | 7,688 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 490.9 | 123.6 | 68 ms | 8.1 ms | 4.0 | 4 | 14,725 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 915.3 | 114.8 | 106 ms | 8.7 ms | 8.0 | 0 | 27,456 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 2,646.7 | 82.9 | 247 ms | 12.1 ms | 32.0 | 0 | 79,392 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,910.4 | 61.0 | 463 ms | 16.4 ms | 64.0 | 0 | 117,313 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,786.2 | 37.1 | 862 ms | 26.9 ms | 128.0 | 0 | 143,488 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 138.1 | 138.7 | 51 ms | 7.2 ms | 1.0 | 1 | 4,141 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 248.5 | 125.2 | 66 ms | 8.0 ms | 2.0 | 2 | 7,454 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 872.0 | 109.2 | 148 ms | 9.2 ms | 8.0 | 0 | 26,161 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,506.5 | 94.1 | 216 ms | 10.6 ms | 16.0 | 0 | 45,184 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,316.3 | 72.2 | 411 ms | 13.9 ms | 32.0 | 0 | 69,472 | 30.0 s |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 131.5 | 132.3 | 73 ms | 7.6 ms | 1.0 | 1 | 3,945 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 234.9 | 117.6 | 93 ms | 8.5 ms | 2.0 | 0 | 7,046 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,282.2 | 80.0 | 371 ms | 12.5 ms | 16.0 | 0 | 38,464 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,819.3 | 56.5 | 658 ms | 17.7 ms | 32.0 | 0 | 54,560 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 120.7 | 120.8 | 132 ms | 8.3 ms | 1.0 | 0 | 3,620 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 376.9 | 94.3 | 245 ms | 10.6 ms | 4.0 | 0 | 11,307 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 665.5 | 83.1 | 403 ms | 12.0 ms | 8.0 | 0 | 19,960 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 984.3 | 61.3 | 706 ms | 16.3 ms | 16.0 | 0 | 29,520 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 103.7 | 103.8 | 272 ms | 9.6 ms | 1.0 | 0 | 3,112 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 159.5 | 79.8 | 352 ms | 12.5 ms | 2.0 | 0 | 4,784 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 300.9 | 75.2 | 519 ms | 13.3 ms | 4.0 | 0 | 9,024 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 505.7 | 62.8 | 847 ms | 15.9 ms | 8.0 | 0 | 15,168 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 671.9 | 41.5 | 1,558 ms | 24.1 ms | 16.0 | 0 | 20,144 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 81.4 | 81.5 | 569 ms | 12.3 ms | 1.0 | 0 | 2,440 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 115.2 | 57.7 | 768 ms | 17.3 ms | 2.0 | 0 | 3,454 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 214.1 | 53.5 | 1,131 ms | 18.7 ms | 4.0 | 0 | 6,420 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 345.4 | 42.7 | 1,833 ms | 23.4 ms | 8.0 | 0 | 10,360 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.