Speed test matrix
ZAYA1-8B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.23.1rc1.dev772+gb1b99a08b · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.23.1rc1.dev772+gb1b99a08b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
b1b99a08b20f
Run
- Served model
- zaya
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-zyphra-vllm-pr-b1b99a08b20f/bin/vllm" serve Zyphra/ZAYA1-8B --served-model-name zaya --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser zaya_xml --max-model-len 131072 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-dtype float32 --dtype bfloat16
Prefill
prompt tok/s by context length8K context46K178 ms TTFT
16K context52.3K311 ms TTFT
32K context42.7K757 ms TTFT
64K context30.7K2,099 ms TTFT
123K context19.5K6,342 ms TTFT
128K context19.2K6,722 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 111.2174 ms TTFT | 167.4320 ms TTFT | 308.2575 ms TTFT | 547.3902 ms TTFT | 871.91,475 ms TTFT | 1,329.22,645 ms TTFT | 1,927.65,053 ms TTFT | Over capacity |
| 16K | 106.4315 ms TTFT | 161.8538 ms TTFT | 293.0930 ms TTFT | 490.81,556 ms TTFT | 734.32,824 ms TTFT | 1,032.45,424 ms TTFT | 1,371.710,852 ms TTFT | Over capacity |
| 32K | 103.6754 ms TTFT | 149.71,214 ms TTFT | 259.72,073 ms TTFT | 404.63,652 ms TTFT | 556.66,820 ms TTFT | 715.913,474 ms TTFT | Over capacity | Over capacity |
| 64K | 92.12,091 ms TTFT | 131.23,294 ms TTFT | 211.55,612 ms TTFT | 298.310,086 ms TTFT | 375.919,175 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 77.96,332 ms TTFT | 107.49,776 ms TTFT | 155.816,414 ms TTFT | 202.029,720 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 46K | — | 178 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 52.3K | — | 311 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 42.7K | — | 757 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.7K | — | 2,099 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 19.5K | — | 6,342 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.2K | — | 6,722 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 111.2 | 111.3 | 174 ms | 9.0 ms | 1.0 | 0 | 3,336 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 167.4 | 83.8 | 320 ms | 11.9 ms | 2.0 | 0 | 5,020 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 308.2 | 77.2 | 575 ms | 13.0 ms | 4.0 | 0 | 9,244 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 547.3 | 68.2 | 902 ms | 14.7 ms | 8.0 | 0 | 16,416 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 871.9 | 53.7 | 1,475 ms | 18.6 ms | 16.0 | 0 | 26,160 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,329.2 | 39.9 | 2,645 ms | 25.1 ms | 32.0 | 0 | 39,872 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,927.6 | 27.3 | 5,053 ms | 36.7 ms | 64.0 | 0 | 57,792 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 106.4 | 106.7 | 315 ms | 9.4 ms | 1.0 | 0 | 3,193 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 161.8 | 80.8 | 538 ms | 12.4 ms | 2.0 | 0 | 4,852 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 293.0 | 72.8 | 930 ms | 13.7 ms | 4.0 | 0 | 8,788 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 490.8 | 60.0 | 1,556 ms | 16.7 ms | 8.0 | 0 | 14,720 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 734.3 | 43.6 | 2,824 ms | 23.0 ms | 16.0 | 0 | 22,016 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,032.4 | 28.9 | 5,424 ms | 34.6 ms | 32.0 | 0 | 30,944 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,371.7 | 17.1 | 10,852 ms | 58.4 ms | 64.0 | 0 | 41,152 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 103.6 | 103.7 | 754 ms | 9.6 ms | 1.0 | 0 | 3,108 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 149.7 | 74.3 | 1,214 ms | 13.5 ms | 2.0 | 0 | 4,488 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 259.7 | 63.3 | 2,073 ms | 15.8 ms | 4.0 | 0 | 7,791 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 404.6 | 47.4 | 3,652 ms | 21.1 ms | 8.0 | 0 | 12,136 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 556.6 | 30.4 | 6,820 ms | 32.9 ms | 16.0 | 0 | 16,688 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 715.9 | 17.3 | 13,474 ms | 57.8 ms | 32.0 | 0 | 21,478 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 92.1 | 92.2 | 2,091 ms | 10.8 ms | 1.0 | 0 | 2,763 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 131.2 | 63.9 | 3,294 ms | 15.7 ms | 2.0 | 0 | 3,935 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 211.5 | 48.9 | 5,612 ms | 20.5 ms | 4.0 | 0 | 6,342 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 298.3 | 31.0 | 10,086 ms | 32.3 ms | 8.0 | 0 | 8,944 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 375.9 | 16.6 | 19,175 ms | 60.1 ms | 16.0 | 0 | 11,264 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 77.9 | 78.0 | 6,332 ms | 12.8 ms | 1.0 | 0 | 2,337 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 107.4 | 49.8 | 9,776 ms | 20.2 ms | 2.0 | 0 | 3,222 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 155.8 | 31.0 | 16,414 ms | 32.4 ms | 4.0 | 0 | 4,672 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 202.0 | 15.7 | 29,720 ms | 63.9 ms | 8.0 | 0 | 6,056 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.