Speed test matrix
Laguna-XS-2.1-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --moe-backend tritonPrefill
prompt tok/s by context length8K context16.2K507 ms TTFT
16K context31.9K507 ms TTFT
32K context21.2K1,522 ms TTFT
64K context21.1K3,043 ms TTFT
128K context14.1K9,130 ms TTFT
251K context8,93928,125 ms TTFT
256K context8,72429,391 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 193.832 ms TTFT | 299.941 ms TTFT | Looped | Looped | 1,224.9137 ms TTFT | Looped | Looped | 4,200.7727 ms TTFT |
| 16K | 189.143 ms TTFT | 292.249 ms TTFT | 511.756 ms TTFT | 770.5119 ms TTFT | Looped | Looped | Looped | Over capacity |
| 32K | 180.669 ms TTFT | 282.576 ms TTFT | 496.879 ms TTFT | Looped | 1,044.0342 ms TTFT | Looped | Over capacity | Over capacity |
| 64K | 166.2125 ms TTFT | 266.9132 ms TTFT | 458.8226 ms TTFT | 633.5366 ms TTFT | 864.8655 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 142.4246 ms TTFT | 238.8332 ms TTFT | 398.7484 ms TTFT | 506.2844 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 112.9558 ms TTFT | 198.3810 ms TTFT | 319.31,217 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.2K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.9K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.2K | — | 1,522 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.1K | — | 3,043 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.1K | — | 9,130 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,939.0 | — | 28,125 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,724.0 | — | 29,391 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 193.8 | 194.2 | 32 ms | 5.2 ms | 1.0 | 1 | 5,813 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 299.9 | 150.3 | 41 ms | 6.7 ms | 2.0 | 2 | 8,996 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,224.9 | 77.3 | 137 ms | 12.9 ms | 16.0 | 0 | 36,736 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 4,200.7 | 33.5 | 727 ms | 29.9 ms | 128.0 | 0 | 125,952 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 189.1 | 189.5 | 43 ms | 5.3 ms | 1.0 | 1 | 5,675 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 292.2 | 147.1 | 49 ms | 6.8 ms | 2.0 | 2 | 8,766 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 511.7 | 129.8 | 56 ms | 7.7 ms | 4.0 | 4 | 15,349 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 770.5 | 96.7 | 119 ms | 10.3 ms | 8.0 | 0 | 23,112 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 180.6 | 181.3 | 69 ms | 5.5 ms | 1.0 | 1 | 5,419 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 282.5 | 142.4 | 76 ms | 7.0 ms | 2.0 | 2 | 8,475 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 496.8 | 125.1 | 79 ms | 8.0 ms | 4.0 | 4 | 14,902 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,044.0 | 65.7 | 342 ms | 15.2 ms | 16.0 | 0 | 31,312 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 166.2 | 167.2 | 125 ms | 6.0 ms | 1.0 | 1 | 4,985 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 266.9 | 134.7 | 132 ms | 7.4 ms | 2.0 | 2 | 8,008 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 458.8 | 114.9 | 226 ms | 8.7 ms | 4.0 | 0 | 13,764 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 633.5 | 79.4 | 366 ms | 12.6 ms | 8.0 | 0 | 19,000 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 864.8 | 54.3 | 655 ms | 18.4 ms | 16.0 | 0 | 25,936 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 142.4 | 144.4 | 246 ms | 6.9 ms | 1.0 | 1 | 4,273 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 238.8 | 119.8 | 332 ms | 8.3 ms | 2.0 | 1 | 7,163 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 398.7 | 99.9 | 484 ms | 10.0 ms | 4.0 | 0 | 11,960 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 506.2 | 63.6 | 844 ms | 15.7 ms | 8.0 | 0 | 15,184 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 112.9 | 113.1 | 558 ms | 8.8 ms | 1.0 | 0 | 3,385 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 198.3 | 99.5 | 810 ms | 10.1 ms | 2.0 | 0 | 5,948 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 319.3 | 80.0 | 1,217 ms | 12.5 ms | 4.0 | 0 | 9,580 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.