Speed test matrix
Laguna-XS-2.1-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --moe-backend triton --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-FP8","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context16.1K508 ms TTFT
16K context32K507 ms TTFT
32K context21.1K1,524 ms TTFT
64K context21.1K3,046 ms TTFT
128K context14.6K8,773 ms TTFT
251K context8,85328,398 ms TTFT
256K context8,76829,242 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | 289.161 ms TTFT | Looped | 717.5137 ms TTFT | Looped | 1,467.8286 ms TTFT | Looped | Over capacity |
| 16K | 175.253 ms TTFT | 243.269 ms TTFT | 414.9108 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
| 32K | 148.775 ms TTFT | Looped | 379.3170 ms TTFT | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 64K | 154.2124 ms TTFT | 219.8168 ms TTFT | Looped | 467.5417 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
| 128K | 132.7255 ms TTFT | 188.0315 ms TTFT | 255.4520 ms TTFT | 308.8818 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 120.5556 ms TTFT | 156.8767 ms TTFT | 193.31,139 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.1K | — | 1,524 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.1K | — | 3,046 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.6K | — | 8,773 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,853.0 | — | 28,398 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,768.0 | — | 29,242 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Valid | 289.1 | 124.8 | 61 ms | 8.0 ms | 2.0 | 2 | 8,670 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 717.5 | 91.6 | 137 ms | 10.9 ms | 8.0 | 0 | 21,510 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 1,467.8 | 45.1 | 286 ms | 22.2 ms | 32.0 | 0 | 44,024 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 175.2 | 170.3 | 53 ms | 5.9 ms | 1.0 | 1 | 5,254 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 243.2 | 134.5 | 69 ms | 7.5 ms | 2.0 | 2 | 7,294 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 414.9 | 95.7 | 108 ms | 10.5 ms | 4.0 | 1 | 12,439 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 148.7 | 160.6 | 75 ms | 6.3 ms | 1.0 | 1 | 4,461 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 379.3 | 89.7 | 170 ms | 11.1 ms | 4.0 | 0 | 11,371 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 154.2 | 158.9 | 124 ms | 6.3 ms | 1.0 | 1 | 4,624 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 219.8 | 112.1 | 168 ms | 8.9 ms | 2.0 | 0 | 6,590 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 467.5 | 58.7 | 417 ms | 17.0 ms | 8.0 | 0 | 14,019 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 132.7 | 139.5 | 255 ms | 7.2 ms | 1.0 | 1 | 3,980 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 188.0 | 95.4 | 315 ms | 10.5 ms | 2.0 | 0 | 5,638 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 255.4 | 66.0 | 520 ms | 15.2 ms | 4.0 | 0 | 7,656 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 308.8 | 38.5 | 818 ms | 26.0 ms | 8.0 | 0 | 9,265 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 120.5 | 118.6 | 556 ms | 8.4 ms | 1.0 | 0 | 3,613 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 156.8 | 78.8 | 767 ms | 12.7 ms | 2.0 | 0 | 4,701 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 193.3 | 50.8 | 1,139 ms | 19.7 ms | 4.0 | 0 | 5,798 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.