Speed test matrix
Laguna-XS-2.1-INT4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --moe-backend marlin --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context16.2K507 ms TTFT
16K context16K1,015 ms TTFT
32K context21.2K1,520 ms TTFT
64K context18.1K3,550 ms TTFT
128K context14K9,179 ms TTFT
251K context8,62729,141 ms TTFT
256K context8,52530,074 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 220.639 ms TTFT | 338.758 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 16K | Looped | 320.763 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 32K | 219.068 ms TTFT | 329.085 ms TTFT | 458.3142 ms TTFT | 684.7229 ms TTFT | 1,021.4411 ms TTFT | Looped | Over capacity | Over capacity |
| 64K | 182.8125 ms TTFT | 279.2132 ms TTFT | 393.6235 ms TTFT | 535.9420 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
| 128K | 147.6244 ms TTFT | 244.1261 ms TTFT | Looped | 354.2809 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 129.2523 ms TTFT | 169.2797 ms TTFT | 213.61,180 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.2K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16K | — | 1,015 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.2K | — | 1,520 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.1K | — | 3,550 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14K | — | 9,179 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,627.0 | — | 29,141 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,525.0 | — | 30,074 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 220.6 | 227.1 | 39 ms | 4.5 ms | 1.0 | 1 | 6,618 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 338.7 | 176.2 | 58 ms | 5.7 ms | 2.0 | 2 | 10,160 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 320.7 | 171.5 | 63 ms | 5.8 ms | 2.0 | 2 | 9,622 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 219.0 | 217.1 | 68 ms | 4.7 ms | 1.0 | 1 | 6,568 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 329.0 | 162.1 | 85 ms | 6.2 ms | 2.0 | 2 | 9,869 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 458.3 | 118.4 | 142 ms | 8.4 ms | 4.0 | 1 | 13,748 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 684.7 | 86.9 | 229 ms | 11.5 ms | 8.0 | 0 | 20,531 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,021.4 | 62.7 | 411 ms | 15.9 ms | 16.0 | 0 | 30,640 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 182.8 | 178.7 | 125 ms | 5.8 ms | 1.0 | 1 | 5,481 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 279.2 | 127.1 | 132 ms | 7.9 ms | 2.0 | 2 | 8,376 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 393.6 | 100.8 | 235 ms | 9.9 ms | 4.0 | 0 | 11,803 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 535.9 | 68.0 | 420 ms | 14.7 ms | 8.0 | 0 | 16,071 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 147.6 | 154.4 | 244 ms | 6.5 ms | 1.0 | 1 | 4,427 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.1 | 105.6 | 261 ms | 9.5 ms | 2.0 | 1 | 7,320 | 30.0 s |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Valid | 354.2 | 44.9 | 809 ms | 22.3 ms | 8.0 | 0 | 10,626 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 129.2 | 159.6 | 523 ms | 6.5 ms | 1.0 | 1 | 3,875 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 169.2 | 85.1 | 797 ms | 11.8 ms | 2.0 | 0 | 5,074 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 213.6 | 53.9 | 1,180 ms | 18.5 ms | 4.0 | 0 | 6,403 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.