Speed test matrix
Laguna-XS-2.1-INT4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --moe-backend marlinPrefill
prompt tok/s by context length8K context16.2K507 ms TTFT
16K context16K1,014 ms TTFT
32K context21.2K1,522 ms TTFT
64K context18.1K3,554 ms TTFT
128K context14.1K9,078 ms TTFT
251K context8,70528,880 ms TTFT
256K context8,57829,890 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 195.131 ms TTFT | 310.740 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | Looped | 303.850 ms TTFT | 555.952 ms TTFT | 874.7116 ms TTFT | Looped | Looped | Looped | Over capacity |
| 32K | 181.666 ms TTFT | 293.877 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 64K | 167.0120 ms TTFT | 274.4137 ms TTFT | Looped | Looped | Looped | 1,251.61,290 ms TTFT | Over capacity | Over capacity |
| 128K | 143.3235 ms TTFT | 244.5260 ms TTFT | 420.8493 ms TTFT | 549.6857 ms TTFT | 712.91,397 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 113.3557 ms TTFT | 202.8717 ms TTFT | 335.31,143 ms TTFT | 389.41,855 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.2K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16K | — | 1,014 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.2K | — | 1,522 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.1K | — | 3,554 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.1K | — | 9,078 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,705.0 | — | 28,880 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,578.0 | — | 29,890 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 195.1 | 195.4 | 31 ms | 5.1 ms | 1.0 | 1 | 5,852 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 310.7 | 155.9 | 40 ms | 6.4 ms | 2.0 | 2 | 9,318 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 303.8 | 152.7 | 50 ms | 6.5 ms | 2.0 | 2 | 9,114 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 555.9 | 140.9 | 52 ms | 7.1 ms | 4.0 | 4 | 16,677 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 874.7 | 110.0 | 116 ms | 9.1 ms | 8.0 | 0 | 26,241 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 181.6 | 182.1 | 66 ms | 5.5 ms | 1.0 | 1 | 5,446 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 293.8 | 148.0 | 77 ms | 6.8 ms | 2.0 | 2 | 8,815 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 167.0 | 168.0 | 120 ms | 6.0 ms | 1.0 | 1 | 5,009 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 274.4 | 138.7 | 137 ms | 7.2 ms | 2.0 | 2 | 8,232 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,251.6 | 39.6 | 1,290 ms | 25.2 ms | 32.0 | 0 | 37,535 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 143.3 | 145.1 | 235 ms | 6.9 ms | 1.0 | 1 | 4,298 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.5 | 125.6 | 260 ms | 8.0 ms | 2.0 | 2 | 7,334 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 420.8 | 105.4 | 493 ms | 9.5 ms | 4.0 | 0 | 12,624 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 549.6 | 68.9 | 857 ms | 14.5 ms | 8.0 | 0 | 16,489 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 712.9 | 44.7 | 1,397 ms | 22.4 ms | 16.0 | 0 | 21,392 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 113.3 | 113.6 | 557 ms | 8.8 ms | 1.0 | 0 | 3,398 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 202.8 | 101.6 | 717 ms | 9.8 ms | 2.0 | 0 | 6,082 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 335.3 | 84.1 | 1,143 ms | 11.9 ms | 4.0 | 0 | 10,058 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 389.4 | 49.0 | 1,855 ms | 20.4 ms | 8.0 | 0 | 11,680 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.