Speed test matrix
Laguna-XS-2.1-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS-2.1-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --linear-backend flashinfer_cutlass --moe-backend flashinfer_cutlass --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-NVFP4","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context16.2K507 ms TTFT
16K context30.9K524 ms TTFT
32K context21.1K1,523 ms TTFT
64K context21.1K3,044 ms TTFT
128K context14.8K8,676 ms TTFT
251K context9,08227,680 ms TTFT
256K context8,97428,572 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 242.540 ms TTFT | 325.858 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | 196.049 ms TTFT | Looped | 553.079 ms TTFT | 844.7137 ms TTFT | Looped | Looped | Looped | Over capacity |
| 32K | 173.480 ms TTFT | Looped | 490.6116 ms TTFT | 658.9242 ms TTFT | Looped | 1,265.7673 ms TTFT | Over capacity | Over capacity |
| 64K | 167.9120 ms TTFT | 276.6144 ms TTFT | 411.3247 ms TTFT | Looped | 688.9738 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 158.2255 ms TTFT | 216.0334 ms TTFT | 337.4486 ms TTFT | 361.2840 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 128.0540 ms TTFT | 177.0759 ms TTFT | 229.61,152 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.2K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.9K | — | 524 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.1K | — | 1,523 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.1K | — | 3,044 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,676 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,082.0 | — | 27,680 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,974.0 | — | 28,572 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 242.5 | 244.4 | 40 ms | 4.1 ms | 1.0 | 1 | 7,274 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 325.8 | 165.0 | 58 ms | 6.1 ms | 2.0 | 2 | 9,772 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 196.0 | 205.5 | 49 ms | 5.0 ms | 1.0 | 1 | 5,878 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 553.0 | 135.7 | 79 ms | 7.4 ms | 4.0 | 4 | 16,584 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 844.7 | 101.0 | 137 ms | 9.9 ms | 8.0 | 2 | 25,342 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 173.4 | 178.4 | 80 ms | 5.6 ms | 1.0 | 1 | 5,201 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 490.6 | 115.0 | 116 ms | 8.7 ms | 4.0 | 2 | 14,715 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 658.9 | 81.5 | 242 ms | 12.3 ms | 8.0 | 0 | 19,758 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,265.7 | 39.6 | 673 ms | 25.3 ms | 32.0 | 0 | 37,949 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 167.9 | 168.8 | 120 ms | 5.9 ms | 1.0 | 1 | 5,036 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 276.6 | 148.4 | 144 ms | 6.7 ms | 2.0 | 2 | 8,295 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 411.3 | 103.6 | 247 ms | 9.7 ms | 4.0 | 0 | 12,338 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 688.9 | 42.9 | 738 ms | 23.3 ms | 16.0 | 0 | 20,644 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.2 | 159.5 | 255 ms | 6.3 ms | 1.0 | 1 | 4,746 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 216.0 | 109.0 | 334 ms | 9.2 ms | 2.0 | 0 | 6,479 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 337.4 | 80.4 | 486 ms | 12.5 ms | 4.0 | 0 | 10,118 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 361.2 | 46.3 | 840 ms | 21.6 ms | 8.0 | 0 | 10,831 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 128.0 | 146.6 | 540 ms | 6.9 ms | 1.0 | 1 | 3,837 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 177.0 | 89.6 | 759 ms | 11.2 ms | 2.0 | 0 | 5,311 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 229.6 | 56.3 | 1,152 ms | 17.8 ms | 4.0 | 0 | 6,881 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.