Speed test matrix
Laguna-XS.2-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS.2-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --linear-backend flashinfer_cutlass --moe-backend flashinfer_cutlassPrefill
prompt tok/s by context length8K context41.6K197 ms TTFT
16K context38.6K420 ms TTFT
32K context32.3K998 ms TTFT
64K context23.9K2,684 ms TTFT
128K context15.4K8,312 ms TTFT
251K context9,20327,319 ms TTFT
256K context9,05228,325 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 180.030 ms TTFT | 302.937 ms TTFT | 565.543 ms TTFT | 928.981 ms TTFT | 1,528.7135 ms TTFT | 2,280.3198 ms TTFT | 3,610.6397 ms TTFT | 5,312.5702 ms TTFT |
| 16K | 176.141 ms TTFT | 298.347 ms TTFT | 550.051 ms TTFT | 890.3103 ms TTFT | Looped | 2,047.5319 ms TTFT | 3,200.6621 ms TTFT | Over capacity |
| 32K | 168.667 ms TTFT | 287.470 ms TTFT | 523.372 ms TTFT | 810.8192 ms TTFT | 1,236.3356 ms TTFT | 1,723.7620 ms TTFT | 2,603.91,195 ms TTFT | Over capacity |
| 64K | 155.8119 ms TTFT | 269.4127 ms TTFT | 482.2132 ms TTFT | 703.2374 ms TTFT | 1,001.2694 ms TTFT | 1,286.01,253 ms TTFT | Over capacity | Over capacity |
| 128K | 134.8247 ms TTFT | 239.1274 ms TTFT | 416.7494 ms TTFT | 548.2821 ms TTFT | 717.61,501 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 107.9551 ms TTFT | 198.7849 ms TTFT | 332.31,196 ms TTFT | 387.02,097 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.6K | — | 197 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.6K | — | 420 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.3K | — | 998 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 23.9K | — | 2,684 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.4K | — | 8,312 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,203.0 | — | 27,319 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,052.0 | — | 28,325 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 180.0 | 180.3 | 30 ms | 5.5 ms | 1.0 | 1 | 5,400 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 302.9 | 152.0 | 37 ms | 6.6 ms | 2.0 | 2 | 9,085 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 565.5 | 143.0 | 43 ms | 7.0 ms | 4.0 | 4 | 16,964 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 928.9 | 116.8 | 81 ms | 8.6 ms | 8.0 | 0 | 27,864 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,528.7 | 96.3 | 135 ms | 10.4 ms | 16.0 | 0 | 45,856 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,280.3 | 72.3 | 198 ms | 13.8 ms | 32.0 | 0 | 68,410 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,610.6 | 57.5 | 397 ms | 17.4 ms | 64.0 | 0 | 108,288 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,312.5 | 42.4 | 702 ms | 23.6 ms | 128.0 | 0 | 159,360 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 176.1 | 176.5 | 41 ms | 5.7 ms | 1.0 | 1 | 5,281 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 298.3 | 149.9 | 47 ms | 6.7 ms | 2.0 | 2 | 8,950 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 550.0 | 139.2 | 51 ms | 7.2 ms | 4.0 | 4 | 16,500 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 890.3 | 111.8 | 103 ms | 8.9 ms | 8.0 | 0 | 26,704 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 2,047.5 | 64.9 | 319 ms | 15.4 ms | 32.0 | 0 | 61,451 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,200.6 | 50.8 | 621 ms | 19.7 ms | 64.0 | 0 | 96,017 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 168.6 | 169.2 | 67 ms | 5.9 ms | 1.0 | 1 | 5,057 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 287.4 | 144.5 | 70 ms | 6.9 ms | 2.0 | 2 | 8,620 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 523.3 | 132.8 | 72 ms | 7.5 ms | 4.0 | 4 | 15,697 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 810.8 | 101.9 | 192 ms | 9.8 ms | 8.0 | 0 | 24,324 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,236.3 | 77.9 | 356 ms | 12.8 ms | 16.0 | 0 | 37,088 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,723.7 | 54.4 | 620 ms | 18.4 ms | 32.0 | 0 | 51,712 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,603.9 | 41.2 | 1,195 ms | 24.2 ms | 64.0 | 0 | 78,117 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 155.8 | 156.8 | 119 ms | 6.4 ms | 1.0 | 1 | 4,674 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 269.4 | 136.0 | 127 ms | 7.4 ms | 2.0 | 2 | 8,082 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 482.2 | 122.5 | 132 ms | 8.2 ms | 3.9 | 4 | 14,462 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 703.2 | 88.2 | 374 ms | 11.3 ms | 8.0 | 0 | 21,096 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,001.2 | 62.9 | 694 ms | 15.9 ms | 16.0 | 0 | 30,032 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,286.0 | 40.7 | 1,253 ms | 24.6 ms | 32.0 | 0 | 38,560 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 134.8 | 137.0 | 247 ms | 7.3 ms | 1.0 | 1 | 4,043 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 239.1 | 120.5 | 274 ms | 8.3 ms | 2.0 | 2 | 7,172 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 416.7 | 104.4 | 494 ms | 9.6 ms | 4.0 | 0 | 12,500 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 548.2 | 68.8 | 821 ms | 14.5 ms | 8.0 | 0 | 16,440 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 717.6 | 45.1 | 1,501 ms | 22.2 ms | 16.0 | 0 | 21,527 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 107.9 | 108.2 | 551 ms | 9.2 ms | 1.0 | 0 | 3,237 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 198.7 | 99.6 | 849 ms | 10.0 ms | 2.0 | 0 | 5,962 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 332.3 | 83.2 | 1,196 ms | 12.0 ms | 4.0 | 0 | 9,972 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 387.0 | 48.6 | 2,097 ms | 20.6 ms | 8.0 | 0 | 11,608 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.