Speed test matrix
Laguna-XS.2-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-poolside-vllm-73a5831127a9/bin/vllm" serve poolside/Laguna-XS.2-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend FLASHINFER --moe-backend tritonPrefill
prompt tok/s by context length8K context37.1K221 ms TTFT
16K context34.7K467 ms TTFT
32K context29.5K1,092 ms TTFT
64K context22.3K2,881 ms TTFT
128K context14.8K8,685 ms TTFT
251K context8,95928,061 ms TTFT
256K context8,68829,513 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 194.829 ms TTFT | 300.035 ms TTFT | 523.441 ms TTFT | 804.872 ms TTFT | 1,253.3131 ms TTFT | 1,839.2224 ms TTFT | 2,840.0464 ms TTFT | 4,315.4855 ms TTFT |
| 16K | 190.042 ms TTFT | 294.049 ms TTFT | 516.457 ms TTFT | Looped | 1,169.3208 ms TTFT | 1,693.4380 ms TTFT | 2,590.2618 ms TTFT | Over capacity |
| 32K | 181.567 ms TTFT | 286.075 ms TTFT | 489.676 ms TTFT | 723.5207 ms TTFT | 1,054.5359 ms TTFT | 1,455.5631 ms TTFT | Over capacity | Over capacity |
| 64K | 166.9124 ms TTFT | 266.8144 ms TTFT | 459.5228 ms TTFT | 632.3382 ms TTFT | 874.4673 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 143.1245 ms TTFT | 239.8343 ms TTFT | 394.7480 ms TTFT | 505.6812 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 113.3561 ms TTFT | 197.2754 ms TTFT | 316.71,164 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.1K | — | 221 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.7K | — | 467 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.5K | — | 1,092 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.3K | — | 2,881 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,685 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,959.0 | — | 28,061 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,688.0 | — | 29,513 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 194.8 | 195.1 | 29 ms | 5.1 ms | 1.0 | 1 | 5,842 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 300.0 | 150.5 | 35 ms | 6.6 ms | 2.0 | 2 | 8,998 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 523.4 | 133.2 | 41 ms | 7.5 ms | 4.0 | 4 | 15,700 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 804.8 | 101.3 | 72 ms | 9.9 ms | 8.0 | 0 | 24,144 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,253.3 | 79.1 | 131 ms | 12.6 ms | 16.0 | 0 | 37,598 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,839.2 | 58.3 | 224 ms | 17.1 ms | 32.0 | 0 | 55,169 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,840.0 | 45.1 | 464 ms | 22.2 ms | 64.0 | 0 | 85,199 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,315.4 | 34.5 | 855 ms | 29.0 ms | 128.0 | 0 | 129,461 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 190.0 | 190.5 | 42 ms | 5.3 ms | 1.0 | 1 | 5,699 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 294.0 | 147.8 | 49 ms | 6.8 ms | 2.0 | 2 | 8,818 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 516.4 | 130.5 | 57 ms | 7.7 ms | 3.9 | 4 | 15,490 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 1,169.3 | 73.9 | 208 ms | 13.5 ms | 16.0 | 0 | 35,072 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,693.4 | 53.6 | 380 ms | 18.7 ms | 32.0 | 0 | 50,784 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,590.2 | 41.0 | 618 ms | 24.4 ms | 64.0 | 0 | 77,696 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 181.5 | 182.1 | 67 ms | 5.5 ms | 1.0 | 1 | 5,444 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 286.0 | 143.6 | 75 ms | 7.0 ms | 2.0 | 2 | 8,578 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 489.6 | 125.1 | 76 ms | 8.0 ms | 4.0 | 4 | 14,688 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 723.5 | 90.8 | 207 ms | 11.0 ms | 8.0 | 0 | 21,705 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,054.5 | 66.4 | 359 ms | 15.1 ms | 16.0 | 0 | 31,632 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,455.5 | 46.0 | 631 ms | 21.8 ms | 32.0 | 0 | 43,648 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 166.9 | 167.9 | 124 ms | 6.0 ms | 1.0 | 1 | 5,006 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 266.8 | 135.1 | 144 ms | 7.4 ms | 2.0 | 2 | 8,004 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 459.5 | 115.0 | 228 ms | 8.7 ms | 4.0 | 0 | 13,784 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 632.3 | 79.4 | 382 ms | 12.6 ms | 8.0 | 0 | 18,968 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 874.4 | 55.0 | 673 ms | 18.2 ms | 16.0 | 0 | 26,224 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 143.1 | 144.9 | 245 ms | 6.9 ms | 1.0 | 1 | 4,293 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 239.8 | 120.4 | 343 ms | 8.3 ms | 2.0 | 2 | 7,166 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 394.7 | 99.0 | 480 ms | 10.1 ms | 4.0 | 0 | 11,840 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 505.6 | 63.5 | 812 ms | 15.8 ms | 8.0 | 0 | 15,168 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 113.3 | 113.6 | 561 ms | 8.8 ms | 1.0 | 0 | 3,399 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 197.2 | 98.9 | 754 ms | 10.1 ms | 2.0 | 0 | 5,916 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 316.7 | 79.4 | 1,164 ms | 12.6 ms | 4.0 | 0 | 9,492 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.