Speed test matrix
Laguna-XS-2.1 on H200 NVL 141GB
vllm · 0.26.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.878189 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1 --max-num-batched-tokens 32768 --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context44K186 ms TTFT
16K context42.3K383 ms TTFT
32K context36.9K872 ms TTFT
64K context30.2K2,124 ms TTFT
128K context21.1K6,072 ms TTFT
251K context13.2K18,980 ms TTFT
256K context13.1K19,593 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 127.057 ms TTFT | 212.9304 ms TTFT | 363.2126 ms TTFT | 575.8180 ms TTFT | 891.9230 ms TTFT | 1,199.8331 ms TTFT | 1,466.6536 ms TTFT | Over capacity |
| 16K | 117.479 ms TTFT | 196.480 ms TTFT | 329.8149 ms TTFT | 517.7189 ms TTFT | 774.0263 ms TTFT | 1,000.2437 ms TTFT | Over capacity | Over capacity |
| 32K | 114.292 ms TTFT | 188.5120 ms TTFT | 305.4179 ms TTFT | 483.2262 ms TTFT | 649.3432 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 107.0129 ms TTFT | 173.8229 ms TTFT | 276.0271 ms TTFT | 399.3448 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 96.8273 ms TTFT | 151.4347 ms TTFT | 221.1548 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 81.1546 ms TTFT | 117.4770 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 44K | — | 186 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.3K | — | 383 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.9K | — | 872 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.2K | — | 2,124 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.1K | — | 6,072 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.2K | — | 18,980 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.1K | — | 19,593 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 127.0 | 126.0 | 57 ms | 7.9 ms | 1.0 | 1 | 3,808 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 212.9 | 105.9 | 304 ms | 9.4 ms | 2.0 | 0 | 6,388 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 363.2 | 90.6 | 126 ms | 11.0 ms | 4.0 | 0 | 10,894 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 575.8 | 71.9 | 180 ms | 13.9 ms | 8.0 | 0 | 17,273 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 891.9 | 55.8 | 230 ms | 17.9 ms | 16.0 | 0 | 26,754 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,199.8 | 37.7 | 331 ms | 26.5 ms | 32.0 | 0 | 35,993 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,466.6 | 23.0 | 536 ms | 43.4 ms | 64.0 | 0 | 43,948 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 117.4 | 117.4 | 79 ms | 8.5 ms | 1.0 | 0 | 3,522 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 196.4 | 98.3 | 80 ms | 10.2 ms | 2.0 | 0 | 5,890 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 329.8 | 82.7 | 149 ms | 12.1 ms | 4.0 | 0 | 9,893 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 517.7 | 65.1 | 189 ms | 15.4 ms | 8.0 | 0 | 15,531 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 774.0 | 48.6 | 263 ms | 20.6 ms | 16.0 | 0 | 23,225 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,000.2 | 31.5 | 437 ms | 31.8 ms | 32.0 | 0 | 30,004 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 114.2 | 114.2 | 92 ms | 8.8 ms | 1.0 | 0 | 3,426 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 188.5 | 94.1 | 120 ms | 10.6 ms | 2.0 | 0 | 5,653 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 305.4 | 76.4 | 179 ms | 13.1 ms | 4.0 | 0 | 9,160 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 483.2 | 60.4 | 262 ms | 16.5 ms | 8.0 | 0 | 14,494 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 649.3 | 40.9 | 432 ms | 24.5 ms | 16.0 | 0 | 19,470 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 107.0 | 107.0 | 129 ms | 9.3 ms | 1.0 | 0 | 3,209 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 173.8 | 87.1 | 229 ms | 11.5 ms | 2.0 | 0 | 5,212 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 276.0 | 69.2 | 271 ms | 14.5 ms | 4.0 | 0 | 8,280 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 399.3 | 50.1 | 448 ms | 20.0 ms | 8.0 | 0 | 11,978 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 96.8 | 97.0 | 273 ms | 10.3 ms | 1.0 | 0 | 2,903 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 151.4 | 75.8 | 347 ms | 13.2 ms | 2.0 | 0 | 4,542 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 221.1 | 55.5 | 548 ms | 18.0 ms | 4.0 | 0 | 6,632 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 81.1 | 81.4 | 546 ms | 12.3 ms | 1.0 | 0 | 2,431 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 117.4 | 58.9 | 770 ms | 17.0 ms | 2.0 | 0 | 3,520 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.