Speed test matrix
Laguna-XS.2 on H200 NVL 141GB
vllm · 0.26.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS.2 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.876036 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context43.2K190 ms TTFT
16K context40.1K404 ms TTFT
32K context37K872 ms TTFT
64K context29.9K2,151 ms TTFT
128K context20.9K6,146 ms TTFT
251K context13.1K19,135 ms TTFT
256K context13.1K19,615 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 208.637 ms TTFT | 355.567 ms TTFT | 589.076 ms TTFT | 905.7159 ms TTFT | 1,449.5199 ms TTFT | 2,158.1317 ms TTFT | 3,232.3479 ms TTFT | Over capacity |
| 16K | 204.663 ms TTFT | 347.0120 ms TTFT | 570.1155 ms TTFT | 864.4218 ms TTFT | 1,362.3262 ms TTFT | 1,967.5475 ms TTFT | 2,800.2681 ms TTFT | Over capacity |
| 32K | 197.7115 ms TTFT | 332.5132 ms TTFT | 537.4203 ms TTFT | 795.7357 ms TTFT | 1,207.7510 ms TTFT | 1,666.6842 ms TTFT | Over capacity | Over capacity |
| 64K | 184.9168 ms TTFT | 306.5208 ms TTFT | 483.5249 ms TTFT | 690.1565 ms TTFT | 988.9965 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 165.2300 ms TTFT | 264.2375 ms TTFT | 406.0558 ms TTFT | 540.8878 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 135.7576 ms TTFT | 213.5769 ms TTFT | 308.71,159 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.2K | — | 190 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 40.1K | — | 404 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 37K | — | 872 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.9K | — | 2,151 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,146 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.1K | — | 19,135 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.1K | — | 19,615 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 208.6 | 209.1 | 37 ms | 4.8 ms | 1.0 | 1 | 6,262 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 355.5 | 178.5 | 67 ms | 5.6 ms | 2.0 | 2 | 10,663 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 589.0 | 148.3 | 76 ms | 6.7 ms | 4.0 | 4 | 17,669 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 905.7 | 113.6 | 159 ms | 8.8 ms | 8.0 | 0 | 27,168 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,449.5 | 91.4 | 199 ms | 10.9 ms | 16.0 | 0 | 43,488 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,158.1 | 68.1 | 317 ms | 14.7 ms | 32.0 | 0 | 64,736 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,232.3 | 51.1 | 479 ms | 19.6 ms | 64.0 | 0 | 96,960 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 204.6 | 205.3 | 63 ms | 4.9 ms | 1.0 | 1 | 6,137 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 347.0 | 174.7 | 120 ms | 5.7 ms | 2.0 | 2 | 10,410 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 570.1 | 144.0 | 155 ms | 6.9 ms | 4.0 | 4 | 17,102 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 864.4 | 108.4 | 218 ms | 9.2 ms | 8.0 | 0 | 25,928 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,362.3 | 85.8 | 262 ms | 11.7 ms | 16.0 | 0 | 40,864 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,967.5 | 61.9 | 475 ms | 16.1 ms | 32.0 | 0 | 59,008 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,800.2 | 44.2 | 681 ms | 22.6 ms | 64.0 | 0 | 83,968 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 197.7 | 198.8 | 115 ms | 5.0 ms | 1.0 | 1 | 5,932 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 332.5 | 167.4 | 132 ms | 6.0 ms | 2.0 | 2 | 9,974 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 537.4 | 136.1 | 203 ms | 7.3 ms | 4.0 | 4 | 16,119 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 795.7 | 99.7 | 357 ms | 10.0 ms | 8.0 | 0 | 23,864 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,207.7 | 76.0 | 510 ms | 13.2 ms | 16.0 | 0 | 36,224 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,666.6 | 52.4 | 842 ms | 19.1 ms | 32.0 | 0 | 49,984 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 184.9 | 186.3 | 168 ms | 5.4 ms | 1.0 | 1 | 5,547 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 306.5 | 155.0 | 208 ms | 6.5 ms | 2.0 | 2 | 9,195 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 483.5 | 123.0 | 249 ms | 8.1 ms | 3.9 | 4 | 14,501 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 690.1 | 86.4 | 565 ms | 11.6 ms | 8.0 | 0 | 20,696 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 988.9 | 62.0 | 965 ms | 16.1 ms | 16.0 | 0 | 29,664 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 165.2 | 167.7 | 300 ms | 6.0 ms | 1.0 | 1 | 4,955 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 264.2 | 135.3 | 375 ms | 7.4 ms | 1.9 | 2 | 7,925 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 406.0 | 101.7 | 558 ms | 9.8 ms | 4.0 | 0 | 12,180 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 540.8 | 67.6 | 878 ms | 14.8 ms | 8.0 | 0 | 16,224 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 135.7 | 141.0 | 576 ms | 7.1 ms | 1.0 | 1 | 4,074 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 213.5 | 107.1 | 769 ms | 9.3 ms | 2.0 | 0 | 6,406 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 308.7 | 77.4 | 1,159 ms | 12.9 ms | 4.0 | 0 | 9,260 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.