Speed test matrix
Laguna-XS-2.1-FP8 on H200 NVL 141GB
vllm · 0.26.0 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 "$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.877197 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context42.4K193 ms TTFT
16K context38.5K421 ms TTFT
32K context36.3K888 ms TTFT
64K context13.5K4,767 ms TTFT
128K context21.5K5,977 ms TTFT
251K context13.8K18,282 ms TTFT
256K context13.5K19,001 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 187.151 ms TTFT | 352.967 ms TTFT | 634.678 ms TTFT | 1,074.9141 ms TTFT | 1,680.0228 ms TTFT | 2,589.9357 ms TTFT | 3,784.9629 ms TTFT | 5,228.0789 ms TTFT |
| 16K | 184.673 ms TTFT | 346.985 ms TTFT | 616.2126 ms TTFT | 1,020.1144 ms TTFT | 1,545.6318 ms TTFT | 2,253.7385 ms TTFT | 3,139.9985 ms TTFT | 4,061.81,171 ms TTFT |
| 32K | 180.4105 ms TTFT | 333.1121 ms TTFT | 570.2155 ms TTFT | 915.0296 ms TTFT | 1,316.8494 ms TTFT | 1,820.4856 ms TTFT | 2,339.51,328 ms TTFT | Over capacity |
| 64K | 172.6180 ms TTFT | 308.4211 ms TTFT | 505.4233 ms TTFT | 759.7608 ms TTFT | 1,017.9900 ms TTFT | 1,297.11,502 ms TTFT | 1,542.22,620 ms TTFT | Over capacity |
| 128K | 158.2305 ms TTFT | 268.8353 ms TTFT | 414.0679 ms TTFT | 567.2920 ms TTFT | 703.31,671 ms TTFT | 825.73,024 ms TTFT | Over capacity | Over capacity |
| 251K | 136.3507 ms TTFT | 219.1749 ms TTFT | 305.91,110 ms TTFT | 380.71,833 ms TTFT | 439.93,212 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.4K | — | 193 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.5K | — | 421 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.3K | — | 888 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.5K | — | 4,767 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.5K | — | 5,977 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.8K | — | 18,282 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.5K | — | 19,001 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 187.1 | 187.7 | 51 ms | 5.3 ms | 1.0 | 1 | 5,617 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 352.9 | 177.1 | 67 ms | 5.6 ms | 2.0 | 2 | 10,584 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 634.6 | 159.8 | 78 ms | 6.3 ms | 3.9 | 4 | 19,039 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,074.9 | 136.3 | 141 ms | 7.3 ms | 8.0 | 8 | 32,246 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,680.0 | 105.7 | 228 ms | 9.5 ms | 16.0 | 0 | 50,400 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,589.9 | 81.7 | 357 ms | 12.2 ms | 32.0 | 0 | 77,696 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,784.9 | 59.7 | 629 ms | 16.8 ms | 64.0 | 0 | 113,536 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,228.0 | 41.1 | 789 ms | 24.3 ms | 128.0 | 0 | 156,800 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 184.6 | 185.5 | 73 ms | 5.4 ms | 1.0 | 1 | 5,539 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 346.9 | 174.1 | 85 ms | 5.7 ms | 2.0 | 2 | 10,404 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 616.2 | 154.9 | 126 ms | 6.5 ms | 4.0 | 4 | 18,487 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,020.1 | 129.1 | 144 ms | 7.7 ms | 8.0 | 8 | 30,598 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,545.6 | 97.0 | 318 ms | 10.3 ms | 16.0 | 0 | 46,368 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,253.7 | 70.9 | 385 ms | 14.1 ms | 32.0 | 0 | 67,616 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,139.9 | 49.4 | 985 ms | 20.3 ms | 64.0 | 0 | 94,208 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,061.8 | 32.0 | 1,171 ms | 31.3 ms | 128.0 | 0 | 121,856 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 180.4 | 181.4 | 105 ms | 5.5 ms | 1.0 | 1 | 5,413 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 333.1 | 167.4 | 121 ms | 6.0 ms | 2.0 | 2 | 9,991 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 570.2 | 143.8 | 155 ms | 7.0 ms | 4.0 | 4 | 17,104 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 915.0 | 114.5 | 296 ms | 8.7 ms | 8.0 | 0 | 27,448 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,316.8 | 82.6 | 494 ms | 12.1 ms | 16.0 | 0 | 39,504 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,820.4 | 57.0 | 856 ms | 17.5 ms | 32.0 | 0 | 54,592 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,339.5 | 36.6 | 1,328 ms | 27.3 ms | 64.0 | 0 | 70,144 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 172.6 | 174.2 | 180 ms | 5.7 ms | 1.0 | 1 | 5,178 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 308.4 | 155.8 | 211 ms | 6.4 ms | 2.0 | 2 | 9,250 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 505.4 | 128.1 | 233 ms | 7.8 ms | 4.0 | 4 | 15,163 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 759.7 | 95.2 | 608 ms | 10.5 ms | 8.0 | 0 | 22,793 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,017.9 | 63.8 | 900 ms | 15.7 ms | 16.0 | 0 | 30,528 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,297.1 | 40.6 | 1,502 ms | 24.6 ms | 32.0 | 0 | 38,912 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,542.2 | 24.4 | 2,620 ms | 41.0 ms | 64.0 | 0 | 46,272 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.2 | 160.7 | 305 ms | 6.2 ms | 1.0 | 1 | 4,747 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 268.8 | 137.2 | 353 ms | 7.3 ms | 2.0 | 2 | 8,064 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 414.0 | 103.6 | 679 ms | 9.7 ms | 4.0 | 0 | 12,421 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 567.2 | 70.9 | 920 ms | 14.1 ms | 8.0 | 0 | 17,008 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 703.3 | 44.1 | 1,671 ms | 22.7 ms | 16.0 | 0 | 21,088 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 825.7 | 25.9 | 3,024 ms | 38.7 ms | 32.0 | 0 | 24,768 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 136.3 | 140.6 | 507 ms | 7.1 ms | 1.0 | 1 | 4,090 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 219.1 | 109.9 | 749 ms | 9.1 ms | 2.0 | 0 | 6,576 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 305.9 | 76.7 | 1,110 ms | 13.0 ms | 4.0 | 0 | 9,176 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 380.7 | 47.7 | 1,833 ms | 21.0 ms | 8.0 | 0 | 11,424 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 439.9 | 27.5 | 3,212 ms | 36.3 ms | 16.0 | 0 | 13,184 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.