Speed test matrix
Laguna-XS-2.1-INT4 on H200 NVL 141GB
vllm · 0.26.0 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.880811 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1 --max-num-batched-tokens 32768 --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context34.6K237 ms TTFT
16K context34.2K474 ms TTFT
32K context31.8K1,013 ms TTFT
64K context26.7K2,402 ms TTFT
128K context19.9K6,454 ms TTFT
251K context13.1K19,249 ms TTFT
256K context12.9K19,942 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 150.858 ms TTFT | 243.463 ms TTFT | 426.9117 ms TTFT | 638.8187 ms TTFT | 982.6254 ms TTFT | 1,214.0346 ms TTFT | 1,478.6562 ms TTFT | 1,695.8981 ms TTFT |
| 16K | 126.287 ms TTFT | 220.9102 ms TTFT | 363.4153 ms TTFT | 544.8226 ms TTFT | 830.0294 ms TTFT | 1,003.2464 ms TTFT | 1,160.6787 ms TTFT | 1,302.81,416 ms TTFT |
| 32K | 119.9104 ms TTFT | 209.2125 ms TTFT | 332.8210 ms TTFT | 480.9291 ms TTFT | 690.3404 ms TTFT | 801.0725 ms TTFT | 911.71,267 ms TTFT | Over capacity |
| 64K | 114.4152 ms TTFT | 193.1230 ms TTFT | 289.2321 ms TTFT | 396.9546 ms TTFT | 525.7897 ms TTFT | 592.71,397 ms TTFT | Over capacity | Over capacity |
| 128K | 103.9285 ms TTFT | 162.8376 ms TTFT | 228.5541 ms TTFT | 289.2914 ms TTFT | 353.11,576 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 84.7546 ms TTFT | 124.7751 ms TTFT | 162.21,118 ms TTFT | 191.61,914 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.6K | — | 237 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.2K | — | 474 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.8K | — | 1,013 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.7K | — | 2,402 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.9K | — | 6,454 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.1K | — | 19,249 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,942 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 150.8 | 146.8 | 58 ms | 6.8 ms | 1.0 | 1 | 4,524 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 243.4 | 124.0 | 63 ms | 8.1 ms | 2.0 | 1 | 7,301 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 426.9 | 107.7 | 117 ms | 9.3 ms | 4.0 | 0 | 12,806 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 638.8 | 79.6 | 187 ms | 12.6 ms | 8.0 | 0 | 19,159 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 982.6 | 61.0 | 254 ms | 16.4 ms | 16.0 | 0 | 29,492 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,214.0 | 37.8 | 346 ms | 26.4 ms | 32.0 | 0 | 36,392 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,478.6 | 23.1 | 562 ms | 43.3 ms | 64.0 | 0 | 44,309 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,695.8 | 13.3 | 981 ms | 75.0 ms | 128.0 | 0 | 50,779 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 126.2 | 128.9 | 87 ms | 7.8 ms | 1.0 | 1 | 3,786 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 220.9 | 110.7 | 102 ms | 9.0 ms | 2.0 | 0 | 6,628 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 363.4 | 90.9 | 153 ms | 11.0 ms | 4.0 | 0 | 10,900 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 544.8 | 68.1 | 226 ms | 14.7 ms | 8.0 | 0 | 16,340 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 830.0 | 51.9 | 294 ms | 19.3 ms | 16.0 | 0 | 24,898 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,003.2 | 31.4 | 464 ms | 31.8 ms | 32.0 | 0 | 30,084 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,160.6 | 18.2 | 787 ms | 54.9 ms | 64.0 | 0 | 34,791 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,302.8 | 10.3 | 1,416 ms | 97.3 ms | 128.0 | 0 | 39,068 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 119.9 | 120.0 | 104 ms | 8.3 ms | 1.0 | 0 | 3,597 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 209.2 | 104.8 | 125 ms | 9.5 ms | 2.0 | 0 | 6,277 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 332.8 | 83.5 | 210 ms | 12.0 ms | 4.0 | 0 | 9,983 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 480.9 | 60.2 | 291 ms | 16.6 ms | 8.0 | 0 | 14,423 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 690.3 | 43.3 | 404 ms | 23.1 ms | 16.0 | 0 | 20,700 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 801.0 | 25.2 | 725 ms | 39.7 ms | 32.0 | 0 | 24,031 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 911.7 | 14.3 | 1,267 ms | 69.8 ms | 64.0 | 0 | 27,309 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 114.4 | 114.6 | 152 ms | 8.7 ms | 1.0 | 0 | 3,431 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 193.1 | 96.7 | 230 ms | 10.3 ms | 2.0 | 0 | 5,792 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 289.2 | 72.4 | 321 ms | 13.8 ms | 4.0 | 0 | 8,675 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 396.9 | 49.7 | 546 ms | 20.1 ms | 8.0 | 0 | 11,903 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 525.7 | 33.0 | 897 ms | 30.3 ms | 16.0 | 0 | 15,761 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 592.7 | 18.6 | 1,397 ms | 53.6 ms | 32.0 | 0 | 17,770 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 103.9 | 104.1 | 285 ms | 9.6 ms | 1.0 | 0 | 3,118 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 162.8 | 81.6 | 376 ms | 12.3 ms | 2.0 | 0 | 4,883 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 228.5 | 57.3 | 541 ms | 17.5 ms | 4.0 | 0 | 6,854 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 289.2 | 36.3 | 914 ms | 27.5 ms | 8.0 | 0 | 8,672 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 353.1 | 22.2 | 1,576 ms | 45.1 ms | 16.0 | 0 | 10,585 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 84.7 | 85.0 | 546 ms | 11.8 ms | 1.0 | 0 | 2,542 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 124.7 | 62.7 | 751 ms | 16.0 ms | 2.0 | 0 | 3,742 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 162.2 | 40.9 | 1,118 ms | 24.5 ms | 4.0 | 0 | 4,866 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 191.6 | 24.1 | 1,914 ms | 41.6 ms | 8.0 | 0 | 5,747 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.