Speed test matrix
Laguna-XS-2.1-INT4 on H200 NVL 141GB
vllm · 0.26.0 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.874835 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context35.2K233 ms TTFT
16K context34.1K475 ms TTFT
32K context31.1K1,035 ms TTFT
64K context26.6K2,416 ms TTFT
128K context19.7K6,524 ms TTFT
251K context13K19,352 ms TTFT
256K context12.7K20,162 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 188.544 ms TTFT | 354.363 ms TTFT | 655.685 ms TTFT | 1,125.3137 ms TTFT | 1,791.2216 ms TTFT | 2,708.3331 ms TTFT | 3,911.0499 ms TTFT | 5,199.9778 ms TTFT |
| 16K | 185.572 ms TTFT | 347.791 ms TTFT | 632.0148 ms TTFT | 1,062.3169 ms TTFT | 1,631.7382 ms TTFT | 2,352.0532 ms TTFT | 3,208.6782 ms TTFT | 4,026.61,323 ms TTFT |
| 32K | 180.9114 ms TTFT | 332.8123 ms TTFT | 586.9166 ms TTFT | 946.7354 ms TTFT | 1,384.5588 ms TTFT | 1,879.9845 ms TTFT | 2,393.81,406 ms TTFT | 2,808.12,322 ms TTFT |
| 64K | 172.5182 ms TTFT | 309.0202 ms TTFT | 518.7211 ms TTFT | 784.3504 ms TTFT | 1,059.7923 ms TTFT | 1,328.61,448 ms TTFT | 1,557.32,394 ms TTFT | Over capacity |
| 128K | 158.6296 ms TTFT | 269.4359 ms TTFT | 424.9574 ms TTFT | 578.2878 ms TTFT | 723.21,589 ms TTFT | 836.13,038 ms TTFT | Over capacity | Over capacity |
| 251K | 136.5588 ms TTFT | 220.2782 ms TTFT | 310.51,136 ms TTFT | 386.71,759 ms TTFT | 447.33,206 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.2K | — | 233 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.1K | — | 475 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,035 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.6K | — | 2,416 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.7K | — | 6,524 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13K | — | 19,352 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.7K | — | 20,162 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 188.5 | 188.9 | 44 ms | 5.3 ms | 1.0 | 1 | 5,656 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 354.3 | 177.8 | 63 ms | 5.6 ms | 2.0 | 2 | 10,630 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 655.6 | 164.9 | 85 ms | 6.1 ms | 4.0 | 4 | 19,667 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,125.3 | 142.3 | 137 ms | 7.0 ms | 8.0 | 8 | 33,758 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,791.2 | 112.6 | 216 ms | 8.9 ms | 16.0 | 0 | 53,728 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,708.3 | 85.3 | 331 ms | 11.7 ms | 32.0 | 0 | 81,249 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,911.0 | 61.7 | 499 ms | 16.2 ms | 64.0 | 0 | 117,312 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,199.9 | 40.9 | 778 ms | 24.4 ms | 128.0 | 0 | 156,032 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 185.5 | 186.1 | 72 ms | 5.4 ms | 1.0 | 1 | 5,567 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 347.7 | 174.6 | 91 ms | 5.7 ms | 2.0 | 2 | 10,430 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 632.0 | 159.2 | 148 ms | 6.3 ms | 4.0 | 4 | 18,958 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,062.3 | 134.0 | 169 ms | 7.5 ms | 8.0 | 8 | 31,867 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,631.7 | 102.6 | 382 ms | 9.7 ms | 16.0 | 0 | 48,944 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,352.0 | 74.0 | 532 ms | 13.5 ms | 32.0 | 0 | 70,561 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,208.6 | 50.4 | 782 ms | 19.9 ms | 64.0 | 0 | 96,256 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,026.6 | 31.6 | 1,323 ms | 31.6 ms | 128.0 | 0 | 120,832 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 180.9 | 182.0 | 114 ms | 5.5 ms | 1.0 | 1 | 5,428 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 332.8 | 167.4 | 123 ms | 6.0 ms | 2.0 | 2 | 9,985 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 586.9 | 148.1 | 166 ms | 6.8 ms | 3.9 | 4 | 17,605 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 946.7 | 118.7 | 354 ms | 8.4 ms | 8.0 | 0 | 28,400 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,384.5 | 86.9 | 588 ms | 11.5 ms | 16.0 | 0 | 41,537 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,879.9 | 58.9 | 845 ms | 17.0 ms | 32.0 | 0 | 56,384 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,393.8 | 37.4 | 1,406 ms | 26.7 ms | 64.0 | 0 | 71,808 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,808.1 | 22.0 | 2,322 ms | 45.4 ms | 128.0 | 0 | 84,224 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 172.5 | 174.1 | 182 ms | 5.7 ms | 1.0 | 1 | 5,176 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 309.0 | 156.0 | 202 ms | 6.4 ms | 2.0 | 2 | 9,269 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 518.7 | 130.6 | 211 ms | 7.7 ms | 4.0 | 4 | 15,559 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 784.3 | 98.2 | 504 ms | 10.2 ms | 8.0 | 0 | 23,528 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,059.7 | 66.4 | 923 ms | 15.1 ms | 16.0 | 0 | 31,792 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,328.6 | 41.6 | 1,448 ms | 24.1 ms | 32.0 | 0 | 39,840 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,557.3 | 24.6 | 2,394 ms | 40.7 ms | 64.0 | 0 | 46,720 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.6 | 160.8 | 296 ms | 6.2 ms | 1.0 | 1 | 4,758 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 269.4 | 137.7 | 359 ms | 7.3 ms | 2.0 | 2 | 8,081 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 424.9 | 106.3 | 574 ms | 9.4 ms | 4.0 | 0 | 12,748 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 578.2 | 72.4 | 878 ms | 13.8 ms | 8.0 | 0 | 17,344 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 723.2 | 45.3 | 1,589 ms | 22.1 ms | 16.0 | 0 | 21,696 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 836.1 | 26.2 | 3,038 ms | 38.2 ms | 32.0 | 0 | 25,056 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 136.5 | 141.8 | 588 ms | 7.1 ms | 1.0 | 1 | 4,097 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 220.2 | 110.5 | 782 ms | 9.0 ms | 2.0 | 0 | 6,606 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 310.5 | 77.9 | 1,136 ms | 12.8 ms | 4.0 | 0 | 9,312 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 386.7 | 48.5 | 1,759 ms | 20.6 ms | 8.0 | 0 | 11,600 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 447.3 | 28.0 | 3,206 ms | 35.7 ms | 16.0 | 0 | 13,408 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.