Speed test matrix
Laguna-XS.2-INT4 on H200 NVL 141GB
vllm · 0.26.0 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS.2-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.874835 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context36.2K227 ms TTFT
16K context33.3K487 ms TTFT
32K context31.3K1,031 ms TTFT
64K context26.4K2,430 ms TTFT
128K context19.8K6,491 ms TTFT
251K context12.9K19,417 ms TTFT
256K context12.7K20,160 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 187.851 ms TTFT | 354.066 ms TTFT | 658.173 ms TTFT | 1,132.1138 ms TTFT | 1,786.0226 ms TTFT | 2,708.4356 ms TTFT | 3,907.9515 ms TTFT | 5,181.7779 ms TTFT |
| 16K | 184.885 ms TTFT | 346.686 ms TTFT | 629.9116 ms TTFT | 1,060.2154 ms TTFT | 1,632.2383 ms TTFT | 2,361.3419 ms TTFT | 3,218.7667 ms TTFT | 4,025.71,206 ms TTFT |
| 32K | 180.8116 ms TTFT | 333.1118 ms TTFT | 588.3165 ms TTFT | 947.1355 ms TTFT | 1,392.2583 ms TTFT | 1,882.2844 ms TTFT | 2,377.51,227 ms TTFT | 2,792.32,368 ms TTFT |
| 64K | 172.7169 ms TTFT | 307.3232 ms TTFT | 518.6206 ms TTFT | 782.1565 ms TTFT | 1,060.8879 ms TTFT | 1,332.41,561 ms TTFT | 1,563.32,302 ms TTFT | Over capacity |
| 128K | 158.3304 ms TTFT | 267.6397 ms TTFT | 421.7616 ms TTFT | 578.5929 ms TTFT | 724.41,529 ms TTFT | 838.72,983 ms TTFT | Over capacity | Over capacity |
| 251K | 136.0598 ms TTFT | 220.0826 ms TTFT | 309.51,101 ms TTFT | 386.41,833 ms TTFT | 447.73,162 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 36.2K | — | 227 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.3K | — | 487 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.3K | — | 1,031 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.4K | — | 2,430 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.8K | — | 6,491 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.9K | — | 19,417 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.7K | — | 20,160 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 187.8 | 188.4 | 51 ms | 5.3 ms | 1.0 | 1 | 5,634 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 354.0 | 177.7 | 66 ms | 5.6 ms | 2.0 | 2 | 10,620 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 658.1 | 165.4 | 73 ms | 6.0 ms | 4.0 | 4 | 19,742 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,132.1 | 143.1 | 138 ms | 7.0 ms | 7.8 | 8 | 33,965 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,786.0 | 112.4 | 226 ms | 8.9 ms | 16.0 | 0 | 53,568 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,708.4 | 85.3 | 356 ms | 11.7 ms | 32.0 | 0 | 81,248 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,907.9 | 61.5 | 515 ms | 16.2 ms | 64.0 | 0 | 117,248 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,181.7 | 40.7 | 779 ms | 24.6 ms | 128.0 | 0 | 155,392 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 184.8 | 185.6 | 85 ms | 5.4 ms | 1.0 | 1 | 5,547 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 346.6 | 174.1 | 86 ms | 5.7 ms | 2.0 | 2 | 10,398 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 629.9 | 159.0 | 116 ms | 6.3 ms | 4.0 | 4 | 18,896 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,060.2 | 134.2 | 154 ms | 7.5 ms | 7.8 | 8 | 31,806 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,632.2 | 102.8 | 383 ms | 9.7 ms | 16.0 | 0 | 48,976 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,361.3 | 74.2 | 419 ms | 13.5 ms | 32.0 | 0 | 70,848 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,218.7 | 50.7 | 667 ms | 19.7 ms | 64.0 | 0 | 96,576 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,025.7 | 31.8 | 1,206 ms | 31.5 ms | 128.0 | 0 | 120,832 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 180.8 | 181.9 | 116 ms | 5.5 ms | 1.0 | 1 | 5,424 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 333.1 | 167.6 | 118 ms | 6.0 ms | 2.0 | 2 | 10,000 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 588.3 | 148.7 | 165 ms | 6.7 ms | 4.0 | 4 | 17,648 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 947.1 | 118.8 | 355 ms | 8.4 ms | 8.0 | 0 | 28,408 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,392.2 | 87.4 | 583 ms | 11.4 ms | 16.0 | 0 | 41,760 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,882.2 | 58.9 | 844 ms | 17.0 ms | 32.0 | 0 | 56,448 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,377.5 | 37.4 | 1,227 ms | 26.7 ms | 64.0 | 0 | 71,296 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,792.3 | 22.0 | 2,368 ms | 45.5 ms | 128.0 | 0 | 83,712 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 172.7 | 174.3 | 169 ms | 5.7 ms | 1.0 | 1 | 5,180 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 307.3 | 155.6 | 232 ms | 6.4 ms | 2.0 | 2 | 9,220 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 518.6 | 130.8 | 206 ms | 7.6 ms | 4.0 | 4 | 15,556 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 782.1 | 98.1 | 565 ms | 10.2 ms | 8.0 | 0 | 23,456 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,060.8 | 66.4 | 879 ms | 15.0 ms | 16.0 | 0 | 31,840 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,332.4 | 41.7 | 1,561 ms | 24.0 ms | 32.0 | 0 | 39,968 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,563.3 | 24.7 | 2,302 ms | 40.5 ms | 64.0 | 0 | 46,848 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.3 | 160.8 | 304 ms | 6.2 ms | 1.0 | 1 | 4,749 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 267.6 | 136.8 | 397 ms | 7.3 ms | 2.0 | 2 | 8,032 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 421.7 | 105.6 | 616 ms | 9.5 ms | 4.0 | 0 | 12,656 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 578.5 | 72.3 | 929 ms | 13.8 ms | 8.0 | 0 | 17,360 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 724.4 | 45.3 | 1,529 ms | 22.1 ms | 16.0 | 0 | 21,728 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 838.7 | 26.3 | 2,983 ms | 38.0 ms | 32.0 | 0 | 25,152 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 136.0 | 141.5 | 598 ms | 7.1 ms | 1.0 | 1 | 4,081 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 220.0 | 110.5 | 826 ms | 9.0 ms | 2.0 | 0 | 6,598 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 309.5 | 77.7 | 1,101 ms | 12.9 ms | 4.0 | 0 | 9,288 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 386.4 | 48.4 | 1,833 ms | 20.7 ms | 8.0 | 0 | 11,592 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 447.7 | 28.1 | 3,162 ms | 35.6 ms | 16.0 | 0 | 13,424 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.