Speed test matrix
Laguna-S-2.1-INT4 on H200 NVL 141GB ×2
vllm · 0.27.1 · INT48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.1
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-vllm/bin/vllm" serve poolside/Laguna-S-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 2 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 1048576 --gpu-memory-utilization 0.860342 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context24K342 ms TTFT
16K context24.2K670 ms TTFT
32K context23.6K1,364 ms TTFT
64K context22K2,915 ms TTFT
128K context18.5K6,943 ms TTFT
256K context13.8K18,534 ms TTFT
512K context9,20555,686 ms TTFT
1019K context5,567183,216 ms TTFT
1024K context5,529185,372 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 155.341 ms TTFT | 283.951 ms TTFT | 485.568 ms TTFT | 792.8140 ms TTFT | 1,243.6178 ms TTFT | 1,904.7291 ms TTFT | 2,878.6405 ms TTFT | 4,021.1694 ms TTFT |
| 16K | 153.786 ms TTFT | 279.672 ms TTFT | 482.989 ms TTFT | 764.5201 ms TTFT | 1,198.0291 ms TTFT | 1,831.9422 ms TTFT | 2,638.7666 ms TTFT | 3,605.81,168 ms TTFT |
| 32K | 151.0112 ms TTFT | 273.6112 ms TTFT | 465.4178 ms TTFT | 734.7258 ms TTFT | 1,124.5401 ms TTFT | 1,666.0672 ms TTFT | 2,321.81,130 ms TTFT | Over capacity |
| 64K | 146.8158 ms TTFT | 260.7192 ms TTFT | 440.8338 ms TTFT | 678.9518 ms TTFT | 996.8854 ms TTFT | 1,392.81,407 ms TTFT | Over capacity | Over capacity |
| 128K | 138.4273 ms TTFT | 242.3344 ms TTFT | 400.7586 ms TTFT | 583.8889 ms TTFT | 817.61,489 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 124.0526 ms TTFT | 209.1734 ms TTFT | 334.41,102 ms TTFT | 460.31,835 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 105.31,216 ms TTFT | 166.01,615 ms TTFT | 252.72,510 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 75.6Requests queued2,456 ms TTFT | 107.6Requests queued3,548 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24K | — | 342 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.2K | — | 670 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 23.6K | — | 1,364 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22K | — | 2,915 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,943 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.8K | — | 18,534 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,205.0 | — | 55,686 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 5,567.0 | — | 183,216 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,529.0 | — | 185,372 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 155.3 | 155.7 | 41 ms | 6.4 ms | 1.0 | 1 | 4,657 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 283.9 | 142.5 | 51 ms | 7.0 ms | 2.0 | 2 | 8,518 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 485.5 | 123.2 | 68 ms | 8.1 ms | 4.0 | 4 | 14,562 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 792.8 | 99.5 | 140 ms | 10.1 ms | 8.0 | 0 | 23,784 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,243.6 | 78.2 | 178 ms | 12.8 ms | 16.0 | 0 | 37,296 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,904.7 | 60.0 | 291 ms | 16.7 ms | 32.0 | 0 | 57,120 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,878.6 | 45.3 | 405 ms | 22.1 ms | 64.0 | 0 | 86,337 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,021.1 | 31.6 | 694 ms | 31.6 ms | 128.0 | 0 | 120,576 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 153.7 | 154.4 | 86 ms | 6.5 ms | 1.0 | 1 | 4,612 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 279.6 | 140.7 | 72 ms | 7.1 ms | 2.0 | 2 | 8,388 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 482.9 | 122.9 | 89 ms | 8.1 ms | 4.0 | 4 | 14,485 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 764.5 | 97.0 | 201 ms | 10.3 ms | 8.0 | 0 | 22,931 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,198.0 | 75.3 | 291 ms | 13.3 ms | 16.0 | 0 | 35,935 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,831.9 | 57.6 | 422 ms | 17.4 ms | 32.0 | 0 | 54,944 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,638.7 | 41.5 | 666 ms | 24.1 ms | 64.0 | 0 | 79,164 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,605.8 | 28.4 | 1,168 ms | 35.2 ms | 128.0 | 0 | 108,160 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 151.0 | 152.1 | 112 ms | 6.6 ms | 1.0 | 1 | 4,531 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 273.6 | 138.0 | 112 ms | 7.2 ms | 1.9 | 2 | 8,206 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 465.4 | 116.6 | 178 ms | 8.6 ms | 4.0 | 0 | 13,961 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 734.7 | 92.1 | 258 ms | 10.9 ms | 8.0 | 0 | 22,040 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,124.5 | 70.6 | 401 ms | 14.2 ms | 16.0 | 0 | 33,728 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,666.0 | 52.4 | 672 ms | 19.1 ms | 32.0 | 0 | 49,984 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,321.8 | 36.5 | 1,130 ms | 27.4 ms | 64.0 | 0 | 69,632 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 146.8 | 148.3 | 158 ms | 6.7 ms | 1.0 | 1 | 4,404 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 260.7 | 132.7 | 192 ms | 7.5 ms | 2.0 | 2 | 7,822 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 440.8 | 110.4 | 338 ms | 9.1 ms | 4.0 | 0 | 13,220 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 678.9 | 85.1 | 518 ms | 11.7 ms | 8.0 | 0 | 20,368 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 996.8 | 62.6 | 854 ms | 16.0 ms | 16.0 | 0 | 29,905 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,392.8 | 43.8 | 1,407 ms | 22.8 ms | 32.0 | 0 | 41,760 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.4 | 140.4 | 273 ms | 7.1 ms | 1.0 | 1 | 4,150 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 242.3 | 121.4 | 344 ms | 8.2 ms | 2.0 | 0 | 7,268 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 400.7 | 100.5 | 586 ms | 10.0 ms | 4.0 | 0 | 12,021 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 583.8 | 73.2 | 889 ms | 13.7 ms | 8.0 | 0 | 17,512 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 817.6 | 51.4 | 1,489 ms | 19.5 ms | 16.0 | 0 | 24,529 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 124.0 | 181.9 | 526 ms | 6.1 ms | 1.0 | 1 | 3,719 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 209.1 | 105.0 | 734 ms | 9.5 ms | 2.0 | 0 | 6,272 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 334.4 | 83.9 | 1,102 ms | 11.9 ms | 4.0 | 0 | 10,036 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 460.3 | 57.8 | 1,835 ms | 17.3 ms | 8.0 | 0 | 13,816 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 105.3 | 105.9 | 1,216 ms | 9.4 ms | 1.0 | 0 | 3,159 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 166.0 | 83.6 | 1,615 ms | 12.0 ms | 2.0 | 0 | 4,980 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 252.7 | 63.7 | 2,510 ms | 15.7 ms | 4.0 | 0 | 7,580 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 75.6 | 80.8 | 2,456 ms | 12.4 ms | 0.9 | 0 | 2,267 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 107.6 | 59.8 | 3,548 ms | 16.7 ms | 1.8 | 0 | 3,227 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.