Speed test matrix
Laguna-XS.2-FP8 on H200 NVL 141GB
vllm · 0.26.0 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 "$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS.2-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.877197 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context41.2K199 ms TTFT
16K context39.2K413 ms TTFT
32K context36.4K884 ms TTFT
64K context30.1K2,136 ms TTFT
128K context21.5K5,974 ms TTFT
251K context13.7K18,326 ms TTFT
256K context13.4K19,089 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 186.552 ms TTFT | 352.567 ms TTFT | 634.774 ms TTFT | 1,077.769 ms TTFT | 1,683.6234 ms TTFT | 2,606.3412 ms TTFT | 3,809.3529 ms TTFT | 5,258.4813 ms TTFT |
| 16K | 184.388 ms TTFT | 345.792 ms TTFT | 610.8154 ms TTFT | 1,010.2145 ms TTFT | 1,542.4320 ms TTFT | 2,274.8421 ms TTFT | 3,154.5700 ms TTFT | 4,065.71,376 ms TTFT |
| 32K | 180.3113 ms TTFT | 331.3134 ms TTFT | 568.8150 ms TTFT | 908.3334 ms TTFT | 1,319.1478 ms TTFT | 1,825.4859 ms TTFT | 2,351.01,283 ms TTFT | Over capacity |
| 64K | 172.3165 ms TTFT | 307.4178 ms TTFT | 503.7217 ms TTFT | 758.4422 ms TTFT | 1,023.0934 ms TTFT | 1,302.61,487 ms TTFT | 1,547.32,399 ms TTFT | Over capacity |
| 128K | 157.7304 ms TTFT | 268.1352 ms TTFT | 412.4565 ms TTFT | 565.3894 ms TTFT | 703.91,620 ms TTFT | 831.02,940 ms TTFT | Over capacity | Over capacity |
| 251K | 135.8573 ms TTFT | 219.8786 ms TTFT | 304.71,129 ms TTFT | 380.61,838 ms TTFT | 441.23,498 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.2K | — | 199 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.2K | — | 413 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.4K | — | 884 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.1K | — | 2,136 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.5K | — | 5,974 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.7K | — | 18,326 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.4K | — | 19,089 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 186.5 | 187.1 | 52 ms | 5.3 ms | 1.0 | 1 | 5,595 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 352.5 | 177.0 | 67 ms | 5.7 ms | 2.0 | 2 | 10,574 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 634.7 | 159.7 | 74 ms | 6.3 ms | 3.9 | 4 | 19,039 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,077.7 | 136.3 | 69 ms | 7.3 ms | 8.0 | 8 | 32,331 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,683.6 | 105.9 | 234 ms | 9.4 ms | 16.0 | 0 | 50,496 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,606.3 | 82.2 | 412 ms | 12.2 ms | 32.0 | 0 | 78,176 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,809.3 | 60.0 | 529 ms | 16.7 ms | 64.0 | 0 | 114,304 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,258.4 | 41.2 | 813 ms | 24.3 ms | 128.0 | 0 | 157,696 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 184.3 | 185.0 | 88 ms | 5.4 ms | 1.0 | 1 | 5,527 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 345.7 | 173.7 | 92 ms | 5.8 ms | 2.0 | 2 | 10,370 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 610.8 | 154.2 | 154 ms | 6.5 ms | 3.9 | 4 | 18,323 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,010.2 | 127.7 | 145 ms | 7.8 ms | 8.0 | 8 | 30,306 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,542.4 | 97.0 | 320 ms | 10.3 ms | 16.0 | 0 | 46,272 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,274.8 | 71.6 | 421 ms | 14.0 ms | 32.0 | 0 | 68,250 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,154.5 | 49.7 | 700 ms | 20.1 ms | 64.0 | 0 | 94,656 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,065.7 | 31.9 | 1,376 ms | 31.3 ms | 128.0 | 0 | 121,984 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 180.3 | 181.3 | 113 ms | 5.5 ms | 1.0 | 1 | 5,408 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 331.3 | 166.8 | 134 ms | 6.0 ms | 2.0 | 2 | 9,944 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 568.8 | 143.4 | 150 ms | 7.0 ms | 3.9 | 4 | 17,061 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 908.3 | 113.9 | 334 ms | 8.8 ms | 8.0 | 0 | 27,249 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,319.1 | 82.8 | 478 ms | 12.1 ms | 16.0 | 0 | 39,568 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,825.4 | 57.1 | 859 ms | 17.5 ms | 32.0 | 0 | 54,752 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,351.0 | 36.8 | 1,283 ms | 27.2 ms | 64.0 | 0 | 70,528 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 172.3 | 173.8 | 165 ms | 5.8 ms | 1.0 | 1 | 5,169 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 307.4 | 155.0 | 178 ms | 6.5 ms | 2.0 | 2 | 9,223 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 503.7 | 127.6 | 217 ms | 7.8 ms | 4.0 | 4 | 15,118 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 758.4 | 95.0 | 422 ms | 10.5 ms | 8.0 | 0 | 22,753 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,023.0 | 64.1 | 934 ms | 15.6 ms | 16.0 | 0 | 30,688 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,302.6 | 40.9 | 1,487 ms | 24.5 ms | 32.0 | 0 | 39,072 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,547.3 | 24.4 | 2,399 ms | 40.9 ms | 64.0 | 0 | 46,400 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 157.7 | 160.3 | 304 ms | 6.2 ms | 1.0 | 1 | 4,734 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 268.1 | 137.2 | 352 ms | 7.3 ms | 2.0 | 2 | 8,045 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 412.4 | 103.3 | 565 ms | 9.7 ms | 4.0 | 0 | 12,381 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 565.3 | 70.7 | 894 ms | 14.1 ms | 8.0 | 0 | 16,960 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 703.9 | 44.1 | 1,620 ms | 22.7 ms | 16.0 | 0 | 21,120 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 831.0 | 26.0 | 2,940 ms | 38.5 ms | 32.0 | 0 | 24,928 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 135.8 | 141.1 | 573 ms | 7.1 ms | 1.0 | 1 | 4,075 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 219.8 | 110.2 | 786 ms | 9.1 ms | 2.0 | 0 | 6,594 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 304.7 | 76.4 | 1,129 ms | 13.1 ms | 4.0 | 0 | 9,140 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 380.6 | 47.7 | 1,838 ms | 21.0 ms | 8.0 | 0 | 11,416 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 441.2 | 27.7 | 3,498 ms | 36.1 ms | 16.0 | 0 | 13,232 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.