Speed test matrix
Laguna-XS.2-NVFP4 on H200 NVL 141GB
vllm · 0.26.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS.2-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.876591 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context32.2K255 ms TTFT
16K context31.7K511 ms TTFT
32K context29.8K1,081 ms TTFT
64K context25.8K2,487 ms TTFT
128K context19K6,742 ms TTFT
251K context12.7K19,804 ms TTFT
256K context12.4K20,600 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 160.957 ms TTFT | 308.773 ms TTFT | 577.879 ms TTFT | 1,009.498 ms TTFT | 1,636.4240 ms TTFT | 2,446.1292 ms TTFT | 3,383.4519 ms TTFT | 4,818.2708 ms TTFT |
| 16K | 159.187 ms TTFT | 304.093 ms TTFT | 563.4145 ms TTFT | 952.1251 ms TTFT | 1,505.2273 ms TTFT | 2,152.2401 ms TTFT | 2,848.6695 ms TTFT | 3,805.41,288 ms TTFT |
| 32K | 155.8115 ms TTFT | 293.7127 ms TTFT | 526.3145 ms TTFT | 869.1338 ms TTFT | 1,287.4617 ms TTFT | 1,743.6823 ms TTFT | 2,173.51,368 ms TTFT | 2,682.22,427 ms TTFT |
| 64K | 149.3172 ms TTFT | 273.2207 ms TTFT | 472.1382 ms TTFT | 722.7630 ms TTFT | 998.9928 ms TTFT | 1,261.41,409 ms TTFT | 1,466.92,338 ms TTFT | Over capacity |
| 128K | 138.8305 ms TTFT | 244.7397 ms TTFT | 390.4563 ms TTFT | 548.7936 ms TTFT | 693.71,676 ms TTFT | 808.63,031 ms TTFT | Over capacity | Over capacity |
| 251K | 123.5563 ms TTFT | 200.9800 ms TTFT | 291.71,102 ms TTFT | 371.51,821 ms TTFT | 436.63,180 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.2K | — | 255 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.7K | — | 511 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.8K | — | 1,081 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.8K | — | 2,487 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19K | — | 6,742 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.7K | — | 19,804 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.4K | — | 20,600 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 160.9 | 161.5 | 57 ms | 6.2 ms | 1.0 | 1 | 4,828 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 308.7 | 155.1 | 73 ms | 6.4 ms | 2.0 | 2 | 9,261 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 577.8 | 145.6 | 79 ms | 6.9 ms | 4.0 | 4 | 17,335 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,009.4 | 128.1 | 98 ms | 7.8 ms | 8.0 | 8 | 30,281 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,636.4 | 102.8 | 240 ms | 9.7 ms | 16.0 | 0 | 49,088 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,446.1 | 77.0 | 292 ms | 13.0 ms | 32.0 | 0 | 73,376 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,383.4 | 53.2 | 519 ms | 18.8 ms | 64.0 | 0 | 101,504 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,818.2 | 37.8 | 708 ms | 26.4 ms | 128.0 | 0 | 144,640 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 159.1 | 159.9 | 87 ms | 6.3 ms | 1.0 | 1 | 4,772 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 304.0 | 152.8 | 93 ms | 6.5 ms | 1.9 | 2 | 9,126 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 563.4 | 141.8 | 145 ms | 7.1 ms | 4.0 | 4 | 16,901 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 952.1 | 119.6 | 251 ms | 8.4 ms | 8.0 | 8 | 28,557 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,505.2 | 94.4 | 273 ms | 10.6 ms | 16.0 | 0 | 45,152 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,152.2 | 67.7 | 401 ms | 14.8 ms | 32.0 | 0 | 64,544 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,848.6 | 44.8 | 695 ms | 22.3 ms | 64.0 | 0 | 85,440 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,805.4 | 29.7 | 1,288 ms | 33.6 ms | 128.0 | 0 | 114,176 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.8 | 156.8 | 115 ms | 6.4 ms | 1.0 | 1 | 4,675 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 293.7 | 147.8 | 127 ms | 6.8 ms | 2.0 | 2 | 8,809 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 526.3 | 132.6 | 145 ms | 7.5 ms | 4.0 | 4 | 15,788 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 869.1 | 108.8 | 338 ms | 9.2 ms | 8.0 | 0 | 26,072 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,287.4 | 80.8 | 617 ms | 12.4 ms | 16.0 | 0 | 38,624 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,743.6 | 54.6 | 823 ms | 18.3 ms | 32.0 | 0 | 52,288 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,173.5 | 34.1 | 1,368 ms | 29.4 ms | 64.0 | 0 | 65,216 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,682.2 | 21.0 | 2,427 ms | 47.6 ms | 128.0 | 0 | 80,503 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 149.3 | 150.9 | 172 ms | 6.6 ms | 1.0 | 1 | 4,482 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 273.2 | 138.3 | 207 ms | 7.2 ms | 1.9 | 2 | 8,197 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 472.1 | 118.2 | 382 ms | 8.5 ms | 4.0 | 0 | 14,160 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 722.7 | 90.5 | 630 ms | 11.1 ms | 8.0 | 0 | 21,688 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 998.9 | 62.5 | 928 ms | 16.0 ms | 16.0 | 0 | 29,968 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,261.4 | 39.5 | 1,409 ms | 25.3 ms | 32.0 | 0 | 37,824 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,466.9 | 23.2 | 2,338 ms | 43.1 ms | 64.0 | 0 | 43,968 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.8 | 141.5 | 305 ms | 7.1 ms | 1.0 | 1 | 4,165 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.7 | 122.6 | 397 ms | 8.2 ms | 2.0 | 0 | 7,344 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 390.4 | 97.7 | 563 ms | 10.2 ms | 4.0 | 0 | 11,716 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 548.7 | 68.6 | 936 ms | 14.6 ms | 8.0 | 0 | 16,472 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 693.7 | 43.4 | 1,676 ms | 23.0 ms | 16.0 | 0 | 20,800 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 808.6 | 25.4 | 3,031 ms | 39.4 ms | 32.0 | 0 | 24,256 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 123.5 | 123.9 | 563 ms | 8.1 ms | 1.0 | 0 | 3,704 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 200.9 | 100.8 | 800 ms | 9.9 ms | 2.0 | 0 | 6,029 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 291.7 | 73.2 | 1,102 ms | 13.7 ms | 4.0 | 0 | 8,754 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 371.5 | 46.6 | 1,821 ms | 21.5 ms | 8.0 | 0 | 11,144 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 436.6 | 27.3 | 3,180 ms | 36.6 ms | 16.0 | 0 | 13,104 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.