Speed test matrix
Laguna-XS-2.1-INT4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- INT4+INT8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.851767 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context33.2K247 ms TTFT
16K context32.3K502 ms TTFT
32K context30K1,074 ms TTFT
64K context25.5K2,521 ms TTFT
128K context18.5K6,924 ms TTFT
251K context12.1K20,784 ms TTFT
256K context12K21,389 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 187.965 ms TTFT | 232.483 ms TTFT | 619.5104 ms TTFT | 970.7131 ms TTFT | 1,451.2196 ms TTFT | 2,633.3323 ms TTFT | Over capacity | Over capacity |
| 16K | 184.888 ms TTFT | 251.9104 ms TTFT | 499.5131 ms TTFT | 838.7196 ms TTFT | 1,606.6329 ms TTFT | 2,383.0634 ms TTFT | Over capacity | Over capacity |
| 32K | 179.3115 ms TTFT | 187.7138 ms TTFT | 556.2217 ms TTFT | 717.8358 ms TTFT | 1,317.5680 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 169.1168 ms TTFT | 213.4306 ms TTFT | 506.3350 ms TTFT | 564.6759 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 152.6262 ms TTFT | 180.1436 ms TTFT | 377.2818 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 127.6501 ms TTFT | 197.0868 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.2K | — | 247 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.3K | — | 502 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30K | — | 1,074 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,521 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.5K | — | 6,924 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.1K | — | 20,784 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12K | — | 21,389 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 187.9 | 188.4 | 65 ms | 5.3 ms | 1.0 | 1 | 5,635 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 232.4 | 115.5 | 83 ms | 8.7 ms | 2.9 | 0 | 6,971 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 619.5 | 155.9 | 104 ms | 6.4 ms | 4.0 | 4 | 18,584 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 970.7 | 128.5 | 131 ms | 7.8 ms | 10.3 | 8 | 29,121 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,451.2 | 91.4 | 196 ms | 10.9 ms | 24.0 | 0 | 43,536 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,633.3 | 80.3 | 323 ms | 12.5 ms | 36.0 | 0 | 79,008 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 184.8 | 185.5 | 88 ms | 5.4 ms | 1.0 | 1 | 5,543 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 251.9 | 118.7 | 104 ms | 8.4 ms | 14.8 | 2 | 7,554 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 499.5 | 137.2 | 131 ms | 7.4 ms | 5.6 | 4 | 14,984 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 838.7 | 105.2 | 196 ms | 9.5 ms | 11.9 | 0 | 25,160 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,606.6 | 99.9 | 329 ms | 10.0 ms | 16.0 | 0 | 48,192 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,383.0 | 73.3 | 634 ms | 13.6 ms | 33.6 | 0 | 71,489 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 179.3 | 180.2 | 115 ms | 5.5 ms | 1.0 | 1 | 5,378 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 187.7 | 90.5 | 138 ms | 11.1 ms | 23.3 | 0 | 5,631 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 556.2 | 138.8 | 217 ms | 7.2 ms | 4.1 | 4 | 16,687 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 717.8 | 90.2 | 358 ms | 11.1 ms | 12.0 | 0 | 21,528 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,317.5 | 80.8 | 680 ms | 12.4 ms | 17.9 | 0 | 39,520 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 169.1 | 170.3 | 168 ms | 5.9 ms | 1.0 | 1 | 5,074 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 213.4 | 102.6 | 306 ms | 9.8 ms | 9.8 | 0 | 6,402 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 506.3 | 126.2 | 350 ms | 7.9 ms | 3.9 | 4 | 15,186 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 564.6 | 71.2 | 759 ms | 14.0 ms | 12.0 | 0 | 16,936 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 152.6 | 154.2 | 262 ms | 6.5 ms | 1.0 | 1 | 4,579 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 180.1 | 88.0 | 436 ms | 11.4 ms | 8.6 | 0 | 5,402 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 377.2 | 91.3 | 818 ms | 11.0 ms | 4.5 | 0 | 11,316 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 127.6 | 130.1 | 501 ms | 7.7 ms | 1.0 | 1 | 3,826 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 197.0 | 95.5 | 868 ms | 10.5 ms | 2.8 | 0 | 5,909 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.