Speed test matrix
Laguna-XS-2.1-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_SHARED_EXPERT_TP1=1 "$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.857801 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 1 --attention-backend fa3 --speculative-algorithm DFLASH --speculative-draft-model-path poolside/Laguna-XS-2.1-DFlash-FP8
Prefill
prompt tok/s by context length8K context39.4K208 ms TTFT
16K context39.2K414 ms TTFT
32K context37.4K861 ms TTFT
64K context29.3K2,189 ms TTFT
128K context20.6K6,236 ms TTFT
251K context13K19,327 ms TTFT
256K context12.9K19,916 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 325.379 ms TTFT | 469.9109 ms TTFT | 915.685 ms TTFT | 1,299.1127 ms TTFT | 2,040.3207 ms TTFT | 2,867.5384 ms TTFT | Over capacity | Over capacity |
| 16K | 238.693 ms TTFT | 288.1148 ms TTFT | 850.0123 ms TTFT | 1,218.7188 ms TTFT | 1,799.6340 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 350.3138 ms TTFT | 413.7134 ms TTFT | 578.2207 ms TTFT | 767.0390 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 211.0200 ms TTFT | 452.6176 ms TTFT | 587.1397 ms TTFT | 879.7819 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 193.6271 ms TTFT | 221.8422 ms TTFT | 476.3775 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 209.2564 ms TTFT | 234.3840 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.4K | — | 208 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.2K | — | 414 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 37.4K | — | 861 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.3K | — | 2,189 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.6K | — | 6,236 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13K | — | 19,327 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,916 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 325.3 | 337.8 | 79 ms | 3.3 ms | 0.9 | 3 | 9,757 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 469.9 | 206.7 | 109 ms | 5.0 ms | 2.1 | 2 | 14,094 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 915.6 | 213.0 | 85 ms | 4.8 ms | 4.0 | 6 | 27,466 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,299.1 | 140.1 | 127 ms | 7.1 ms | 9.0 | 8 | 38,942 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,040.3 | 99.2 | 207 ms | 10.1 ms | 20.2 | 6 | 61,191 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,867.5 | 73.6 | 384 ms | 13.6 ms | 37.1 | 6 | 86,028 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 238.6 | 242.9 | 93 ms | 4.1 ms | 1.0 | 1 | 7,156 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 288.1 | 146.3 | 148 ms | 6.8 ms | 16.9 | 1 | 8,643 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 850.0 | 159.3 | 123 ms | 6.3 ms | 4.4 | 6 | 25,497 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,218.7 | 138.7 | 188 ms | 7.2 ms | 10.0 | 6 | 36,558 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,799.6 | 97.4 | 340 ms | 10.3 ms | 18.5 | 4 | 53,973 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 350.3 | 480.3 | 138 ms | 2.1 ms | 1.0 | 2 | 10,509 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 413.7 | 209.4 | 134 ms | 4.8 ms | 5.1 | 3 | 12,412 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 578.2 | 165.3 | 207 ms | 6.1 ms | 5.0 | 4 | 17,346 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 767.0 | 92.9 | 390 ms | 10.8 ms | 11.4 | 0 | 23,002 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 211.0 | 211.9 | 200 ms | 4.7 ms | 1.0 | 1 | 6,328 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 452.6 | 191.7 | 176 ms | 5.2 ms | 2.7 | 3 | 13,574 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 587.1 | 137.1 | 397 ms | 7.3 ms | 5.3 | 3 | 17,614 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 879.7 | 98.0 | 819 ms | 10.2 ms | 9.7 | 3 | 26,373 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 193.6 | 200.4 | 271 ms | 5.0 ms | 1.0 | 1 | 5,807 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 221.8 | 108.2 | 422 ms | 9.2 ms | 5.6 | 0 | 6,655 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 476.3 | 124.8 | 775 ms | 8.0 ms | 4.0 | 3 | 14,287 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 209.2 | 217.4 | 564 ms | 5.0 ms | 1.0 | 1 | 6,273 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 234.3 | 123.2 | 840 ms | 8.1 ms | 4.4 | 1 | 7,027 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.