Speed test matrix
Step-3.5-Flash on H200 SXM 141GB ×4
vllm · 0.29.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- stepfun-ai/Step-3.5-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve stepfun-ai/Step-3.5-Flash --served-model-name stepfun-ai/Step-3.5-Flash --trust-remote-code --tensor-parallel-size 4 --reasoning-parser step3p5 --enable-auto-tool-choice --tool-call-parser step3p5 --max-model-len 262144 --gpu-memory-utilization 0.85 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --disable-cascade-attn
Prefill
prompt tok/s by context length8K context34.2K239 ms TTFT
16K context35K464 ms TTFT
32K context34.2K944 ms TTFT
64K context31.2K2,063 ms TTFT
128K context26.5K4,859 ms TTFT
251K context20.6K12,261 ms TTFT
256K context20K12,878 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 156.744 ms TTFT | 262.256 ms TTFT | 449.881 ms TTFT | 730.9106 ms TTFT | 1,188.9175 ms TTFT | 1,842.4303 ms TTFT | 2,966.1591 ms TTFT | 4,673.11,129 ms TTFT |
| 16K | 155.864 ms TTFT | 260.971 ms TTFT | 448.0110 ms TTFT | 729.8162 ms TTFT | 1,158.3273 ms TTFT | 1,804.8506 ms TTFT | 2,854.71,007 ms TTFT | Over capacity |
| 32K | 154.0107 ms TTFT | 259.7114 ms TTFT | 447.5188 ms TTFT | 710.7304 ms TTFT | 1,133.2543 ms TTFT | 1,726.41,040 ms TTFT | Over capacity | Over capacity |
| 64K | 150.9196 ms TTFT | 253.1207 ms TTFT | Looped | 688.5595 ms TTFT | 1,061.31,060 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 144.8389 ms TTFT | 243.4411 ms TTFT | 409.0787 ms TTFT | 634.51,298 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 133.6877 ms TTFT | 228.41,289 ms TTFT | 369.01,951 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.2K | — | 239 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 35K | — | 464 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 34.2K | — | 944 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 31.2K | — | 2,063 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.5K | — | 4,859 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 20.6K | — | 12,261 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20K | — | 12,878 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 156.7 | 156.9 | 44 ms | 6.4 ms | 1.0 | 1 | 4,701 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 262.2 | 131.2 | 56 ms | 7.6 ms | 2.0 | 2 | 7,866 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 449.8 | 112.4 | 81 ms | 8.9 ms | 4.0 | 0 | 13,492 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 730.9 | 91.4 | 106 ms | 10.9 ms | 8.0 | 0 | 21,920 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,188.9 | 74.4 | 175 ms | 13.4 ms | 16.0 | 0 | 35,664 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,842.4 | 58.0 | 303 ms | 17.2 ms | 32.0 | 0 | 55,264 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,966.1 | 46.8 | 591 ms | 21.4 ms | 64.0 | 0 | 88,960 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,673.1 | 36.9 | 1,129 ms | 27.1 ms | 128.0 | 0 | 140,160 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 155.8 | 156.1 | 64 ms | 6.4 ms | 1.0 | 1 | 4,673 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 260.9 | 130.7 | 71 ms | 7.7 ms | 2.0 | 2 | 7,825 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 448.0 | 111.9 | 110 ms | 8.9 ms | 4.0 | 0 | 13,436 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 729.8 | 91.2 | 162 ms | 11.0 ms | 8.0 | 0 | 21,888 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,158.3 | 72.5 | 273 ms | 13.8 ms | 16.0 | 0 | 34,736 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,804.8 | 56.8 | 506 ms | 17.6 ms | 32.0 | 0 | 54,144 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,854.7 | 45.1 | 1,007 ms | 22.2 ms | 64.0 | 0 | 85,632 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 154.0 | 154.6 | 107 ms | 6.5 ms | 1.0 | 1 | 4,619 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 259.7 | 130.1 | 114 ms | 7.7 ms | 2.0 | 2 | 7,790 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 447.5 | 111.7 | 188 ms | 8.9 ms | 4.0 | 0 | 13,424 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 710.7 | 88.9 | 304 ms | 11.2 ms | 8.0 | 0 | 21,322 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,133.2 | 70.9 | 543 ms | 14.1 ms | 16.0 | 0 | 33,984 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,726.4 | 54.2 | 1,040 ms | 18.4 ms | 32.0 | 0 | 51,776 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 150.9 | 152.2 | 196 ms | 6.6 ms | 1.0 | 1 | 4,528 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 253.1 | 127.8 | 207 ms | 7.8 ms | 2.0 | 2 | 7,592 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 688.5 | 86.0 | 595 ms | 11.6 ms | 8.0 | 0 | 20,648 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,061.3 | 66.5 | 1,060 ms | 15.0 ms | 16.0 | 0 | 31,840 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 144.8 | 147.4 | 389 ms | 6.8 ms | 1.0 | 1 | 4,345 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 243.4 | 129.1 | 411 ms | 7.8 ms | 1.9 | 2 | 7,302 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 409.0 | 102.4 | 787 ms | 9.8 ms | 4.0 | 0 | 12,268 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 634.5 | 79.4 | 1,298 ms | 12.6 ms | 8.0 | 0 | 19,032 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 133.6 | 139.5 | 877 ms | 7.2 ms | 1.0 | 1 | 4,007 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 228.4 | 114.6 | 1,289 ms | 8.7 ms | 2.0 | 0 | 6,850 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 369.0 | 92.7 | 1,951 ms | 10.8 ms | 4.0 | 0 | 11,069 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.