Speed test matrix
DeepSeek-V4-Flash-Vision-Exp on H200 SXM 141GB ×4
sglang · 0.5.6.post3.dev9812+gd5a3b7725 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9812+gd5a3b7725
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
d5a3b7725374
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-vision-sglang-pr-37253/bin/sglang" serve --model-path deepseek-ai/DeepSeek-V4-Flash-Vision-Exp --served-model-name deepseek --trust-remote-code --tp 4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --context-length 1048576 --mem-fraction-static 0.850786 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend marlin
Prefill
prompt tok/s by context length8K context21.2K387 ms TTFT
16K context19.6K831 ms TTFT
32K context20.6K1,568 ms TTFT
64K context20.5K3,148 ms TTFT
128K context18.8K6,843 ms TTFT
256K context16K16,101 ms TTFT
512K context12.3K42,072 ms TTFT
1019K context8,130126,173 ms TTFT
1024K context7,924130,088 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | 136.9242 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 64K | 135.4469 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 128K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 256K | 128.51,363 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 512K | 130.22,518 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 1019K | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.2K | — | 387 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 19.6K | — | 831 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.6K | — | 1,568 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.5K | — | 3,148 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.8K | — | 6,843 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16K | — | 16,101 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 12.3K | — | 42,072 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,130.0 | — | 126,173 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,924.0 | — | 130,088 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 136.9 | 138.2 | 242 ms | 7.2 ms | 1.0 | 1 | 4,108 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 135.4 | 137.1 | 469 ms | 7.3 ms | 1.0 | 1 | 4,060 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 128.5 | 134.4 | 1,363 ms | 7.4 ms | 1.0 | 1 | 3,855 | 30.0 s |
| Sustained decode | 256K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 130.2 | 130.2 | 2,518 ms | 7.7 ms | 0.9 | 1 | 3,644 | 28.0 s |
| Sustained decode | 512K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.