vLLM parallel prefill with long-prompt caps to reduce queue blocking

AI Impact Summary

At scale, prefill is the bottleneck: a long prompt can saturate the GPU during the prefill phase, causing subsequent requests to wait before their first token. The latest vLLM update adds parallel prefills with a cap on long prompts, enabling short requests to achieve a faster time-to-first-token while longer prompts wait their turn. For a 24 H100 GPU cluster serving 50+ apps, this trade-off improves responsiveness for typical short queries but leaves total completion time governed by decode throughput; alternative architectures like disaggregated prefill or per-length routing can further optimize utilization.