Continuous batching for LLM inference using KV caching and prefill optimization

AI Impact Summary

This CAPABILITY introduces continuous batching in LLM inference, enabling multiple conversations to be processed in parallel by sharing and reusing computations across prefill and decode stages. By optimizing Q, K, and V handling and caching, it increases throughput and reduces per-token compute in high-load serving scenarios. Targeted models like Qwen and Claude can experience lower latency under concurrency, necessitating pipeline changes to manage multi-conversation buffers, attention masks, and KV caches coherently.