KV Cache implemented in nanoVLM to accelerate Vision-Language model generation

AI Impact Summary

Implementing KV caching in nanoVLM reuses previously computed K and V across generation steps, converting full-sequence attention into an incremental update and delivering about 38% faster generation. The change touches the Attention block and per-layer cache tracking (LanguageModelGroupedAttention and the overall VisionLanguageModel), requiring a block_kv_cache structure, start_pos handling for rotary embeddings, and a split between prefill and decode phases. This pattern reduces compute and memory redundancy for autoregressive inference in Vision-Language models, but increases memory usage for caching K/V across layers; teams should ensure cache lifetimes and hardware memory budgets are accounted for when deploying.