CLIP demonstrates multimodal neurons with cross-input concept alignment

AI Impact Summary

CLIP's internal representations show neurons that respond to the same concept across literal, symbolic, and conceptual inputs, revealing a shared cross-modal concept space. This could explain CLIP's resilience to surprising visual renditions and underpins improved generalization for zero-shot and symbolic reasoning tasks. For engineering teams, plan to expand evaluation to cross-modal concept consistency and examine potential shifts in biases or failure modes when inputs vary in modality.