Seeing Only When Needed: A Multimodal Architecture for Context-Aware Social Dialogue

Abstract

Large Vision–Language Models are increasingly used for visually grounded social dialogue, yet most systems assume that vision should be active continuously, adding computational load and increasing the risk of unnecessary or hallucinated descriptions. We present a multimodal architecture that treats vision as a selective, context-dependent resource. A lightweight vision-gating module triggers visual grounding only when a user utterance requires it, while a complementary ambient monitoring component detects gradual scene changes at a low frame rate. Both pathways contribute cues only when relevant, enabling the robot to use visual information meaningfully without overuse. A preliminary evaluation with 10 participants (≈ 95 minutes) shows that the gating mechanism identified vision-relevant turns with 93.4% accuracy, and that grounded descriptions aligned with the scene in 90.7% of cases.