Show me your care: The impact of multimodal attentive behaviours for robots on user experience in a healthcare context

Abstract

While robot attentive behaviours hold significant potential for improving user experience in healthcare contexts, how multimodal cues are jointly interpreted by users remains poorly understood. This study examines how multimodal attentive behaviours shape user experience in a simulated ward-round interaction with a Pepper robot acting as a healthcare assistant. A 2 × 2 × 2 within-subject experiment manipulated attentive behaviours across auditory, visual, and tactile modalities, including voice pitch (low vs high), lean forward (with vs without), and robot-initiated touch (with vs without). Subjective perceptions (questionnaires), visual attention (eye-tracking), and neural activity (functional near-infrared spectroscopy, fNIRS) were measured. Results showed that a high-pitched voice, a forward-leaning posture, and robot-initiated touch positively influence users’ perceptions. Robot-initiated touch attracted visual attention and was associated with increased neural activity in the left dorsolateral prefrontal cortex (DLPFC). Significant interactions revealed a cross-modal compensatory mechanism, whereby the absence of cues in one modality increased sensitivity to cues in another. Both visual and tactile cues independently improved user perceptions compared to the auditory-only condition, while combining visual and tactile modalities yielded diminishing incremental benefits. These findings provide converging subjective, behavioural, and neurophysiological evidence that multimodal attentive behaviours can shape users’ experience through cross-modal compensatory mechanisms rather than linearly additive effects.