Expressive Furhat: Generating Real-Time Facial Expressions for Human-Robot Dialogue with LLMs

Abstract

Enabling natural robot communication through dynamic, context-aware facial expressions remains a key challenge in human-robot interaction. The field lacks a system that can generate facial expressions in real time and can be easily adapted to different contexts. Early work in this area considered inherently limited rule-based systems or deep learning-based models, requiring large datasets. Recent systems using large language models (LLMs) could not yet generate context-appropriate facial expressions in real time. This paper introduces Expressive Furhat, an open-source algorithm and Python library that leverages LLMs to generate real-time, adaptive facial gestures for the Furhat robot. Our modular approach separates gesture rendering, new gesture generation, and gaze aversion, ensuring flexibility and seamless integration with the Furhat API. User studies demonstrate significant improvements in user perception over a baseline system, with participants praising the system's emotional responsiveness and naturalness.