Lost in Translation? Assessing User Perception of Prompt-Engineered Chatbot Tone-of-Voice for Smoking Cessation

Abstract

Large Language Models (LLMs) offer significant potential for personalised health coaching applications, such as smoking cessation support, where communication style is crucial for user engagement. This study explored the effectiveness of using system prompts with an open-source LLM to generate perceptibly distinct communication tones – specifically social-oriented, task-oriented, and neutral – for a just-in-time chatbot intervention reacting to contextually detected smoking triggers such as stress. We investigated whether end-users could perceive these intended tonal variations in a between-subjects online study (N = 297). Participants evaluated chatbot message vignettes, generated under one of the three tonal conditions, presented in a familiar mock WhatsApp interface. Perceived tone was measured using a semantic differential scale assessing task versus social orientation. Despite clear differences in the guiding prompts and resulting textual outputs (e.g., length, directness), ANOVA and ANCOVA (controlling for age, sex, employment status) revealed no statistically significant differences in perceived tone across the conditions. These findings suggest that users may not easily distinguish subtle, prompt-engineered tonal variations in LLM-generated messages. This highlights challenges in achieving perceptible communication style control in current LLMs for health contexts using simple system prompting alone, possibly due to strong baseline model fine-tuning or training data.