融合自适应非语言行为与语言模型的实时动机性访谈虚拟代理

摘要

我们提出一个开源的虚拟代理平台，用于开展实时动机性访谈，该平台结合了先进的语言模型和扩散模型，能够根据用户的行为和特征进行自适应调整。通过使用Greta 2.0平台，该系统支持多种主题，包括以营养和运动为重点的干预措施，为增强数字化治疗互动提供了一种灵活且经过验证的工具。

摘要

对治疗支持日益增长的需求正不断超过现有专业人员的承载能力。一种能够执行动机性访谈（MI）的虚拟代理为帮助患者在与人类治疗师会话的间隔期间实现行为改变目标提供了有前景的解决方案。MI本质上是一种合作且具有适应性的沟通形式。因此，开发一种能够根据情境调整其对话策略的代理，可显著提升治疗效果。在MI会话过程中，人类治疗师会根据患者的反应及其个人特征，调整自身的言语和非言语行为。根据患者动机水平的不同，治疗师将相应地调整其干预方式。因此，个性化与适应性对于开发有效的MI虚拟代理至关重要。本文介绍了一种能够通过实时动态调整言语和非言语行为来开展MI会话的虚拟代理。该系统依托最先进的模型，实现了MI交互功能。该虚拟代理基于Greta 2.0平台构建，其非言语行为通过一种名为MODIFF的扩散模型生成，该模型能够适应用户的面部表情及其行为改变准备度。这些面部表情特征是在一个MI语料库上训练获得，并通过专门的用户研究进行了验证。对话内容由最先进的大语言模型（LLM）生成，并由专为MI设计的对话管理器进行增强，该管理器采用强化学习方法，并经过用户测试验证。此外，该对话管理器能够适应不同用户特征。最终构建的平台为开源系统，支持生成实时、多模态的MI对话，为数字化辅助的治疗性交互提供了新工具。

引言

Motivational interviewing (MI) is a collaborative therapeutic approach that encourages behavioral change. During MI sessions, MI practitioners help patients articulate and foster their motivation for change¹. To this aim, they use dialogue strategies, such as reflections or questions, reinforced by non-verbal behaviors, such as smiles or specific head and body postures.

As the frequency of mental health issues has increased in recent years, a gap has emerged between the demand for mental health services and the available resources². This has led to longer waiting times for patients before they can access therapy^2,3. One proposed solution to alleviate this issue involves virtual agents capable of replicating MI for patients awaiting appointments. By offering immediate support and intervention, these virtual agents can help reduce the impact of waiting periods, especially in therapeutic contexts that require multiple sessions⁴. Several studies have demonstrated the effectiveness of such agents or chatbots^5,6,7,8,9 whose goal is not to replace MI experts; they are intended to serve as a complementary tool within the therapeutic process. For instance, they could provide MI intervention in between sessions.

Although previous MI agents often exhibit limited adaptive behaviors, MI is inherently a cooperative and adaptive form of communication. Therefore, developing an agent capable of adapting its dialogue to the context could significantly enhance the effectiveness of therapy¹⁰. During MI sessions, therapists adjust both their verbal (conversational strategies) and nonverbal behaviors based on the patient's responses, motivation, or engagement level^11,12, as well as their individual profiles¹³. Depending on the patient's level of motivation, the therapist will modify their approach accordingly. Thus, personalization and adaptability are essential for developing effective MI virtual agents.

The Greta 2.0 platform¹⁴ is a modular virtual agent framework that supports real-time interactions. It has recently been extended to incorporate capabilities that enable the dynamic adaptation of generative AI models during interaction.

The platform includes a model called MODIFF-8¹⁵, which generates real-time facial expressions that dynamically adapt to the user's own facial expressions. This diffusion-based model was trained on multimodal data collected in the context of MI, namely the AnnoMI corpus, constituted of counselling session videos between therapist and client pairs, collected on streaming platforms, such as YouTube and Viméo, and annotated in dialogue acts (a classification of utterances that denotes the actions performed e.g., Question, Greeting…) and action units (a taxonomy of human facial expression). This data conditions the generation not only on the user's current facial expression but also on MI-specific dialogue acts identified during the interaction, linked to the conversational strategies adopted. These include MI-relevant users' dialogue acts, such as change talk, as well as the agent's own dialogue acts, such as reflection or providing information. By integrating these multimodal signals in real time, the system allows the virtual agent to express socially and contextually appropriate facial expressions. By aligning its non-verbal behavior (NVB) to the context of the interaction, this virtual agent is designed to foster social rapport and support the initiation of intrinsic motivation throughout the interview process¹⁵.

Additionally, the platform integrates a dialogue system specifically designed for MI, called DREAM¹⁶. DREAM comprises a dialogue manager that selects the agent's next dialogue act based on the current dialogue state and the user's profile. Indeed, our analyses of the AnnoMI corpus revealed three profiles of MI patients: Open to Change (when the patient is motivated to change and seeks support), Receptive (when the patient has not yet considered change), and Resistant (when the patient does not intend to change)¹³. Before interacting with the virtual agent, users are attributed one of these three profiles according to their answers to the Decision Balance Scale (DBS) questionnaire¹⁷. This dialogue system is complemented by a large language model (LLM) that generates the agent's verbal output, corresponding to the dialogue act selected by the dialogue manager. This component enables contextual adaptation of the dialogue, marking an improvement over more rigid, rule-based approaches⁶. While recent studies have explored the use of LLMs to generate MI-consistent dialogue, they do not account for user profiles^18,19. In contrast, this module dynamically adapts to the user's profile, which enhances rapport compared to these baseline systems.

The integration of these models into the Greta 2.0 platform enables dynamic MI interventions. Its modular implementation allows for the independent evaluation of each model's impact. In this paper, we present the method used to conduct experiments that assess the contributions of the MODIFF-8 and DREAM models to the MI dialogue delivered by the platform used. This method can also be used to deliver tailored MI interventions between sessions with MI practitioners. It requires a Windows computer with a 16 GB GPU and the installation of open-source software. Using this method, we demonstrate that our adaptation models improve the perceived quality of the MI delivery agent and enhance rapport building. We also observe improvement in the motivation of patients to change their behaviors when the agent uses adapted behaviors at the verbal and nonverbal levels, and conversational strategies levels. The results presented in this paper demonstrate the positive impact of tailoring the agent's MI intervention to each patient.

To support the design of this system, an in-depth literature review was conducted to explore the theoretical and practical foundations of nutritional coaching, including motivational interviewing, behavior change models, and user typologies. This review led to the development of a qualitative knowledge base structured around the intersection of two key dimensions: 1) users' readiness for change following the Transtheoretical Model¹⁹ that identifies 5 stages: pre-contemplation, contemplation, preparation, action, and maintenance, and 2) their eating behavior profiles²⁰ based on diet and psychologically related factors. Three profiles were identified: the intuitive eater (i.e., listens to their hunger and fullness cues, maintains a varied and predominantly plant-based diet, prioritizes high-quality, often organic foods), the emotional eater (i.e., copes with stress, loneliness, and joy through frequent snacking on sweets or fatty foods, often feeling guilty afterward) and, the restrictive eater (i.e., obsessed with food control, they count calories, follow frequent diets with yo-yo effects, and may have an eating disorder).

For each combination of stage and profile, context-specific dialogue strategies were proposed, including tailored SMART goals, motivational formulations, and concrete behavioral suggestions. The aim was to enable the virtual agent to dynamically adapt its interventions based on both psychological readiness and personal tendencies. This groundwork ensures a higher degree of personalization and empathy in interaction, which are crucial to sustaining user engagement and long-term behavioral change.

方案

The method received ethical IRB approval from the INSEAD institute (Institut européen d'administration des affaires) with acceptance numbers: INSEAD 2024-78 and INSEAD 2025-23. All participants provided informed consent and were compensated for their experiment time.

1. Recruitment of participants

1. Call for participants using the institute's mailing lists. No constraints on participants, except familiarity with digital interfaces and fluency in French. Three participants were excluded due to internet resource drops, which significantly increased the global system response latency.

2. Pre-installation requirements

1. Use a Windows computer. We use an Intel I7 CPU, 20 cores, 28 threads (50% usage), RAM 32GB (20% usage), 32GB Nvidia RTX 5000 GPU (20% usage).
2. Install Java SE Development Kit 8 (https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html). Install Visual C++ Redistributable for Visual Studio 2013 (https://www.microsoft.com/en-us/download/details.aspx?id=40784).
3. Request a Cereproc license (https://app.cereproc.com/). Install OpenFace (https://github.com/TadasBaltrusaitis/OpenFace). Get a webcam.

3. Greta installation

1. Download the software release from https://github.com/isir/greta/releases. Compile using NetBeans (detailed instructions at: https://github.com/isir/greta/wiki/Quick-start).

4. Mistral API key configuration

1. Get a Mistral API key from https://console.mistral.ai/api-keys.
2. Create the file: greta\bin\Common\Data\LLM\MICounselorRL\api_key.txt. Paste the API key into the created file.

5. DeepGram API key configuration

1. Get a DeepGram API key from https://console.deepgram.com/.
2. Create the file: greta\bin\Common\Data\DeepASR\DeepGram\api_key.txt.Paste the API key into the created file.

6. MODIFF model importation

1. Download the model weights from https://drive.google.com/file/d/1n3udJf5pSNnFmxs_kt_zcFcF0m7zoLLr/view?usp=sharing and place the file into greta\bin\Common\Data\MODIFF.
2. Download and uncompress dlib face detection models from https://dlib.net/files/mmod_human_face_detector.dat.bz2 and https://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2, then place them into bin/Common/Data/TurnManagement/dlib_models.
3. Download all VAP models in model/VAP from https://drive.google.com/drive/folders/10_MTPkRyJCoFLDt4LO_M9kRgoreh6gEF?usp=sharing and place them into bin/Common/Data/TurnManagement/models/.

7. Launching Greta 2.0

1. Launch Modular.jar. Click File > Open > Greta > Advanced > 20250128 Greta - Expé Lucie_full.xml.
2. Ensure that the configuration matches Figure 1.

8. Launching OpenFace

1. Start OpenFace Offline ZeroMQ.exe. A specific version of OpenFace allows direct communication with the Greta platform using the ZeroMQ protocol through port 5000.
2. In the Record tab, uncheck everything except Broadcast with ZeroMQ. It allows the software to send the extracted signals (AUs, Gaze, Head movements …) to the Greta platform.
3. In the File tab, click Open Webcam. Authorize live feature extraction using the available webcam.
4. Select the webcam to use and wait until the webcam is loaded (see Figure 2). Once the webcam is loaded, the tool will automatically start the live feature extraction and send the data to the Greta platform.

9. Launching MODIFF-8

1. In OpenFace2 Output Stream Reader, click the Connect button and wait for the list of available features, sent from OpenFace, to populate (see Figure 3).
2. Click Select All > Set, to validate the set of features you want the platform to continuously receive from Openface.
3. Under Modiff, click Launch and wait for the message: Connected to server at localhost 5560.
4. Click Connect. This step activates the connection between the facial expression generation module Modiff and the Greta OpenFace data receiver.
5. Under Filter, click Perform, allowing the virtual agent to perform the data generated by the Modiff Module.
6. In the MODIFF window, click Enable. Wait 90 s for the model to stabilize.

10. Launching ASR

1. In the DeepGram window, press Enable.

11. Launching DREAM

1. Select the Condition RL to use DREAM and Baseline to use a plain LLM.
2. In the MI Counsellor RL window, click Enable. Wait 30 s for the model to start.

12. Set up the room

1. Use a large monitor to put on a table with a chair in front. Put a webcam on top of the monitor facing the chair. Put a directional microphone on the table in front of the chair.

13. Welcoming the participant

1. Welcome the participant to a private room (see Figure 4). Brief the participant as per the speech in Supplementary File 1.
2. Have the participant fill out the DBS questionnaire on a 5-step Likert scale (see items in Table 1).
3. Ask the participant to talk into the microphone. Ensure the webcam captures the participant's head and upper body

14. Identify the participant profile

1. If the DBS score is negative, classify the participant as Resistant to Change. If the DBS score is between 0 and 2, classify the participant as Receptive. Otherwise, classify the participant as Open to Change.

15. Set the theme and profile

1. In the MI Counsellor RL window, set the theme to be discussed as the participant chose. Set the profile as identified in step 14.

16. Start the dialogue

1. In the MI Counsellor RL window, click Start to have the agent initiates the discussion with the participant. Clicking this button makes the dialogue DREAM, and multimodal behaviors MODIFF models automatically drive the agent's behaviors.

17. During the dialogue

1. Monitor the Answer paragraph of the MI Counsellor window to ensure the LLM output is not harmful.
2. In the DeepGram window, click Listen after the agent's turn is finished.
3. If the speech is not recognized, fill the Request paragraph of the MI Counsellor RL window with the participant's speech and click Send. This automatically creates the agent's response.
4. Wait for the participant's answer.
5. Repeat 17.1 to 17.4 until the dialogue ends. The dialogue stops on its own.

18. Post-intervention

1. Ask the participant to complete the post-intervention questionnaires (see items 1-9 in Table 1). Debrief the participant using the speech in Supplementary File 1.

结果

在本方案中，我们采用针对拟测量行为调整的问卷，收集了交互文本记录和用户反馈。我们比较了使用模型与不使用模型时的交互结果。

使用该方案评估 MODIFF-8 自适应表情生成的影响
MODIFF-8 模块旨在通过使虚拟代理生成既符合语境又与用户行为同步的面部表情，从而增强其社交反应能力¹⁵。该设计背后的核心假设是，适应性的非语言行为，尤其是与交流内容相协调的面部表情，在动机性访谈（MI）过程中有助于建立融洽关系并促进自我表露^19,20,21。

为了评估这种实时自适应能力的影响，我们开展了一项用户研究。通过索邦大学行为实验室的INSEAD招募平台，我们招募了63名参与者（N = 63；年龄18-45岁；男性30名，女性33名）。该研究采用被试间设计，在三个实验...

讨论

本文介绍了一种在受控且可重复的交互框架内，实施个性化动机性访谈干预并测试个体行为生成模块的实验方案。该方法基于 Greta 2.0 平台的模块化架构，能够控制智能体行为所涉及的各个组件，使研究人员可轻松开启或关闭特定模块。这种灵活性使 Greta 2.0 非常适用于在人机多模态交互中进行实验性操控以及对新功能进行有针对性的评估。本方案可用于两个目的：测试 Greta 平台中新模块的有效性（例如，生成自动手势的模块），或提供质量达到治疗阈值以上的个性化动机性访谈。如实验结果所示，采用这些适应性模型可增强用户对智能体的感知体验及其行为改变的动机水平。

该平台的一个关键应用是在人类专家会话间隙期间提供动机性访谈（MI）的对话支持，特别是在缺乏经过MI培训人员的环境中。该平台支持非专家监督，从而提高了MI干预的可及性。同时，它还为系统性测试新的行为模块（如全身姿态生成、注视行为和轮流发言策略）提供了研究环境。由于该平台采用预先验证的对话结构以及集成的表情生成模块（DREAM和MODIFF-8），无需从零开始设计和验证新的交互内容，避免了这一耗时过程对实验结果有效性的影响。

材料

本文使用的材料清单

姓名	公司	目录编号	评论
计算机	Dell		配备 Intel Core i7-14700 处理器和 Nvidia RTX5000 显卡的计算机
带麦克风的耳机	Epos		带麦克风的耳机
网络摄像头	Logitech		用于面部表情提取
网络摄像头2	Logitech		用于视频录制