Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models

Summary

We present an open-source virtual agent platform for conducting real-time motivational interviews, combining state-of-the-art language and diffusion models to adapt to users' behavior and profile. Utilizing the Greta 2.0 platform, it supports various topics, including nutrition and sport-focused interventions, and offers a flexible, validated tool for enhancing digital therapeutic interactions.

Abstract

The growing demand for therapeutic support increasingly exceeds the capacity of available professionals. A virtual agent capable of performing motivational interviewing (MI) offers a promising solution to assist patients in reaching their goal of behavior change between sessions with human therapists. MI is inherently a cooperative and adaptive form of communication. Therefore, developing an agent capable of adapting its conversational strategies to the context could significantly enhance the effectiveness of therapy. During MI sessions, human therapists adjust both their verbal and nonverbal behaviors based on the human patients' responses, as well as their profiles. Depending on the patient's level of motivation, the therapist will modify their approach accordingly. Thus, personalization and adaptability are essential for developing effective MI virtual agents. In this paper, we present a virtual agent capable of conducting MI sessions by dynamically adapting verbally and nonverbally to users in real time. Leveraging state-of-the-art models, this system enables MI interactions. The virtual agent is embodied using the Greta 2.0 platform. Its nonverbal behavior is generated through a diffusion model called MODIFF, which adapts to the user's facial expressions and their readiness to change. These facial expressions were learned on an MI corpus and validated through a dedicated user study. The dialogue is generated using a state-of-the-art large language model (LLM), enhanced by a dialogue manager specifically designed for MI, with a reinforcement learning approach, and validated through user testing. Furthermore, the dialogue manager is able to adapt to different user profiles. The resulting platform is open-source and facilitates the generation of real-time, multimodal MI dialogues, providing new tools for digitally mediated therapeutic interactions.

Introduction

Motivational interviewing (MI) is a collaborative therapeutic approach that encourages behavioral change. During MI sessions, MI practitioners help patients articulate and foster their motivation for change¹. To this aim, they use dialogue strategies, such as reflections or questions, reinforced by non-verbal behaviors, such as smiles or specific head and body postures.

As the frequency of mental health issues has increased in recent years, a gap has emerged between the demand for mental health services and the available resources². This has led to longer waiting times for patients before they can access therapy^2,3. One proposed solution to alleviate this issue involves virtual agents capable of replicating MI for patients awaiting appointments. By offering immediate support and intervention, these virtual agents can help reduce the impact of waiting periods, especially in therapeutic contexts that require multiple sessions⁴. Several studies have demonstrated the effectiveness of such agents or chatbots^5,6,7,8,9 whose goal is not to replace MI experts; they are intended to serve as a complementary tool within the therapeutic process. For instance, they could provide MI intervention in between sessions.

Although previous MI agents often exhibit limited adaptive behaviors, MI is inherently a cooperative and adaptive form of communication. Therefore, developing an agent capable of adapting its dialogue to the context could significantly enhance the effectiveness of therapy¹⁰. During MI sessions, therapists adjust both their verbal (conversational strategies) and nonverbal behaviors based on the patient's responses, motivation, or engagement level^11,12, as well as their individual profiles¹³. Depending on the patient's level of motivation, the therapist will modify their approach accordingly. Thus, personalization and adaptability are essential for developing effective MI virtual agents.

The Greta 2.0 platform¹⁴ is a modular virtual agent framework that supports real-time interactions. It has recently been extended to incorporate capabilities that enable the dynamic adaptation of generative AI models during interaction.

The platform includes a model called MODIFF-8¹⁵, which generates real-time facial expressions that dynamically adapt to the user's own facial expressions. This diffusion-based model was trained on multimodal data collected in the context of MI, namely the AnnoMI corpus, constituted of counselling session videos between therapist and client pairs, collected on streaming platforms, such as YouTube and Viméo, and annotated in dialogue acts (a classification of utterances that denotes the actions performed e.g., Question, Greeting…) and action units (a taxonomy of human facial expression). This data conditions the generation not only on the user's current facial expression but also on MI-specific dialogue acts identified during the interaction, linked to the conversational strategies adopted. These include MI-relevant users' dialogue acts, such as change talk, as well as the agent's own dialogue acts, such as reflection or providing information. By integrating these multimodal signals in real time, the system allows the virtual agent to express socially and contextually appropriate facial expressions. By aligning its non-verbal behavior (NVB) to the context of the interaction, this virtual agent is designed to foster social rapport and support the initiation of intrinsic motivation throughout the interview process¹⁵.

Additionally, the platform integrates a dialogue system specifically designed for MI, called DREAM¹⁶. DREAM comprises a dialogue manager that selects the agent's next dialogue act based on the current dialogue state and the user's profile. Indeed, our analyses of the AnnoMI corpus revealed three profiles of MI patients: Open to Change (when the patient is motivated to change and seeks support), Receptive (when the patient has not yet considered change), and Resistant (when the patient does not intend to change)¹³. Before interacting with the virtual agent, users are attributed one of these three profiles according to their answers to the Decision Balance Scale (DBS) questionnaire¹⁷. This dialogue system is complemented by a large language model (LLM) that generates the agent's verbal output, corresponding to the dialogue act selected by the dialogue manager. This component enables contextual adaptation of the dialogue, marking an improvement over more rigid, rule-based approaches⁶. While recent studies have explored the use of LLMs to generate MI-consistent dialogue, they do not account for user profiles^18,19. In contrast, this module dynamically adapts to the user's profile, which enhances rapport compared to these baseline systems.

The integration of these models into the Greta 2.0 platform enables dynamic MI interventions. Its modular implementation allows for the independent evaluation of each model's impact. In this paper, we present the method used to conduct experiments that assess the contributions of the MODIFF-8 and DREAM models to the MI dialogue delivered by the platform used. This method can also be used to deliver tailored MI interventions between sessions with MI practitioners. It requires a Windows computer with a 16 GB GPU and the installation of open-source software. Using this method, we demonstrate that our adaptation models improve the perceived quality of the MI delivery agent and enhance rapport building. We also observe improvement in the motivation of patients to change their behaviors when the agent uses adapted behaviors at the verbal and nonverbal levels, and conversational strategies levels. The results presented in this paper demonstrate the positive impact of tailoring the agent's MI intervention to each patient.

To support the design of this system, an in-depth literature review was conducted to explore the theoretical and practical foundations of nutritional coaching, including motivational interviewing, behavior change models, and user typologies. This review led to the development of a qualitative knowledge base structured around the intersection of two key dimensions: 1) users' readiness for change following the Transtheoretical Model¹⁹ that identifies 5 stages: pre-contemplation, contemplation, preparation, action, and maintenance, and 2) their eating behavior profiles²⁰ based on diet and psychologically related factors. Three profiles were identified: the intuitive eater (i.e., listens to their hunger and fullness cues, maintains a varied and predominantly plant-based diet, prioritizes high-quality, often organic foods), the emotional eater (i.e., copes with stress, loneliness, and joy through frequent snacking on sweets or fatty foods, often feeling guilty afterward) and, the restrictive eater (i.e., obsessed with food control, they count calories, follow frequent diets with yo-yo effects, and may have an eating disorder).

For each combination of stage and profile, context-specific dialogue strategies were proposed, including tailored SMART goals, motivational formulations, and concrete behavioral suggestions. The aim was to enable the virtual agent to dynamically adapt its interventions based on both psychological readiness and personal tendencies. This groundwork ensures a higher degree of personalization and empathy in interaction, which are crucial to sustaining user engagement and long-term behavioral change.

Protocol

The method received ethical IRB approval from the INSEAD institute (Institut européen d'administration des affaires) with acceptance numbers: INSEAD 2024-78 and INSEAD 2025-23. All participants provided informed consent and were compensated for their experiment time.

1. Recruitment of participants

1. Call for participants using the institute's mailing lists. No constraints on participants, except familiarity with digital interfaces and fluency in French. Three participants were excluded due to internet resource drops, which significantly increased the global system response latency.

2. Pre-installation requirements

1. Use a Windows computer. We use an Intel I7 CPU, 20 cores, 28 threads (50% usage), RAM 32GB (20% usage), 32GB Nvidia RTX 5000 GPU (20% usage).
2. Install Java SE Development Kit 8 (https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html). Install Visual C++ Redistributable for Visual Studio 2013 (https://www.microsoft.com/en-us/download/details.aspx?id=40784).
3. Request a Cereproc license (https://app.cereproc.com/). Install OpenFace (https://github.com/TadasBaltrusaitis/OpenFace). Get a webcam.

3. Greta installation

1. Download the software release from https://github.com/isir/greta/releases. Compile using NetBeans (detailed instructions at: https://github.com/isir/greta/wiki/Quick-start).

4. Mistral API key configuration

1. Get a Mistral API key from https://console.mistral.ai/api-keys.
2. Create the file: greta\bin\Common\Data\LLM\MICounselorRL\api_key.txt. Paste the API key into the created file.

5. DeepGram API key configuration

1. Get a DeepGram API key from https://console.deepgram.com/.
2. Create the file: greta\bin\Common\Data\DeepASR\DeepGram\api_key.txt.Paste the API key into the created file.

6. MODIFF model importation

1. Download the model weights from https://drive.google.com/file/d/1n3udJf5pSNnFmxs_kt_zcFcF0m7zoLLr/view?usp=sharing and place the file into greta\bin\Common\Data\MODIFF.
2. Download and uncompress dlib face detection models from https://dlib.net/files/mmod_human_face_detector.dat.bz2 and https://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2, then place them into bin/Common/Data/TurnManagement/dlib_models.
3. Download all VAP models in model/VAP from https://drive.google.com/drive/folders/10_MTPkRyJCoFLDt4LO_M9kRgoreh6gEF?usp=sharing and place them into bin/Common/Data/TurnManagement/models/.

7. Launching Greta 2.0

1. Launch Modular.jar. Click File > Open > Greta > Advanced > 20250128 Greta - Expé Lucie_full.xml.
2. Ensure that the configuration matches Figure 1.

8. Launching OpenFace

1. Start OpenFace Offline ZeroMQ.exe. A specific version of OpenFace allows direct communication with the Greta platform using the ZeroMQ protocol through port 5000.
2. In the Record tab, uncheck everything except Broadcast with ZeroMQ. It allows the software to send the extracted signals (AUs, Gaze, Head movements …) to the Greta platform.
3. In the File tab, click Open Webcam. Authorize live feature extraction using the available webcam.
4. Select the webcam to use and wait until the webcam is loaded (see Figure 2). Once the webcam is loaded, the tool will automatically start the live feature extraction and send the data to the Greta platform.

9. Launching MODIFF-8

1. In OpenFace2 Output Stream Reader, click the Connect button and wait for the list of available features, sent from OpenFace, to populate (see Figure 3).
2. Click Select All > Set, to validate the set of features you want the platform to continuously receive from Openface.
3. Under Modiff, click Launch and wait for the message: Connected to server at localhost 5560.
4. Click Connect. This step activates the connection between the facial expression generation module Modiff and the Greta OpenFace data receiver.
5. Under Filter, click Perform, allowing the virtual agent to perform the data generated by the Modiff Module.
6. In the MODIFF window, click Enable. Wait 90 s for the model to stabilize.

10. Launching ASR

1. In the DeepGram window, press Enable.

11. Launching DREAM

1. Select the Condition RL to use DREAM and Baseline to use a plain LLM.
2. In the MI Counsellor RL window, click Enable. Wait 30 s for the model to start.

12. Set up the room

1. Use a large monitor to put on a table with a chair in front. Put a webcam on top of the monitor facing the chair. Put a directional microphone on the table in front of the chair.

13. Welcoming the participant

1. Welcome the participant to a private room (see Figure 4). Brief the participant as per the speech in Supplementary File 1.
2. Have the participant fill out the DBS questionnaire on a 5-step Likert scale (see items in Table 1).
3. Ask the participant to talk into the microphone. Ensure the webcam captures the participant's head and upper body

14. Identify the participant profile

1. If the DBS score is negative, classify the participant as Resistant to Change. If the DBS score is between 0 and 2, classify the participant as Receptive. Otherwise, classify the participant as Open to Change.

15. Set the theme and profile

1. In the MI Counsellor RL window, set the theme to be discussed as the participant chose. Set the profile as identified in step 14.

16. Start the dialogue

1. In the MI Counsellor RL window, click Start to have the agent initiates the discussion with the participant. Clicking this button makes the dialogue DREAM, and multimodal behaviors MODIFF models automatically drive the agent's behaviors.

17. During the dialogue

1. Monitor the Answer paragraph of the MI Counsellor window to ensure the LLM output is not harmful.
2. In the DeepGram window, click Listen after the agent's turn is finished.
3. If the speech is not recognized, fill the Request paragraph of the MI Counsellor RL window with the participant's speech and click Send. This automatically creates the agent's response.
4. Wait for the participant's answer.
5. Repeat 17.1 to 17.4 until the dialogue ends. The dialogue stops on its own.

18. Post-intervention

1. Ask the participant to complete the post-intervention questionnaires (see items 1-9 in Table 1). Debrief the participant using the speech in Supplementary File 1.

Results

In the presented protocol, we collected interaction transcripts and user feedback using questionnaires adapted to the behaviors we want to measure. We compared the results of the interaction with or without our models.

Using the protocol to evaluate the impact of the MODIFF-8 adaptive expressions generation
The MODIFF-8 module was designed to enhance the social responsiveness of the virtual agent by enabling it to generate facial expressions that are both contextually appr...

Discussion

In this paper, we present a protocol for delivering tailored Motivational Interviewing interventions and for testing individual behavior generation modules within a controlled and reproducible interaction framework. This method is built upon the modular architecture of the Greta 2.0 platform, which offers control over the components involved in agent behavior, allowing researchers to easily toggle specific modules on or off. This flexibility makes Greta 2.0 well-suited for both experimental manipulations and targeted eva...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Computer	Dell		Computer with processor Intel Core i7-14700 and a Nvidia RTX5000
Microphone Headset	Epos		A Headset with a Microphone
Webcam	Logitech		For facial expression extraction
Webcam2	Logitech		For video recording