Name: Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models (Video) | JoVE Journal
Uploaded: 2025-12-23T14:15:00.000+00:00
Duration: 7 min 14 s
Description: Learn to build a virtual agent for real-time motivational interviewing using adaptive nonverbal behavior and large language models. This computational method enhances digital therapeutic interactions and user engagement.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This work was partially funded by the ANR-DFG-JST Panorama, ANR-JST-CREST TAPAS (19-JSTS-0001-01), Enhancer (ANR-22-EXEN-0004), and PEPR-Ensemble PC3 (ANR-22-EXEN-0004) projects. The evaluation presented in this paper was funded by the Idex Sorbonne Universit&#233; as part of the State support for the Investments for the Future programs. This work was also supported by a French government grant managed by the Agence Nationale de la Recherche as part of the France 2030 program, reference ANR-22-EXEN-0004 (PEPR eNSEMBLE / MATCHING) and 22-PESN-0009 (PEPR AUTONOM-HEALTH).

The method received ethical IRB approval from the INSEAD institute (Institut europ&#233;en d&#39;administration des affaires) with acceptance numbers: INSEAD 2024-78 and INSEAD 2025-23. All participants provided informed consent and were compensated for their experiment time.
1. Recruitment of participants
<ol>
	<li>Call for participants using the institute&#39;s mailing lists. No constraints on participants, except familiarity with digital interfaces and fluency in French. Three participants were excluded due to internet resource drops, which significantly increased the global system response latency.</li>
</ol>
2. Pre-installation requirements
<ol>
	<li>Use a Windows computer. We use an Intel I7 CPU, 20 cores, 28 threads (50% usage), RAM 32GB (20% usage), 32GB Nvidia RTX 5000 GPU (20% usage).</li>
	<li>Install Java SE Development Kit 8 (https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html). Install Visual C++ Redistributable for Visual Studio 2013 (https://www.microsoft.com/en-us/download/details.aspx?id=40784).</li>
	<li>Request a Cereproc license (https://app.cereproc.com/). Install OpenFace (https://github.com/TadasBaltrusaitis/OpenFace). Get a webcam.</li>
</ol>
3. Greta installation
<ol>
	<li>Download the software release from&#160;https://github.com/isir/greta/releases. Compile using NetBeans (detailed instructions at: https://github.com/isir/greta/wiki/Quick-start).</li>
</ol>
4. Mistral API key configuration
<ol>
	<li>Get a Mistral API key from https://console.mistral.ai/api-keys.</li>
	<li>Create the file: greta\bin\Common\Data\LLM\MICounselorRL\api_key.txt. Paste the API key into the created file.</li>
</ol>
5. DeepGram API key configuration
<ol>
	<li>Get a DeepGram API key from&#160;https://console.deepgram.com/.</li>
	<li>Create the file: greta\bin\Common\Data\DeepASR\DeepGram\api_key.txt.Paste the API key into the created file.</li>
</ol>
6. MODIFF model importation
<ol>
	<li>Download the model weights from&#160;https://drive.google.com/file/d/1n3udJf5pSNnFmxs_kt_zcFcF0m7zoLLr/view?usp=sharing and place the file into greta\bin\Common\Data\MODIFF.</li>
	<li>Download and uncompress dlib face detection models from&#160;https://dlib.net/files/mmod_human_face_detector.dat.bz2 and https://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2, then place them into bin/Common/Data/TurnManagement/dlib_models.</li>
	<li>Download all VAP models in model/VAP from&#160;https://drive.google.com/drive/folders/10_MTPkRyJCoFLDt4LO_M9kRgoreh6gEF?usp=sharing and place them into bin/Common/Data/TurnManagement/models/.</li>
</ol>
7. Launching Greta 2.0
<ol>
	<li>Launch Modular.jar. Click File &#62; Open &#62; Greta &#62; Advanced &#62; 20250128 Greta - Exp&#233; Lucie_full.xml.</li>
	<li>Ensure that the configuration matches Figure 1.</li>
</ol>
8. Launching OpenFace
<ol>
	<li>Start OpenFace Offline ZeroMQ.exe. A specific version of OpenFace allows direct communication with the Greta platform using the ZeroMQ protocol through port 5000.</li>
	<li>In the Record tab, uncheck everything except Broadcast with ZeroMQ. It allows the software to send the extracted signals (AUs, Gaze, Head movements &#8230;) to the Greta platform.</li>
	<li>In the File tab, click Open Webcam. Authorize live feature extraction using the available webcam.</li>
	<li>Select the webcam to use and wait until the webcam is loaded (see Figure 2). Once the webcam is loaded, the tool will automatically start the live feature extraction and send the data to the Greta platform.</li>
</ol>
9. Launching MODIFF-8
<ol>
	<li>In OpenFace2 Output Stream Reader, click the Connect button and wait for the list of available features, sent from OpenFace, to populate (see Figure 3).</li>
	<li>Click Select All &#62; Set, to validate the set of features you want the platform to continuously receive from Openface.</li>
	<li>Under Modiff, click Launch and wait for the message: Connected to server at localhost 5560.</li>
	<li>Click Connect. This step activates the connection between the facial expression generation module Modiff and the Greta OpenFace data receiver.</li>
	<li>Under Filter, click Perform, allowing the virtual agent to perform the data generated by the Modiff Module.</li>
	<li>In the MODIFF window, click Enable. Wait 90 s for the model to stabilize.</li>
</ol>
10. Launching ASR
<ol>
	<li>In the DeepGram window, press Enable.</li>
</ol>
11. Launching DREAM
<ol>
	<li>Select the Condition RL to use DREAM and Baseline to use a plain LLM.</li>
	<li>In the MI Counsellor RL window, click Enable. Wait 30 s for the model to start.</li>
</ol>
12. Set up the room
<ol>
	<li>Use a large monitor to put on a table with a chair in front. Put a webcam on top of the monitor facing the chair. Put a directional microphone on the table in front of the chair.</li>
</ol>
13. Welcoming the participant
<ol>
	<li>Welcome the participant to a private room (see Figure 4). Brief the participant as per the speech in Supplementary File 1.</li>
	<li>Have the participant fill out the DBS questionnaire on a 5-step Likert scale (see items in Table 1).</li>
	<li>Ask the participant to talk into the microphone. Ensure the webcam captures the participant&#39;s head and upper body</li>
</ol>
14. Identify the participant profile
<ol>
	<li>If the DBS score is negative, classify the participant as Resistant to Change. If the DBS score is between 0 and 2, classify the participant as Receptive. Otherwise, classify the participant as Open to Change.</li>
</ol>
15. Set the theme and profile
<ol>
	<li>In the MI Counsellor RL window, set the theme to be discussed as the participant chose. Set the profile as identified in step 14.</li>
</ol>
16. Start the dialogue
<ol>
	<li>In the MI Counsellor RL window, click Start to have the agent initiates the discussion with the participant. Clicking this button makes the dialogue DREAM, and multimodal behaviors MODIFF models automatically drive the agent&#39;s behaviors.</li>
</ol>
17. During the dialogue
<ol>
	<li>Monitor the Answer paragraph of the MI Counsellor window to ensure the LLM output is not harmful.</li>
	<li>In the DeepGram window, click Listen after the agent&#39;s turn is finished.</li>
	<li>If the speech is not recognized, fill the Request paragraph of the MI Counsellor RL window with the participant&#39;s speech and click Send. This automatically creates the agent&#39;s response.</li>
	<li>Wait for the participant&#39;s answer.</li>
	<li>Repeat 17.1 to 17.4 until the dialogue ends. The dialogue stops on its own.</li>
</ol>
18. Post-intervention
<ol>
	<li>Ask the participant to complete the post-intervention questionnaires (see items 1-9 in Table 1). Debrief the participant using the speech in Supplementary File 1.</li>
</ol>

<ol><li>Miller, W. R., Rollnick, S. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aMiller%2C+WR+%22Motivational+Interviewing%3A+Helping+People+Change%22">Motivational Interviewing: Helping People Change</a>. , Guilford Press. (2012).</li>
<li>Towards a chatbot for digital counselling. Cameron, G., et al. Proc 31st Br Comp Soc Human Comp Interact Conf, , 1-7 (2017).</li>
<li>Denecke, K., Vaaheesan, S., Arulnathan, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+Mental+Health+Chatbot+for+Regulating+Emotions+%28SERMO%29+-+Concept+and+Usability+Test%5BTitle%5D)+AND+%22IEEE+Transact+Emerging+Topics+Comput%22%5BJournal%5D)">A Mental Health Chatbot for Regulating Emotions (SERMO) - Concept and Usability Test.</a> IEEE Transact Emerging Topics Comput. 9 (3), 1170-1182 (2021).</li>
<li>Fiske, A., Henningsen, P., Buyx, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Your+Robot+Therapist+Will+See+You+Now%3A+Ethical+Implications+of+Embodied+Artificial+Intelligence+in+Psychiatry%2C+Psychology%2C+and+Psychotherapy%5BTitle%5D)+AND+%22J+Med+Internet+Res%22%5BJournal%5D)">Your Robot Therapist Will See You Now: Ethical Implications of Embodied Artificial Intelligence in Psychiatry, Psychology, and Psychotherapy.</a> J Med Internet Res. 21 (5), e13216(2019).</li>
<li>Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. Bickmore, T. W., Pfeifer, L. M., Jack, B. W. Proc SIGCHI Conf Human Factors Comp Syst, , 1265-1274 (2009).</li>
<li>Fitzpatrick, K. K., Darcy, A., Vierhile, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Delivering+Cognitive+Behavior+Therapy+to+Young+Adults+With+Symptoms+of+Depression+and+Anxiety+Using+a+Fully+Automated+Conversational+Agent+%28Woebot%29%3A+A+Randomized+Controlled+Trial%5BTitle%5D)+AND+%22JMIR+Mental+Health%22%5BJournal%5D)">Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial.</a> JMIR Mental Health. 4 (2), e19(2017).</li>
<li>An Affectively Aware Virtual Therapist for Depression Counseling. Ring, L., Bickmore, T., Pedrelli, P. ACM SIGCHI Conf Human Factors Comput Syst (CHI) Workshop Comput Mental Health, , 1-4 (2016).</li>
<li>Managing Chronic Conditions with a Smartphone-based Conversational Virtual Agent. Bickmore, T. W., et al. Proc 18th Int Conf Intelligent Virtual Agents, , 119-124 (2018).</li>
<li>Mercado, J., Espinosa-Curiel, I. E., Martínez-Miranda, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Embodied+Conversational+Agents+Providing+Motivational+Interviewing+to+Improve+Health-Related+Behaviors%3A+Scoping+Review%5BTitle%5D)+AND+%22J+Med+Internet+Res%22%5BJournal%5D)">Embodied Conversational Agents Providing Motivational Interviewing to Improve Health-Related Behaviors: Scoping Review.</a> J Med Internet Res. 25, e52097(2023).</li>
<li>Masterson Creber, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Motivational+interviewing+to+improve+self-care+for+patients+with+chronic+heart+failure%3A+MITI-HF+randomized+controlled+trial%5BTitle%5D)+AND+%22Patient+Educ+Couns%22%5BJournal%5D)">Motivational interviewing to improve self-care for patients with chronic heart failure: MITI-HF randomized controlled trial.</a> Patient Educ Couns. 99 (2), 256-264 (2016).</li>
<li>Beyond Words: Decoding Facial Expression Dynamics in Motivational Interviewing. Younsi, N., Pelachaud, C., Chaby, L. Proc 2024 Joint Int Conf Comp Linguistics Lang Resources Evaluation, , 2365-2374 (2024).</li>
<li>Galland, L., Pecune, F., Pelachaud, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((EMMI+-+Empathic+Multimodal+Motivational+Interviews%3A+Automatic+Multimodal+Features+Extraction+And+Patient+Type+Definition%5BTitle%5D)+AND+%22Lang+Resources+Evaluation%22%5BJournal%5D)">EMMI - Empathic Multimodal Motivational Interviews: Automatic Multimodal Features Extraction And Patient Type Definition.</a> Lang Resources Evaluation. , (2025).</li>
<li>Galland, L., Pelachaud, C., Pecune, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((EMMI+--+Empathic+Multimodal+Motivational+Interviews+Dataset%3A+Analyses+and+Annotations%5BTitle%5D)+AND+%22arXiv%22%5BJournal%5D)">EMMI -- Empathic Multimodal Motivational Interviews Dataset: Analyses and Annotations.</a> arXiv. , (2024).</li>
<li>Greta 2.0: Social Interactive Agent system, optimized for neural network integration. Saga, T., Galland, L., Younsi, N., Pelachaud, C. Proc ACM Int Conf Intelligent Virtual Agents, , 1-10 (2025).</li>
<li>Younsi, N., Pelachaud, C., Chaby, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Modiff-8+to+Better+Motivate%3A+Live+Adaptive+Human-Socially+Interactive+Agent+Interaction%5BTitle%5D)+AND+%22SSRN%22%5BJournal%5D)">Modiff-8 to Better Motivate: Live Adaptive Human-Socially Interactive Agent Interaction.</a> SSRN. , (2025).</li>
<li>Galland, L., Pelachaud, C., Pecune, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Tailored+Conversations+beyond+LLMs%3A+A+RL-Based+Dialogue+Manager%5BTitle%5D)+AND+%22ACL+ARR%22%5BJournal%5D)">Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager.</a> ACL ARR. , (2025).</li>
<li>Velicer, W. F., DiClemente, C. C., Prochaska, J. O., Brandenburg, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Decisional+balance+measure+for+assessing+and+predicting+smoking+status%5BTitle%5D)+AND+%22J+Personality+Social+Psychol%22%5BJournal%5D)">Decisional balance measure for assessing and predicting smoking status.</a> J Personality Social Psychol. 48 (5), 1279-1289 (1985).</li>
<li>Virtual Agents for Alcohol Use Counseling: Exploring LLM-Powered Motivational Interviewing. Steenstra, I., Nouraei, F., Arjmand, M., Bickmore, T. Proc ACM Int Conf Intelligent Virtual Agents, , 1-10 (2024).</li>
<li>Galvão Gomes da Silva, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Experiences+of+a+Motivational+Interview+Delivered+by+a+Robot%3A+Qualitative+Study%5BTitle%5D)+AND+%22J+Med+Internet+Res%22%5BJournal%5D)">Experiences of a Motivational Interview Delivered by a Robot: Qualitative Study.</a> J Med Internet Res. 20 (5), e116(2018).</li>
<li>Torre, I., Goslin, J., White, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((If+your+device+could+smile%3A+People+trust+happy-sounding+artificial+agents+more%5BTitle%5D)+AND+%22Comp+Human+Behav%22%5BJournal%5D)">If your device could smile: People trust happy-sounding artificial agents more.</a> Comp Human Behav. 105, 106215(2020).</li>
<li>Rapport and facial expression. 2009 3rd Int Conf Affective Comp Intelligent Interact Workshops. Wang, N., Gratch, J. , 1-6 (2009).</li>
<li>Guy, L., Jean-Noël, G., Val, M., Cross, E. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Tell+me+more%21+Assessing+interactions+with+social+robots+from+speech%5BTitle%5D)+AND+%22Paladyn%2C+J+Behav+Robotics%22%5BJournal%5D)">Tell me more! Assessing interactions with social robots from speech.</a> Paladyn, J Behav Robotics. 12 (1), 136-159 (2020).</li>
<li>Krämer, N., Kopp, S., Becker-Asano, C., Sommer, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Smile+and+the+world+will+smile+with+you-The+effects+of+a+virtual+agent%27s+smile+on+users%27+evaluation+and+behavior%5BTitle%5D)+AND+%22Int+J+Human-Comp+Studies%22%5BJournal%5D)">Smile and the world will smile with you-The effects of a virtual agent's smile on users' evaluation and behavior.</a> Int J Human-Comp Studies. 71 (3), 335-349 (2013).</li>
<li>Huang, L., Morency, L. P., Gratch, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Virtual+Rapport+2.0%5BTitle%5D)+AND+%22Intell+Virtual+Agents%22%5BJournal%5D)">Virtual Rapport 2.0.</a> Intell Virtual Agents. , 68-79 (2011).</li>
<li>Kramer Schmidt, L., Andersen, K., Nielsen, A. S., Moyers, T. B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Lessons+learned+from+measuring+fidelity+with+the+Motivational+Interviewing+Treatment+Integrity+code+%28MITI+4%29%5BTitle%5D)+AND+%22J+Subst+Abuse+Treatment%22%5BJournal%5D)">Lessons learned from measuring fidelity with the Motivational Interviewing Treatment Integrity code (MITI 4).</a> J Subst Abuse Treatment. 97, 59-67 (2019).</li>
<li>Madson, M. B., Mohn, R. S., Schumacher, J. A., Landry, A. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Measuring+Client+Experiences+of+Motivational+Interviewing+during+a+Lifestyle+Intervention%5BTitle%5D)+AND+%22Meas+Evaluat+Counseling+Dev%22%5BJournal%5D)">Measuring Client Experiences of Motivational Interviewing during a Lifestyle Intervention.</a> Meas Evaluat Counseling Dev. 48 (2), 140-151 (2015).</li>
<li>Burgoon, J. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Expectancy+Violations+Theory%5BTitle%5D)+AND+%22Int+Encycl+Interpersonal+Comm%22%5BJournal%5D)">Expectancy Violations Theory.</a> Int Encycl Interpersonal Comm. , 1-9 (2015).</li>
<li>Baker, Z. G., Watlington, E. M., Knee, C. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+role+of+rapport+in+satisfying+one%27s+basic+psychological+needs%5BTitle%5D)+AND+%22Motivat+Emot%22%5BJournal%5D)">The role of rapport in satisfying one's basic psychological needs.</a> Motivat Emot. 44 (2), 329-343 (2020).</li>
<li>Ryan, R. M., Deci, E. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Self-Determination+Theory%5BTitle%5D)+AND+%22Encycl+Quality+Life+Well-Being+Res%22%5BJournal%5D)">Self-Determination Theory.</a> Encycl Quality Life Well-Being Res. , 6229-6235 (2023).</li>
<li>SMART-DREAM: To Condition or Not to Condition A Study on the Impact of LLM Conditioning on Motivational Interview Dialog Virtual Agent. Galland, L., Pelachaud, C., Pecune, F. Proc ACM Int Conf Intelligent Virtual Agents, , 1-9 (2025).</li>
<li>Can Virtual Human Build Rapport and Promote Learning. Wang, N., Gratch, J. Proc 2009 Conf Artif Intelligence Educat, , 737-739 (2009).</li>
<li>Eeckhout, C., Francaux, M., Heeren, A., Philippot, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Mesure+de+la+balance+d%C3%A9cisionnelle+en+vue+de+pratiquer+une+activit%C3%A9+physique+r%C3%A9guli%C3%A8re+%28BDAP%29+%3A+adaptation+et+validation+francophone+de+l%27%C3%A9chelle+Decisional+Balance+for+Exercise%5BTitle%5D)+AND+%22Eur+Rev+Appl+Psychol%22%5BJournal%5D)">Mesure de la balance décisionnelle en vue de pratiquer une activité physique régulière (BDAP) : adaptation et validation francophone de l'échelle Decisional Balance for Exercise.</a> Eur Rev Appl Psychol. 63 (3), 185-191 (2013).</li>
<li>Carey, K. B., Maisto, S. A., Carey, M. P., Purnine, D. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Measuring+Readiness-to-Change+Substance+Misuse+Among+Psychiatric+Outpatients+I.+Reliability+and+Validity+of+Self-Report+Measures%5BTitle%5D)+AND+%22J+Studies+Alcohol%22%5BJournal%5D)">Measuring Readiness-to-Change Substance Misuse Among Psychiatric Outpatients I. Reliability and Validity of Self-Report Measures.</a> J Studies Alcohol. 62 (1), 79-88 (2001).</li>
<li>Skantze, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Turn-taking+in+Conversational+Systems+and+Human-Robot+Interaction%3A+A+Review%5BTitle%5D)+AND+%22Comp+Speech+Lang%22%5BJournal%5D)">Turn-taking in Conversational Systems and Human-Robot Interaction: A Review.</a> Comp Speech Lang. 67, 101178(2021).</li>
<li>Cross, R., Greaves, C., Withall, J., Kritz, M., Stathi, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+qualitative+longitudinal+study+of+motivation+in+the+REtirement+in+ACTion+%28REACT%29+physical+activity+intervention+for+older+adults+with+mobility+limitations%5BTitle%5D)+AND+%22Int+J+Behav+Nutrit+Phys+Activity%22%5BJournal%5D)">A qualitative longitudinal study of motivation in the REtirement in ACTion (REACT) physical activity intervention for older adults with mobility limitations.</a> Int J Behav Nutrit Phys Activity. 20 (1), 50(2023).</li>
</ol>

The authors have nothing to disclose.

In this paper, we present a protocol for delivering tailored Motivational Interviewing interventions and for testing individual behavior generation modules within a controlled and reproducible interaction framework. This method is built upon the modular architecture of the Greta 2.0 platform, which offers control over the components involved in agent behavior, allowing researchers to easily toggle specific modules on or off. This flexibility makes Greta 2.0 well-suited for both experimental manipulations and targeted eva...

Motivational interviewing (MI) is a collaborative therapeutic approach that encourages behavioral change. During MI sessions, MI practitioners help patients articulate and foster their motivation for change1. To this aim, they use dialogue strategies, such as reflections or questions, reinforced by non-verbal behaviors, such as smiles or specific head and body postures.
As the frequency of mental health issues has increased in recent years, a gap has emerged between the demand for mental health services and the available resources2. This has led to longer waiting times for patients before they can access therapy2,3. One proposed solution to alleviate this issue involves virtual agents capable of replicating MI for patients awaiting appointments. By offering immediate support and intervention, these virtual agents can help reduce the impact of waiting periods, especially in therapeutic contexts that require multiple sessions4. Several studies have demonstrated the effectiveness of such agents or chatbots5,6,7,8,9 whose goal is not to replace MI experts; they are intended to serve as a complementary tool within the therapeutic process. For instance, they could provide MI intervention in between sessions.
Although previous MI agents often exhibit limited adaptive behaviors, MI is inherently a cooperative and adaptive form of communication. Therefore, developing an agent capable of adapting its dialogue to the context could significantly enhance the effectiveness of therapy10. During MI sessions, therapists adjust both their verbal (conversational strategies) and nonverbal behaviors based on the patient&#39;s responses, motivation, or engagement level11,12, as well as their individual profiles13. Depending on the patient&#39;s level of motivation, the therapist will modify their approach accordingly. Thus, personalization and adaptability are essential for developing effective MI virtual agents.
The Greta 2.0 platform14 is a modular virtual agent framework that supports real-time interactions. It has recently been extended to incorporate capabilities that enable the dynamic adaptation of generative AI models during interaction.
The platform includes a model called MODIFF-815, which generates real-time facial expressions that dynamically adapt to the user&#39;s own facial expressions. This diffusion-based model was trained on multimodal data collected in the context of MI, namely the AnnoMI corpus, constituted of counselling session videos between therapist and client pairs, collected on streaming platforms, such as YouTube and Vim&#233;o, and annotated in dialogue acts (a classification of utterances that denotes the actions performed e.g., Question, Greeting&#8230;) and action units (a taxonomy of human facial expression). This data conditions the generation not only on the user&#39;s current facial expression but also on MI-specific dialogue acts identified during the interaction, linked to the conversational strategies adopted. These include MI-relevant users&#39; dialogue acts, such as change talk, as well as the agent&#39;s own dialogue acts, such as reflection or providing information. By integrating these multimodal signals in real time, the system allows the virtual agent to express socially and contextually appropriate facial expressions. By aligning its non-verbal behavior (NVB) to the context of the interaction, this virtual agent is designed to foster social rapport and support the initiation of intrinsic motivation throughout the interview process15.
Additionally, the platform integrates a dialogue system specifically designed for MI, called DREAM16. DREAM comprises a dialogue manager that selects the agent&#39;s next dialogue act based on the current dialogue state and the user&#39;s profile. Indeed, our analyses of the AnnoMI corpus revealed three profiles of MI patients: Open to Change (when the patient is motivated to change and seeks support), Receptive (when the patient has not yet considered change), and Resistant (when the patient does not intend to change)13. Before interacting with the virtual agent, users are attributed one of these three profiles according to their answers to the Decision Balance Scale (DBS) questionnaire17. This dialogue system is complemented by a large language model (LLM) that generates the agent&#39;s verbal output, corresponding to the dialogue act selected by the dialogue manager. This component enables contextual adaptation of the dialogue, marking an improvement over more rigid, rule-based approaches6. While recent studies have explored the use of LLMs to generate MI-consistent dialogue, they do not account for user profiles18,19. In contrast, this module dynamically adapts to the user&#39;s profile, which enhances rapport compared to these baseline systems.
The integration of these models into the Greta 2.0 platform enables dynamic MI interventions. Its modular implementation allows for the independent evaluation of each model&#39;s impact. In this paper, we present the method used to conduct experiments that assess the contributions of the MODIFF-8 and DREAM models to the MI dialogue delivered by the platform used. This method can also be used to deliver tailored MI interventions between sessions with MI practitioners. It requires a Windows computer with a 16 GB GPU and the installation of open-source software. Using this method, we demonstrate that our adaptation models improve the perceived quality of the MI delivery agent and enhance rapport building. We also observe improvement in the motivation of patients to change their behaviors when the agent uses adapted behaviors at the verbal and nonverbal levels, and conversational strategies levels. The results presented in this paper demonstrate the positive impact of tailoring the agent&#39;s MI intervention to each patient.
To support the design of this system, an in-depth literature review was conducted to explore the theoretical and practical foundations of nutritional coaching, including motivational interviewing, behavior change models, and user typologies. This review led to the development of a qualitative knowledge base structured around the intersection of two key dimensions: 1) users&#39; readiness for change following the Transtheoretical Model19 that identifies 5 stages: pre-contemplation, contemplation, preparation, action, and maintenance, and 2) their eating behavior profiles20 based on diet and psychologically related factors. Three profiles were identified: the intuitive eater (i.e., listens to their hunger and fullness cues, maintains a varied and predominantly plant-based diet, prioritizes high-quality, often organic foods), the emotional eater (i.e., copes with stress, loneliness, and joy through frequent snacking on sweets or fatty foods, often feeling guilty afterward) and, the restrictive eater (i.e., obsessed with food control, they count calories, follow frequent diets with yo-yo effects, and may have an eating disorder).
For each combination of stage and profile, context-specific dialogue strategies were proposed, including tailored SMART goals, motivational formulations, and concrete behavioral suggestions. The aim was to enable the virtual agent to dynamically adapt its interventions based on both psychological readiness and personal tendencies. This groundwork ensures a higher degree of personalization and empathy in interaction, which are crucial to sustaining user engagement and long-term behavioral change.

<table><tbody><tr><td>Computer</td><td>Dell</td><td></td><td>Computer with processor Intel Core i7-14700 and a Nvidia RTX5000</td></tr><tr><td>Microphone Headset</td><td>Epos</td><td></td><td>A Headset with a Microphone</td></tr><tr><td>Webcam</td><td>Logitech</td><td></td><td>For facial expression extraction</td></tr><tr><td>Webcam2</td><td>Logitech</td><td></td><td>For video recording</td></tr></tbody></table>

virtual agent for real-time motivational interviewing by integrating adaptive nonverbal behavior and language models

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models

The growing demand for therapeutic support increasingly exceeds the capacity of available professionals. A virtual agent capable of performing motivational interviewing (MI) offers a promising solution to assist patients in reaching their goal of behavior change between sessions with human therapists. MI is inherently a cooperative and adaptive form of communication. Therefore, developing an agent capable of adapting its conversational strategies to the context could significantly enhance the effectiveness of therapy. During MI sessions, human therapists adjust both their verbal and nonverbal behaviors based on the human patients' responses, as well as their profiles. Depending on the patient's level of motivation, the therapist will modify their approach accordingly. Thus, personalization and adaptability are essential for developing effective MI virtual agents. In this paper, we present a virtual agent capable of conducting MI sessions by dynamically adapting verbally and nonverbally to users in real time. Leveraging state-of-the-art models, this system enables MI interactions. The virtual agent is embodied using the Greta 2.0 platform. Its nonverbal behavior is generated through a diffusion model called MODIFF, which adapts to the user's facial expressions and their readiness to change. These facial expressions were learned on an MI corpus and validated through a dedicated user study. The dialogue is generated using a state-of-the-art large language model (LLM), enhanced by a dialogue manager specifically designed for MI, with a reinforcement learning approach, and validated through user testing. Furthermore, the dialogue manager is able to adapt to different user profiles. The resulting platform is open-source and facilitates the generation of real-time, multimodal MI dialogues, providing new tools for digitally mediated therapeutic interactions.

In the presented protocol, we collected interaction transcripts and user feedback using questionnaires adapted to the behaviors we want to measure. We compared the results of the interaction with or without our models.
Using the protocol to evaluate the impact of the MODIFF-8 adaptive expressions generation 
The MODIFF-8 module was designed to enhance the social responsiveness of the virtual agent by enabling it to generate facial expressions that are both contextually appr...

We present an open-source virtual agent platform for conducting real-time motivational interviews, combining state-of-the-art language and diffusion models to adapt to users' behavior and profile. Utilizing the Greta 2.0 platform, it supports various topics, including nutrition and sport-focused interventions, and offers a flexible, validated tool for enhancing digital therapeutic interactions.

The growing demand for therapeutic support increasingly exceeds the capacity of available professionals. A virtual agent capable of performing ...

Our research investigates multimodal interaction with a focus on adapting verbal and nonverbal behavior of a virtual agent, such as facial expression during interaction. Recent analysis in machine learning, particularly large language model, enables more natural and flexible human-computer interaction. To begin, prepare a Windows computer for the experiment.Install Java SE Development Kit 8 from the official distribution source. Then install the Visual C+Redistributable for Visual Studio 2013. Request a CereProc license through the online application portal.Install OpenFace from the provided repository. Obtain a webcam for video capture. Next, download the Greta software release from the specified repository.Compile the software using NetBeans following the provided instructions. Obtain a Mistral application programming interface key from the online console. Create the file, then paste the application programming interface key into the created file.Now, obtain a Deepgram application programming interface key from the online console. Create the file path and paste the application programming interface key into the created file. For MoDiff model import, first download the MoDiff model weights from the provided source.Place the downloaded file into the MoDiff data folder. Next launch Modular JAR. Click on File, Open, Greta, Advanced, and choose 20250128 Greta Expe Lucie_full.xml.Ensure that the configuration displayed matches the expected setup. Now start the OpenFace offline ZeroMQ program. In the Record tab, uncheck all options except broadcast with ZeroMQ.In the File tab, click Open Webcam. Authorize live feature extraction using the available webcam. Select the webcam to use.Wait until the webcam is loaded and live feature extraction starts automatically. Next, in the OpenFace2 Output Stream Reader, click Connect. Wait for the available feature list to populate.Click on Select All and then set to validate the selected features. Under MoDiff, click Launch and wait for the connection confirmation message. Then press Connect to activate the connection between MoDiff and the data receiver.Under Filter, click Perform to allow execution of the generated data. In the MoDiff window, click Enable. Wait 90 seconds for the model to stabilize.To launch ASR, in the Dee gram window, press Enable. Then select the condition RL to use dream or baseline to use a plain large language model. In the MI Counselor RL window, click Enable.Wait 30 seconds for the model to start. Place a large monitor on the table with a chair positioned in front. Position a webcam on top of the monitor facing the chair.Place a directional microphone on the table in front of the chair. Have the participant complete the Decision Balance Scale or DBS questionnaire using a five-step Likert scale. Then ask the participant to speak into the microphone.To begin, identify the participant profile and evaluate the DBS score. Classify the participant as resistant, hesitant, or open to change based on the score. In the MI Counselor RL window, choose the discussion theme chosen by the participant.Click Start to allow the agent to initiate the discussion with the participant. Monitor the answer section of the MI Counselor window for harmful output. In the Deepgram window, click Listen after the agent finishes its turn.If speech is not recognized, enter the participant's speech into the request field and click Send. Wait for the participant's answer. Ask the participant to complete the post-intervention questionnaires.Then debrief the participant using the prepared speech. Users interacting with the agent with adaptive facial expressions expressed more positive sentimentality in their disclosure than users in the other behavior conditions. No significant differences were observed across the three expression conditions in subjective questionnaire scores assessing attitude, social rapport, and motivational interviewing quality.The mismatched facial expression condition, that is when interpersonal contingency is no longer respected, was generally rated lower than both the inexpressive, that is when the agent shows no expression, and adaptive, that is the agent's expression is driven by our model conditions across subjective measures. Perceived attitude had a strong direct effect on perceived interviewing quality. The relationship between perceived attitude and motivational interviewing quality was partially mediated by social rapport, accounting for 43%of the effect.Participants interacting with the adaptive dream dialogue manager reported significantly higher rapport than those interacting with the plain large language model baseline. The dream dialogue manager showed better adaptation to different participant profiles than the baseline model in motivation scores measured by the Decisional Balance Scale. Both the baseline and dream conditions produced client evaluation of motivational interviewing scores above the therapeutic threshold.Our finding shows that adapting verbal and nonverbal behavior significantly improved the effectiveness of the agent and improved the perception of social interactive agents. Our platform enables the evaluation of other adaptive components, such as gesture generation and the expression of other dialogue topics. Future research will focus on long-term interaction and continuous adaptation across multiple dialogue sessions.

virtual-agent-for-real-time-motivational-interviewing-integrating

Research

JoVE Journal

Behavior

This study employs ERPs to investigate how agent type (human/AI) and language style (humorous/rational) influence ride-hailing service recovery satisfaction, aiming to uncover neurocognitive mechanisms.

Humor or Rationality? The Neural Mechanisms of How Agent Type and Language Style Influence Satisfaction with Ride-Hailing Service Failure Recovery

Here, we describe virtual-hand and virtual-face illusion paradigms that can be used to study body-related self-perception/-representation. They have already been used in various studies to demonstrate that, under specific conditions, a virtual hand or face can be incorporated into one's body representation, suggesting that body representations are rather flexible.

Creating Virtual-hand and Virtual-face Illusions to Investigate Self-representation

The visual world paradigm monitors participants' eye movements in the visual workspace as they are listening to or speaking a spoken language. This paradigm can be used to investigate the online processing of a wide range of psycholinguistic questions, including semantically complex statements, such as disjunctive statements.

Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

Here, we present a protocol to develop the computerized adaptive testing system of the functional assessment of stroke (CAT-FAS). The CAT-FAS can simultaneously assess four functions (two motor functions [upper and lower extremities], postural control, and basic activities of daily living) with sufficient reliability and administrative efficiency.

Computerized Adaptive Testing System of Functional Assessment of Stroke

Using the novel FishSim Animation Toolchain, we present a protocol for non-invasive visual manipulation of public information in the context of mate-choice copying in sailfin mollies. FishSim Animation Toolchain provides an easy-to-use framework for the design, animation and presentation of computer-animated fish stimuli for behavioral experiments with live test fish.

Using the FishSim Animation Toolchain to Investigate Fish Behavior: A Case Study on Mate-Choice Copying In Sailfin Mollies

Here we present a protocol for decomposing the variance in reading comprehension into the unique and common effects of language and decoding.

Decomposing the Variance in Reading Comprehension to Reveal the Unique and Common Effects of Language and Decoding

The article describes a novel approach for analyzing dynamic online social interactions (in an online context) exemplified by a study of an online community of recovery from alcohol and drug addiction.

Integrating Computerized Linguistic and Social Network Analyses to Capture Addiction Recovery Capital in an Online Community

We present a protocol to dissociate the intertwining factors of integrative difficulty and unexpectedness in semantically anomalous sentences by applying multiple repetitions to enhance participant's expectancy for anomalous sentences. The dissociation helps to investigate the major contributor of elicited event-related potentials (ERP) effects such as N400 in language studies.

Dissociation of the Confounding Influences of Expectancy and Integrative Difficulty Residing in Anomalous Sentences in Event-related Potential Studies

Eye tracking studies using a preferential looking paradigm can be used to study infants' emerging understanding of, and attention to, their external visual world.

Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm

This paper describes real-time electroencephalography-triggered transcranial magnetic stimulation to study and modulate human brain networks.

Brain State-dependent Brain Stimulation with Real-time Electroencephalography-Triggered Transcranial Magnetic Stimulation

Learn to build a virtual agent for real-time motivational interviewing using adaptive nonverbal behavior and large language models. This computational method enhances digital therapeutic interactions and user engagement.

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models

Explore More Videos