Name: Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation (Video) | JoVE Journal
Uploaded: 2016-08-26T13:00:00.000+00:00
Duration: 7 min 12 s
Description: Learn standardized data capture and analysis for automated facial expression analysis in sensory evaluation. This protocol measures emotional responses to beverages and liquefied foods using AFEA software.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This project was funded, in part, by ConAgra Foods (Omaha, NE, USA), the Virginia Agricultural Experiment Station, the Hatch Program of the National Institute of Food and Agriculture, U.S. Department of Agriculture, and the Virginia Tech Water INTERface Interdisciplinary Graduate Education Program.

Ethics Statement: This study was pre-approved by Virginia Tech Institutional Review Board (IRB) (IRB 14-229) prior to starting the project.
Caution: Human subject research requires informed consent prior to participation. In addition to IRB approval, consent for use of still or video images is also required prior to releasing any images for print, video, or graphic imaging. Additionally, food allergens are disclosed prior to testing. Participants are asked prior to panel start if they have any intolerance, allergies or other concerns.
Note: Exclusion Criteria: Automated facial expression analysis is sensitive to thick framed glasses, heavily bearded faces and skin tone. Participants who have these criteria are incompatible with software analysis due to an increased risk of failed videos. This is attributed to the software&#39;s inability to find the face.
1. Sample Preparation and Participant Recruitment
<ol>
	<li>Prepare beverage or soft food samples.
	<ol>
		<li>Prepare intensified dairy solutions using 2% milk and suggested flavors from Costello and Clark (2009)22 as well as other flavors. Prepare the following solutions: (1) unflavored milk (2% reduced fat milk); (2) unflavored water (drinking water); (3) vanilla extract flavor in milk (0.02 g/ml) (imitation clear vanilla flavor); and (4) salty flavor in milk (0.004 g/ml iodized salt). 
		Note: These solutions are used for demonstration purposes only.</li>
		<li>Pour half ounce aliquots (~15 g) of each solution into 2 oz. transparent plastic sample cups and cap with color coded lids. 
		Note: It is recommended to use transparent cups; however, it is up to the researcher&#39;s discretion.</li>
	</ol>
	</li>
	<li>Recruit participants from the campus or the local community to participate in the study. 
	Note: Participant sample size needed for a study is up to the discretion of the researcher. We recommend a range of 10 to 50 participants.</li>
	<li>Obtain human subject consent prior to participation in the study.</li>
</ol>
2. Preparation of Panel Room for Video Capture
Note: This protocol is for data capture in a sensory evaluation laboratory. This protocol is to make AFEA data capture useful for a sensory booth setting.
<ol>
	<li>Use individual booths with a touchscreen monitor in front of them (face level) to keep their focus forward and to prevent looking down.</li>
	<li>Use adjustable height chairs with back support. 
	Note: These are essential for allowing participants to be vertically adjusted and placed in a suitable range for video capture. Use stationary chairs (no rolling feature) with adjustable back height support so the participant&#39;s movements are reduced.</li>
	<li>Set overhead lighting at &#34;100% daylight&#34; for optimal facial emotional video capture (Illuminant 6504K; R=206; G=242; B=255). 
	Note: To avoid intense shadowing, diffuse frontal lighting is ideal while the light intensity or color is not as relevant20. Ultimately, it is up to the discretion of the researcher, individual protocol/methodology, and environment to control lighting for capture.</li>
	<li>Affix an adjustable camera above the touchscreen monitor for recording.
	<ol>
		<li>Use a camera with a resolution of at least 640 x 480 pixels (or higher)20. Discuss the required camera capabilities with the software provider before purchase and installation20. Note: The aspect ratio is not important20.</li>
		<li>Set camera capture speed to 30 frames per second (or other standard speed) for consistency.</li>
		<li>Connect and ensure media recording software is set up to the camera to record and save participant videos.</li>
	</ol>
	</li>
</ol>
3. Participant Adjustment and Verbal Directions
<ol>
	<li>Have only one participant at a time evaluate the samples in the sensory booth. 
	Note: Testing more than one participant at the same time may interfere with the testing environment and disrupt the concentration of the participant or create bias.</li>
	<li>Upon arrival, give participants verbal instructions about the process and standard operating procedures.
	<ol>
		<li>Have the participants sit straight up and against the back of the chair.</li>
		<li>Adjust chair height, position of the chair (distance from the camera), and camera angle so that the participant&#39;s face is captured in the center of the video recording, with no shadows on chin or around eyes. 
		Note: In the sensory booth, the participant&#39;s head is roughly 20 - 24 inches away from the camera and the monitor with the face centered in the camera video feed.</li>
		<li>Instruct participants to remain seated as positioned and focused facing towards the monitor display. Additionally, instruct participants to refrain from any sudden movements post-sample consumption during the 30 sec evaluation period per sample.</li>
		<li>Instruct the participant to consume the entire beverage or liquefied food sample and swallow.</li>
		<li>Instruct the participant to quickly move the sample cup below the chin and down to the table immediately after the sample is in the mouth. This is to eliminate facial occlusion. Remind them to keep looking toward the monitor. 
		Note: The sample carrier to deliver the sample is up to the discretion of the researcher. A straw or cup may be used. Regardless, initial facial occlusion is unavoidable because the face will be occluded or distorted due to consumption.</li>
	</ol>
	</li>
	<li>Instruct the participant to follow the instructions as they appear on the touchscreen monitor. Note: Instructions are automatically sequenced as programmed into the automated sensory software.</li>
</ol>
4. Individual Participant Process for Video Capture
<ol>
	<li>Confirm video camera is optimally capturing participant&#8217;s face while the participant is seated comfortably in the booth (before sample presentation) by viewing the computer monitor on which the video capture is displayed. Begin recording by clicking the record button on the computer monitor.</li>
	<li>Instruct participants to sip water to cleanse their palate.</li>
	<li>Provide treatments one at a time, starting with a baseline or control treatment (unflavored water). Identify each sample by a unique colored index card placed on top of each sample relating to the sample color code for sample treatment identification within the video. 
	Note: Programmed guidance on the touchscreen monitor instructs participants. The instructions direct the participant through a series of standardized steps for each treatment sample.</li>
	<li>Via the touchscreen monitor, direct the participant to:
	<ol>
		<li>Hold up the associated color index card pre-consumption for sample identification in the video. 
		Note: The color card is a way researchers can identify treatments in the video and mark the appropriate time frame (time zero) for sample evaluation.</li>
		<li>After holding the card briefly, place the card back on the tray.</li>
		<li>Fully consume the sample and wait approximately 30 seconds, enforced through the programmed guidance on the monitor, while facing towards the camera. 
		Note: The 30 sec controlled sampling period encompasses a time span adequate for the entire sampling evaluation period (i.e., showing the index card, opening a sample (removing the lid), consumption, and emotional capture).</li>
		<li>Enter their hedonic acceptability score on the touchscreen monitor (1=dislike extremely, 2=dislike very much, 3=dislike moderately, 4=dislike slightly, 5=neither like nor dislike, 6=like slightly, 7=like moderately, 8=like very much, 9=like extremely).</li>
		<li>Rinse mouth with drinking water before the next sample process.</li>
	</ol>
	</li>
</ol>
5. Evaluating Automated Facial Expression Analysis Options
Note: Many facial expression analysis software programs exist. Software commands and functions may vary. It is important to follow the manufacturer&#39;s user guidelines and reference manual20.
<ol>
	<li>Save recordings in a media format and transfer to the automated facial expression analysis software.</li>
	<li>Analyze participant videos using automated facial analysis software.
	<ol>
		<li>Double click on the software icon on the computer desktop.</li>
		<li>Once the program is open, click &#34;File&#34;, select &#34;New&#8230;&#34;, and select &#34;Project&#8230;&#34;</li>
		<li>In the pop up window, name the project and save the project.</li>
		<li>Add participants to the project by clicking the &#34;Add participants&#34; icon (Person with a (+) sign). More participants can be added by repeating this step.</li>
		<li>Add participant&#39;s video to the respective participant for analysis.
		<ol>
			<li>On the left side of the screen click the icon of the film reel with a plus (+) sign to add a video to analyze.</li>
			<li>Click the &#34;magnifying glass&#34; under the participant of interest to browse the video to add.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Analyze videos frame-by-frame under continuous calibration analysis settings in the software.
	<ol>
		<li>Click the pencil icon to adjust settings at the bottom of the window, under the &#34;settings&#34; tab for each participant video.
		<ol>
			<li>Set &#34;Face Model&#34; to General. Set &#34;Smoothen classifications&#34; to Yes. Set &#34;Sample Rate&#34; to Every frame.</li>
			<li>Set &#34;Image rotation&#34; to No. Set &#34;Continuous calibration&#34; to Yes. Set &#34;Selected calibration&#34; to None.</li>
		</ol>
		</li>
		<li>Save project settings.</li>
		<li>Press the batch analysis icon (the same red and black target-like symbol) to analyze the project videos.</li>
		<li>Save the results once analysis is completed. 
		Note: Other video settings exist in the software if researcher preference warrants another analysis method.</li>
		<li>Consider videos failures if serious facial occlusions or the inability to map the face persists during the specified post-consumption window (Figure 1). Additionally, if the model fails data will say &#34;FIT_FAILED&#34; or &#34;FIND_FAILED&#34; in the exported output files (Figure 2). This represents lost data since the software cannot classify or analyze the participant&#39;s emotions. 
		Note: AFEA translates facial muscle motion to neutral, happy, disgusted, sad, angry, surprised and scared on a scale from 0 (not expressed) to 1 (fully expressed) for each emotion.</li>
	</ol>
	</li>
	<li>Export the AFEA data output as log files (.txt) for further analysis.
	<ol>
		<li>Once analyses are complete, export the whole project.
		<ol>
			<li>Click &#34;File&#34;, &#34;Export&#34;, &#34;Export Project Results&#34;.</li>
			<li>When a window opens, choose the location of where the exports should be saved and save the log files (.txt) to a folder.</li>
			<li>Convert each participant log life to a data spreadsheet (.csv or .xlsx) to extract relevant data.
			<ol>
				<li>Open data spreadsheet software and select the &#34;Data&#34; tab.</li>
				<li>On the &#34;Data&#34; tab, in the &#34;Get External Data&#34; group, click &#34;From Text&#34;.</li>
				<li>In the &#34;Address bar&#34;, locate, double-click the participant text file to import, and follow the on screen wizard instructions.</li>
				<li>Continue the export process for all relevant participant files.</li>
			</ol>
			</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>
6. Timestamp Participant Videos for Data Analysis
<ol>
	<li>Using the AFEA software, manually review each participant&#8217;s video and identify post-consumption time zero for each sample. Record the timestamp in a data spreadsheet. Post-consumption is defined when the sample cup is below the participant&#8217;s chin and no longer occludes the face. 
	Note: The placement of the timestamp is critical for evaluation. The point where the cup no longer occludes the face is the optimal recommendation and timestamps need to be consistent for all participants.</li>
	<li>Save the timestamp data spreadsheet (.csv) as a reference for extracting relevant data from videos. 
	Note: Participant videos may also be coded internally in the software as &#34;Event Marking&#34;.</li>
</ol>
7. Time Series Emotional Analysis
Note: Consider the &#34;baseline&#34; to be the control (i.e., unflavored water in this example). The researcher has the ability to create a different &#34;baseline treatment stimulus&#34; or a &#34;baseline time without stimulus&#34; for paired comparison dependent on the interests of the investigation. The method proposed accounts for a &#34;default&#34; state by using a paired statistical test. In other words, the procedure uses statistical blocking (i.e., a paired test) to adjust for the default appearance of each participant and therefore reduces the variability across participants.
<ol>
	<li>Extract relevant data from the exported files (.csv or .xlsx).
	<ol>
		<li>Identify a time frame relevant to the study evaluation (seconds).</li>
		<li>Manually extract respective data (time frame) from the exported participant files consulting the participant timestamp (time zero).</li>
		<li>Compile each participant&#39;s treatment data (participant number, treatment, original video time, and emotion response) per emotion (happy, neutral, sad, angry, surprised, scared, and disgusted) for the select time frame (seconds) in a new data spreadsheet for future analysis (Figure 3).</li>
		<li>Continue this process for all participants.</li>
	</ol>
	</li>
	<li>Identify the corresponding time zero from the timestamp file for each participant-treatment pair and adjust video time to a true time &#34;0&#34; for direct comparison (Figure 4, Figure 5). 
	Note: Participant data is collected in a continuous video therefore each treatment &#34;time zero&#34; is different (i.e., unflavored water video time zero is 02:13.5 and unflavored milk video time zero is 03:15.4) in Figure 4. Due to the different treatment &#34;time zeroes&#34;, the video times need to be readjusted and realigned to start at &#34;0:00.0&#34; or other standard start time in order for direct time comparison of treatment emotional response data.</li>
	<li>For each participant, emotion, and adjusted time point, extract the paired treatment (e.g., unflavored milk) and control treatment (e.g., unflavored water) quantitative emotional score. In other words, align a participant&#39;s treatment and control time series of responses for each emotion (Figure 5).</li>
	<li>Compile all participant&#39;s information (participant, adjusted time, and paired treatment (e.g., unflavored water and unflavored milk) at each time point (Figure 6). 
	Note: The steps below demonstrate the steps for a paired Wilcox test by hand. Most data analysis software programs will do this automatically. It is recommended to discuss the statistical analysis process with a statistician.</li>
	<li>Once the samples are reset and aligned with new adjusted video times, directly compare between the emotional results of a respective sample and the control (unflavored water) using sequential paired nonparametric Wilcoxon tests across the participants (Figure 7). 
	Note: The new time alignment of the samples will allow for direct comparison within the 5 seconds post-consumption time frame. If a paired observation is not present in a treatment, drop the participant from that time point comparison.
	<ol>
		<li>Calculate the difference between the control and the respective sample for each paired comparison using data spreadsheet management software. 
		Note: The comparison will be dependent on the frame rate selected for emotional analysis in the software. The protocol demonstrates 30 individual comparisons per second for 5 seconds (selected time frame). 
		Note: Use Figure 7 as a reference for columns and steps.
		<ol>
			<li>Subtract the value of milk (e.g., unflavored milk) from the value of the control (e.g., unflavored water) to determine the difference. In the data spreadsheet management software in a new column titled &#34;Treatment Difference&#34;, enter &#34;=(C2)-(D2)&#34;, where &#34;C2&#34; is the control emotional values and &#34;D2&#34; is the selected treatment emotional values. Continue this process for all time points.</li>
			<li>Calculate the absolute value of the treatment difference. In the data spreadsheet management software in a new column, enter &#34;=ABS(E2)&#34;, where &#34;E2&#34; is the Treatment Difference. Continue this process for all time points.</li>
			<li>Determine the rank order of the treatment difference. In the data spreadsheet management software in a new column, enter &#34;=RANK(G2, $G$2:$G$25, 1)&#34; where &#34;G2&#34; is the Absolute Difference and &#34;1&#34; is &#34;ascending&#34;. Continue this process for all time points.</li>
			<li>Determine the signed rank of the rank order on the spreadsheet. Change the sign to negative if the treatment difference was negative (Column I).</li>
			<li>Calculate the positive sum (=SUMIF(I2:I25, &#34;&#62;0&#34;, I2:I25) and negative sum =SUMIF(I2:I25,&#34;&#60;0&#34;,I2:I25) of the rank values.</li>
			<li>Determine the test statistic. The test statistic is the absolute value lower sum.</li>
			<li>Consult statistical tables for Wilcoxon Signed Ranked Test Statistic using the number of observations included at the specific time and a selected alpha value to determine the critical value.</li>
			<li>If the test statistic is less than the critical value reject the null hypothesis. If it is greater, accept the null hypothesis.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Graph the results on the associated treatment graph (i.e., unflavored milk compared to unflavored water) for the times when the null hypothesis is rejected. Use the sign of the difference to determine which treatment has the greater emotion (Figure 8).
	<ol>
		<li>In the data spreadsheet management software, create a graph using the values of presence or absence of significance.
		<ol>
			<li>Click &#34;Insert&#34; tab.</li>
			<li>Select &#34;Line&#34;</li>
			<li>Right click on the graph box.</li>
			<li>Click &#34;select data&#34; and follow the screen prompts to select and graph relevant data (Figure 8). 
			Note: The graphs will portray emotional results where the sample or control is higher and significant. Graph dependent, the emotion is higher at that specific time allowing the ability to discern how participant&#39;s emotions evolve over the 5 second time period between two samples. 
			Note: Statistical support with a statistician is highly recommended to extract relevant data. Development of statistical coding is required to analyze emotional results.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>

<ol><li>De Wijk, R. A., Kooijman, V., Verhoeven, R. H. G., Holthuysen, N. T. E., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Autonomic+nervous+system+responses+on+and+facial+expressions+to+the+sight%2C+smell%2C+and+taste+of+liked+and+disliked+foods%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">Autonomic nervous system responses on and facial expressions to the sight, smell, and taste of liked and disliked foods.</a> Food Qual Prefer. 26 (2), 196-203 (2012).</li>
<li>De Wijk, R. A., He, W., Mensink, M. G. J., Verhoeven, R. H. G., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((ANS+responses+and+facial+expression+differentiate+between+the+taste+of+commercial+breakfast+drinks%5BTitle%5D)+AND+%22PLoS+ONE%22%5BJournal%5D)">ANS responses and facial expression differentiate between the taste of commercial breakfast drinks.</a> PLoS ONE. 9 (4), 1-9 (2014).</li>
<li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Behavioural+and+physiological+responses+to+two+food+odours%5BTitle%5D)+AND+%22Appetite%22%5BJournal%5D)">Behavioural and physiological responses to two food odours.</a> Appetite. 59 (2), 628(2012).</li>
<li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Dynamics+of+autonomic+nervous+system+responses+and+facial+expressions+to+odors%5BTitle%5D)+AND+%22Front+Psychol%22%5BJournal%5D)">Dynamics of autonomic nervous system responses and facial expressions to odors.</a> Front Psychol. 5 (110), 1-8 (2014).</li>
<li>Danner, L., Sidorkina, L., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Make+a+face%21+Implicit+and+explicit+measurement+of+facial+expressions+elicited+by+orange+juices+using+face+reading+technology%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology.</a> Food Qual Prefer. 32 (2014), 167-172 (2013).</li>
<li>Danner, L., Haindl, S., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Facial+expression+and+autonomous+nervous+system+responses+elicited+by+tasting+different+juices%5BTitle%5D)+AND+%22Food+Res+Int%22%5BJournal%5D)">Facial expression and autonomous nervous system responses elicited by tasting different juices.</a> Food Res Int. 64 (2014), 81-90 (2014).</li>
<li>Arnade, E. A. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=%22Measuring+consumer+emotional+response+to+tastes+and+foods+through+facial+expression+analysis+%5B%5D%22">Measuring consumer emotional response to tastes and foods through facial expression analysis [thesis]</a>. , Virginia Tech. Blacksburg. 1-187 (2013).</li>
<li>Leitch, K. A., Duncan, S. E., O'Keefe, S., Rudd, R., Gallagher, D. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Characterizing+consumer+emotional+response+to+sweeteners+using+an+emotion+terminology+questionnaire+and+facial+expression+analysis%5BTitle%5D)+AND+%22Food+Res+Int%22%5BJournal%5D)">Characterizing consumer emotional response to sweeteners using an emotion terminology questionnaire and facial expression analysis.</a> Food Res Int. 76, 283-292 (2015).</li>
<li>Crist, C. A., et al. Application of emotional facial analysis technology to consumer acceptability using a basic tastes model. Technical Abstracts and Proceedings of the 2014 Institute of Food Technologists Annual Meeting, 2014 Jun 21 - 24, New Orleans, LA, , (2014).</li>
<li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Facial+affective+reactions+to+bitter-tasting+foods+and+body+mass+index+in+adults%5BTitle%5D)+AND+%22Appetite%22%5BJournal%5D)">Facial affective reactions to bitter-tasting foods and body mass index in adults.</a> Appetite. 71 (2013), 178-186 (2013).</li>
<li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Exploring+the+hedonic+and+incentive+properties+in+preferences+for+bitter+foods+via+self-reports%2C+facial+expressions+and+instrumental+behaviours%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">Exploring the hedonic and incentive properties in preferences for bitter foods via self-reports, facial expressions and instrumental behaviours.</a> Food Qual Prefer. 39 (2015), 73-81 (2015).</li>
<li>Lewinski, P., Fransen, M. L., Tan, E. S. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Predicting+advertising+effectiveness+by+facial+expressions+in+response+to+amusing+persuasive+stimuli%5BTitle%5D)+AND+%22J.+Neurosci.+Psychol.+Econ%22%5BJournal%5D)">Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli.</a> J. Neurosci. Psychol. Econ. 7 (1), 1-14 (2014).</li>
<li>Ekman, P., Friesen, W. V. Facial action coding system: A technique for the measurement of facial movement. , Consulting Psychologists Press. Palo Alto, California. (1978).</li>
<li>Viola, P., Jones, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Rapid+object+detection+using+a+boosted+cascade+of+simple+features%5BTitle%5D)+AND+%22Proceedings+of+the+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern%22%5BJournal%5D)">Rapid object detection using a boosted cascade of simple features.</a> Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern. 1, 511-518 (2001).</li>
<li>Sung, K. K., Poggio, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Example-based+learning+for+view-based+human+face+detection%5BTitle%5D)+AND+%22IEEE+Trans.+Pattern+Anal.+Mach.+Intell%22%5BJournal%5D)">Example-based learning for view-based human face detection.</a> IEEE Trans. Pattern Anal. Mach. Intell. 20 (1), 39-51 (1998).</li>
<li><a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=%22FaceReader+5%E2%84%A2+Technical+Specifications%22">FaceReader 5™ Technical Specifications</a>. , Noldus Information Technology. Wageningen, The Netherlands. Available from: 
 
 <a target="_blank" href="http://www.webshop.noldus.com/media/techspec/facereader _techspec.pdf">http://www.webshop.noldus.com/media/techspec/facereader _techspec.pdf</a> (2014).</li>
<li>Cootes, T., Taylor, C. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aCootes%2C+T+%22Statistical+models+of+appearance+for+computer+vision%3A+Technical+report%22">Statistical models of appearance for computer vision: Technical report</a>. , University of Manchester, Wolfson Image Analysis Unit, Imaging Science and Biomedical Engineering. (2000).</li>
<li>Bishop, C. M. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aBishop%2C+CM+%22Neural+networks+for+pattern+recognition%22">Neural networks for pattern recognition</a>. , Oxford University Press. New York. (1995).</li>
<li>Lewinski, P., den Uyl, T. M., Butler, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Automated+facial+coding%3A+validation+of+basic+emotions+and+FACS+AUs+in+FaceReader%5BTitle%5D)+AND+%22J.+Neurosci.+Psychol.+Econ%22%5BJournal%5D)">Automated facial coding: validation of basic emotions and FACS AUs in FaceReader.</a> J. Neurosci. Psychol. Econ. 7 (4), 227-236 (2014).</li>
<li>Noldus Information Technology. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=%22FaceReader+Reference+Manual+Version+6%22">FaceReader Reference Manual Version 6</a>. , Wageningen, The Netherlands. (2014).</li>
<li>Alves, N. T., Fukusima, S. S., Aznar-Casanova, J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Models+of+brain+asymmetry+in+emotional+processing%5BTitle%5D)+AND+%22Psychol+Neurosci%22%5BJournal%5D)">Models of brain asymmetry in emotional processing.</a> Psychol Neurosci. 1 (1), 63-66 (2008).</li>
<li>Costello, M., Clark, S. Preparation of samples for instructing students and staff in dairy products evaluation (Appendix F). <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aCostello%2C+M+%22The+sensory++evaluation+of+dairy+foods%22">The sensory evaluation of dairy foods</a>. Clark, S., Costello, M., Drake, M., Bodyfelt, F. , 2nd, Springer. New York. 551-560 (2009).</li>
<li>Porcherot, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((How+do+you+feel+when+you+smell+this%3F+Optimization+of+a+verbal+measurement+of+odor-elicited+emotions%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">How do you feel when you smell this? Optimization of a verbal measurement of odor-elicited emotions.</a> Food Qual Prefer. 21, 938-947 (2010).</li>
<li>Warrenburg, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Effects+of+fragrance+on+emotions%3A+Moods+and+physiology%5BTitle%5D)+AND+%22Chem.+Senses%22%5BJournal%5D)">Effects of fragrance on emotions: Moods and physiology.</a> Chem. Senses. 30, i248-i249 (2005).</li>
<li>Bredie, W. L. P., Tan, H. S. G., Wendin, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+comparative+study+on+facially+expressed+emotions+in+response+to+basic+tastes%5BTitle%5D)+AND+%22Chem.+Percept%22%5BJournal%5D)">A comparative study on facially expressed emotions in response to basic tastes.</a> Chem. Percept. 7 (1), 1-9 (2014).</li>
<li>Wendin, K., Allesen-Holm, B. H., Bredie, L. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Do+facial+reactions+add+new+dimensions+to+measuring+sensory+responses+to+basic+tastes%3F%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">Do facial reactions add new dimensions to measuring sensory responses to basic tastes?</a> Food Qual Prefer. 22, 346-354 (2011).</li>
<li>Rosenstein, D., Oster, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Differential+facial+responses+to+four+basic+tastes+in+newborns%5BTitle%5D)+AND+%22Child+Dev%22%5BJournal%5D)">Differential facial responses to four basic tastes in newborns.</a> Child Dev. 59 (6), 1555-1568 (1988).</li>
<li>Rosenstein, D., Oster, H. Differential facial responses to four basic tastes in newborns. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aRosenstein%2C+D+%22What+the+face+reveals%3A+Basic+and+applied+studies+of+spontaneous+expression+using+the+facial+action+coding+system+%28FACS%29%22">What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS)</a>. P, E. kman, E, R. osenberg , Oxford University Press, Inc. New York. 302-327 (1997).</li>
<li>Rozin, P., Fallon, A. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+perspective+on+disgust%5BTitle%5D)+AND+%22Psychol.+Rev%22%5BJournal%5D)">A perspective on disgust.</a> Psychol. Rev. 94 (1), 23-41 (1987).</li>
<li>Delarue, J., Blumenthal, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Temporal+aspects+of+consumer+preferences%5BTitle%5D)+AND+%22Curr.+Opin.+Food+Sci%22%5BJournal%5D)">Temporal aspects of consumer preferences.</a> Curr. Opin. Food Sci. 3, 41-46 (2015).</li>
<li>Sudre, J., Pineau, N., Loret, C., Marin, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Comparison+of+methods+to+monitor+liking+of+food+during+consumption%5BTitle%5D)+AND+%22Food+Qual+Prefer%22%5BJournal%5D)">Comparison of methods to monitor liking of food during consumption.</a> Food Qual Prefer. 24 (1), 179-189 (2012).</li>
<li>Weiland, R., Ellgring, H., Macht, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Gustofacial+and+olfactofacial+responses+in+human+adults%5BTitle%5D)+AND+%22Chem.+Senses%22%5BJournal%5D)">Gustofacial and olfactofacial responses in human adults.</a> Chem. Senses. 35 (9), 841-853 (2010).</li>
<li>Ekman, P. Universal and cultural differences in facial expressions of emotion. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aEkman%2C+P+%22Nebraska+symposium+on+motivation%22">Nebraska symposium on motivation</a>. Cole, J. , University of Nebraska Press. Lincoln (NE). 207-283 (1971).</li>
<li>Griemel, E., Macht, M., Krumhuber, E., Ellgring, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Facial+and+affective+reactions+to+tastes+and+their+modulation+by+sadness+and+joy%5BTitle%5D)+AND+%22Physiol+Behav%22%5BJournal%5D)">Facial and affective reactions to tastes and their modulation by sadness and joy.</a> Physiol Behav. 89 (2), 261-269 (2006).</li>
</ol>

The authors have nothing to disclose.

AFEA application in literature related to food and beverage is very limited1-11. The application to food is new, creating an opportunity for establishing methodology and data interpretation. Arnade (2013)7 found high individual variability among individual emotional response to chocolate milk and white milk using area under the curve analysis and analysis of variance. However, even with participant variability, participants generated a happy response longer while sad and disgusted had shorter time r...

Automated facial expression analysis (AFEA) is a prospective analytical tool for characterizing emotional responses to beverages and foods. Emotional analysis can add an extra dimension to existing sensory science methodologies, food evaluation practices, and hedonic scale ratings typically used both in research and industry settings. Emotional analysis could provide an additional metric that reveals a more accurate response to foods and beverages. Hedonic scoring may include participant bias due to failure to record reactions1.
AFEA research has been used in many research applications including computer gaming, user behavior, education/pedagogy, and psychology studies on empathy and deceit. Most food-associated research has focused on characterizing emotional response to food quality and human behavior with food. With the recent trend in gaining insights into food behaviors, a growing body of literature reports use of AFEA for characterizing the human emotional response associated with foods, beverages, and odorants1-12.
AFEA is derived from the Facial Action Coding System (FACS). The facial action coding system (FACS) discriminates facial movements characterized by action units (AUs) on a 5-point intensity scale13. The FACS approach requires trained review experts, manual coding, significant evaluation time, and provides limited data analysis options. AFEA was developed as a rapid evaluation method to determine emotions. AFEA software relies on facial muscular movement, facial databases, and algorithms to characterize the emotional response14-18. The AFEA software used in this study reached a &#34;FACS index of agreement of 0.67 on average on both the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP) and Amsterdam Dynamic Facial Expression Set (ADFES), which is close to a standard agreement of 0.70 for manual coding&#34;19. Universal emotions included in the analysis are happy (positive), sad (negative), disgusted (negative), surprised (positive or negative), angry (negative), scared (negative) and neutral each on a separate scale of 0 to 1 (0=not expressed; 1=fully expressed)20. In addition, psychology literature includes happy, surprised, and angry as &#34;approach&#34; emotions (toward stimuli) and sad, scared, and disgusted as &#34;withdrawal&#34; emotions (away from aversive stimuli)21.
One limitation of the current AFEA software for characterizing emotions associated with foods is interference from facial movements associated with chewing and swallowing as well as other gross motor motions, such as extreme head movements. The software targets smaller facial muscular motions, relating position and degree of movement, based on over 500 muscle points on the face16,17. Chewing motions interfere with classification of expressions. This limitation may be addressed using liquefied foods. However, other methodology challenges can also decrease video sensitivity and AFEA analysis including data collection environment, technology, researcher instructions, participant behavior, and participant attributes.
A standard methodology has not been developed and verified for optimal video capture and data analysis using AFEA for emotional response to foods and beverages in a sensory evaluation laboratory setting. Many aspects can affect the video capture environment including lighting, shadowing due to lighting, participant directions, participant behavior, participant height, as well as, camera height, camera angling, and equipment settings. Moreover, data analysis methodologies are inconsistent and lack a standard methodology for assessing emotional response. Here, we will demonstrate our standard operating procedure for capturing emotional data and processing data into meaningful results using beverages (flavored milk, unflavored milk and unflavored water) for evaluation. To our knowledge only one peer reviewed publication, from our lab group, has utilized time series for data interpretation for emotions analysis8; however, the method has been updated for our presented method. Our aim is to develop an improved and consistent methodology to help with reproducibility in a sensory evaluation laboratory setting. For demonstration, the objective of the study model is to evaluate if AFEA could supplement traditional hedonic acceptability assessment of flavored milk, unflavored milk and unflavored water. The intention of this video protocol is to help establish AFEA methodology, standardize video capture criteria in a sensory evaluation laboratory (sensory booth setting), and illustrate a method for temporal emotional data analysis of a population.

<table><tbody><tr><td>2% Reduced Fat Milk</td><td>Kroger Brand, Cincinnati, OH or DZA Brands, LLC, Salisbury, NC</td><td>na</td><td>for solutions</td></tr><tr><td>Drinking Water</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>Imitation Clear Vanilla Flavor</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>Iodized Salt</td><td>Kroger Brand, Cincinnati, OH</td><td>na</td><td>for solutions</td></tr><tr><td>FaceReader 6</td><td>Noldus Information Technology, Wageningen, The Netherlands</td><td>na</td><td>For Facial Analysis</td></tr><tr><td>Sensory Information Management System (SIMS) 2000</td><td>Sensory Computer Systems, Berkeley Heights, NJ</td><td>Version 6</td><td>For Sensory Data Capture</td></tr><tr><td>Rhapsody</td><td>Acuity Brands Lighting, Inc., Conyers, GA</td><td></td><td>For Environment Illumination</td></tr><tr><td>R Version&amp;nbsp;</td><td>R Core Team 2015</td><td>3.1.1</td><td>For Statistical Analysis</td></tr><tr><td>Microsoft Office</td><td>Microsoft</td><td>na</td><td>For Statistical Analysis</td></tr><tr><td>JMP</td><td>Statistical Analysis Software (SAS) Version 9.2, SAS Institute, Cary, NC</td><td>na</td><td>For Statistical Analysis</td></tr><tr><td>Media Recorder 2.5</td><td>Noldus Information Technology, Wageningen, The Netherlands</td><td>na</td><td>For capturing participants sensory evaluation</td></tr><tr><td>Axis M1054 Camera</td><td>Axis Communications, Lund, Sweden</td><td>na</td><td></td></tr><tr><td>Beverage</td><td></td><td>na</td><td>Beverage or soft food for evaluation</td></tr></tbody></table>

protocol for data collection and analysis applied to automated facial expression analysis technology and temporal analysis for sensory evaluation

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis (AFEA) software. Additionally, we demonstrate a method for extracting relevant emotional data output and plotting the emotional response of a population over a specified time frame. By time pairing each participant's treatment response to a control stimulus (baseline), the overall emotional response over time and across multiple participants can be quantified. AFEA is a prospective analytical tool for assessing unbiased response to food and beverages. At present, most research has mainly focused on beverages. Methodologies and analyses have not yet been standardized for the application of AFEA to beverages and foods; however, a consistent standard methodology is needed. Optimizing video capture procedures and resulting video quality aids in a successful collection of emotional response to foods. Furthermore, the methodology of data analysis is novel for extracting the pertinent data relevant to the emotional response. The combinations of video capture optimization and data analysis will aid in standardizing the protocol for automated facial expression analysis and interpretation of emotional response data.

The method proposes a standard protocol for AFEA data collection. If suggested protocol steps are followed, unusable emotional data output (Figure 1) resulting from poor data collection (Figure 2: A; Left Picture) may be limited. Time series analysis cannot be utilized if log files (.txt) predominantly contain "FIT_FAILED" and "FIND_FAILED" as this is bad data (Figure 1). Furthermore, the method includes a protocol for direct statistical ...

A protocol for capturing and statistically analyzing emotional response of a population to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis software is described.

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial ...

The overall goal of this methodology is to capture and analyze emotional responses to beverages and liquefied foods in a sensory evaluation laboratory using automated facial analysis software. This method can help answer key questions in the sensory science field such as how to measure a participant's emotional response to beverages and soft foods. The main advantage of this protocol is that it shows us standardized data capture methodology using automated facial expression analysis software.It also describes the method for analyzing automated facial expression output. The implications of this technique extend towards standardizing methodology of automated facial expression analysis as it often varies between laboratories. Although this method can provide insight to an individual's implicit emotional response, it can also be applied to other systems such as packaging, marketing, and labels.Lester will serve as a representative participant. In preparation for the experiment, create sample solutions by pouring half ounce aliquots of each original solution into two ounce transparent plastic sample cups, and then cap with color-coded lids. Next, escort the participant to the sensory booth and have them sit straight up against the back of the chair.Instruct the participant about the experimental process and standard operating procedures. Adjust the position of the chair and the camera angle so that the participants face is captured in the center of the video recording with no shadows on the chin or around the eyes. Instruct the participant to focus on the monitor display, follow the directions on the screen, and refrain from any sudden movements for 30 seconds after each sample is consumed.Demonstrate an impromptu practice session by first instructing the participant to hold up the associated identification card. Then instruct them to open the sample container, consume the entire beverage sample, and swallow. After consumption, instruct the participant to quickly move the sample cup away from the chin and place it on the table to avoid facial occlusion.Remind them to keep looking toward the monitor. Remind the participant that water will be provided on the serving tray to cleanse their palate in between samples when prompted by the monitor. Finally, instruct the participant to follow the instructions as they appear on the touch screen monitor.Confirm that the video camera is directed toward the participant's face by viewing the computer monitor on which the video capture is displayed. Then begin recording by clicking the record button on the computer monitor. Prompt the participant, via the touch screen monitor, to sip water to cleanse their palate.Place a unique-colored index card on top of each sample, allowing for identification of each treatment sample. Next, the touch screen monitor prompts the participant to hold up the associated index card pre-consumption for sample identification in the video, and then have them place the card back on the tray. Provide only one sample at a time on the serving tray, starting with a baseline sample of unflavored water.Have the participant consume the sample while facing the camera, and then wait approximately 30 seconds, a time enforced by the touch screen monitor. After 30 seconds, prompt the participant, via the touch screen monitor, to enter their hedonic acceptability score. Finally, have the participant rinse their mouth with drinking water before moving on to the next sample.Use facial analysis software to analyze the experimental videos. Add participants to the project by clicking the add participants icon. Press the batch analysis icon, which is the red and black target-like symbol, to analyze the project videos.Export the AFEA data output as log files for further analysis. Using the AFEA software, manually review each video and identify post-consumption time zero for each sample, which is defined as when the sample cup is below the participant's chin and longer occludes the face. Finally, record the timestamp in the data spreadsheet, and save this spreadsheet as a reference for extracting relevant data from videos.Unflavored milk, unflavored water, and vanilla extract flavor in milk were not different in mean acceptability scores. And salty flavor in milk had a lower mean hedonic acceptability score. However, AFEA time series analysis indicated unflavored milk generated less disgusted, surprised, sad, and neutral responses than did unflavored water.Furthermore, the vanilla extract flavor in milk introduced more happy and less sad expressions when compared to unflavored water. Salty flavor in milk generated more disgusted and less neutral responses when compared to unflavored water. Once mastered, this technique can be done in 30 minutes if it is performed properly.While attempting this procedure, it's important to remember to give the participant complete instructions and expectations. Following this procedure, other evaluations can be performed in order to answer additional questions like what is the influence on behavior and emotional response? After its development, this technique paved the way for researchers in the field of sensory science to explore implicit emotional responses in human participants to beverages and soft foods.After watching this video, you should have a good understanding of how to standardize the sensory lab environment, and how to instruct participants in order to capture the best data and results, as well as analyze automated facial expression analysis output. Don't forget that working with human participants can be sensitive. It is important to discuss ingredients prior to performing this procedure.

protocol-for-data-collection-analysis-applied-to-automated-facial

Research

JoVE Journal

Behavior

Source:Hila Man1, Paul F. Funk2, Ieva Vebraite-Adereth3, Christoph Anders4, Orlando Guntinas-Lichius2, Yael Hanein1,3,51Sagol School of Neuroscience, Tel Aviv University, 2Department of Otorhinolaryngology, Jena University Hospital, Friedrich-Schiller-University Jena, 3School of Electrical Engineering, Tel Aviv University, 4Division Motor Research, Pathophysiology and Biomechanics, Department of Trauma, Hand and Reconstructive Surgery, Jena University Hospital, Friedrich-Schiller-University...

Wireless Surface Electromyography Recording of Facial Muscle Activity During Expressions

This paper aims to describe the techniques involved in the collection and synchronization of the multiple dimensions (behavioral, affective and cognitive) of learners’ engagement during a task.

Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective

An EEG experimental protocol is designed to clarify the interplay between conscious and non-conscious representations of emotional faces in patients with Asperger's syndrome. The technique suggests that patients with Asperger's syndrome have deficits in non-conscious representation of emotional faces, but have comparable performance in conscious representation with healthy controls.

Conscious and Non-conscious Representations of Emotional Faces in Asperger's Syndrome

We describe a protocol to assess facial muscle activity in response to experienced and observed tactile stimulation using facial electromyography.

Using Facial Electromyography to Assess Facial Muscle Reactions to Experienced and Observed Affective Touch in Humans

This protocol offers a digitization of portions of traditional clinical tasks commonly used to measure cognition and motor control in Parkinson’s disease. Clinical tasks are digitized while biophysical rhythms are co-registered from different functional levels of the nervous systems, ranging from voluntary, spontaneous, automatic to autonomic.

Dynamic Digital Biomarkers of Motor and Cognitive Function in Parkinson's Disease

The present protocol combines functional near-infrared spectroscopy (fNIRS) and video-based observationto measure interpersonal synchronization in quartets during a collaborative drawing task.

Group Synchronization During Collaborative Drawing Using Functional Near-Infrared Spectroscopy

This article presents a protocol for a contextual and cued fear conditioning test using a video analyzing system to assess fear learning and memory in mice.

Contextual and Cued Fear Conditioning Test Using a Video Analyzing System in Mice

This paper describes how to use the emotional oddball task and fMRI to measure brain activation in children and adolescents at familial high risk for schizophrenia (FHR). FMRI was used to measure differences in fronto-striato-limbic regions during an emotional oddball task. Children with FHR exhibited abnormal functional activation during adolescence.

Measurement of Fronto-limbic Activity Using an Emotional Oddball Task in Children with Familial High Risk for Schizophrenia

Mouse ultrasonic vocalizations are used as proxies to model the genetic bases of vocal communication deficits in mouse models for neuropsychiatric disorders. The present protocol describes three experimental contexts that reliably elicit ultrasonic vocalizations from pups (throughout development) and adult mice (same-sex interactions, male-estrus female interactions).

Recording Mouse Ultrasonic Vocalizations to Evaluate Social Communication

Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

Learn standardized data capture and analysis for automated facial expression analysis in sensory evaluation. This protocol measures emotional responses to beverages and liquefied foods using AFEA software.

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation (Video) | JoVE Journal

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

Explore More Videos