Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

The overall goal of this methodology is to capture and analyze emotional responses to beverages and liquefied foods in a sensory evaluation laboratory using automated facial analysis software. This method can help answer key questions in the sensory science field such as how to measure a participant's emotional response to beverages and soft foods. The main advantage of this protocol is that it shows us standardized data capture methodology using automated facial expression analysis software.

It also describes the method for analyzing automated facial expression output. The implications of this technique extend towards standardizing methodology of automated facial expression analysis as it often varies between laboratories. Although this method can provide insight to an individual's implicit emotional response, it can also be applied to other systems such as packaging, marketing, and labels.

Lester will serve as a representative participant. In preparation for the experiment, create sample solutions by pouring half ounce aliquots of each original solution into two ounce transparent plastic sample cups, and then cap with color-coded lids. Next, escort the participant to the sensory booth and have them sit straight up against the back of the chair.

Instruct the participant about the experimental process and standard operating procedures. Adjust the position of the chair and the camera angle so that the participants face is captured in the center of the video recording with no shadows on the chin or around the eyes. Instruct the participant to focus on the monitor display, follow the directions on the screen, and refrain from any sudden movements for 30 seconds after each sample is consumed.

Demonstrate an impromptu practice session by first instructing the participant to hold up the associated identification card. Then instruct them to open the sample container, consume the entire beverage sample, and swallow. After consumption, instruct the participant to quickly move the sample cup away from the chin and place it on the table to avoid facial occlusion.

Remind them to keep looking toward the monitor. Remind the participant that water will be provided on the serving tray to cleanse their palate in between samples when prompted by the monitor. Finally, instruct the participant to follow the instructions as they appear on the touch screen monitor.

Confirm that the video camera is directed toward the participant's face by viewing the computer monitor on which the video capture is displayed. Then begin recording by clicking the record button on the computer monitor. Prompt the participant, via the touch screen monitor, to sip water to cleanse their palate.

Place a unique-colored index card on top of each sample, allowing for identification of each treatment sample. Next, the touch screen monitor prompts the participant to hold up the associated index card pre-consumption for sample identification in the video, and then have them place the card back on the tray. Provide only one sample at a time on the serving tray, starting with a baseline sample of unflavored water.

Have the participant consume the sample while facing the camera, and then wait approximately 30 seconds, a time enforced by the touch screen monitor. After 30 seconds, prompt the participant, via the touch screen monitor, to enter their hedonic acceptability score. Finally, have the participant rinse their mouth with drinking water before moving on to the next sample.

Use facial analysis software to analyze the experimental videos. Add participants to the project by clicking the add participants icon. Press the batch analysis icon, which is the red and black target-like symbol, to analyze the project videos.

Export the AFEA data output as log files for further analysis. Using the AFEA software, manually review each video and identify post-consumption time zero for each sample, which is defined as when the sample cup is below the participant's chin and longer occludes the face. Finally, record the timestamp in the data spreadsheet, and save this spreadsheet as a reference for extracting relevant data from videos.

Unflavored milk, unflavored water, and vanilla extract flavor in milk were not different in mean acceptability scores. And salty flavor in milk had a lower mean hedonic acceptability score. However, AFEA time series analysis indicated unflavored milk generated less disgusted, surprised, sad, and neutral responses than did unflavored water.

Furthermore, the vanilla extract flavor in milk introduced more happy and less sad expressions when compared to unflavored water. Salty flavor in milk generated more disgusted and less neutral responses when compared to unflavored water. Once mastered, this technique can be done in 30 minutes if it is performed properly.

While attempting this procedure, it's important to remember to give the participant complete instructions and expectations. Following this procedure, other evaluations can be performed in order to answer additional questions like what is the influence on behavior and emotional response? After its development, this technique paved the way for researchers in the field of sensory science to explore implicit emotional responses in human participants to beverages and soft foods.

After watching this video, you should have a good understanding of how to standardize the sensory lab environment, and how to instruct participants in order to capture the best data and results, as well as analyze automated facial expression analysis output. Don't forget that working with human participants can be sensitive. It is important to discuss ingredients prior to performing this procedure.