Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content

Summary

We present detection, acquisition, and analysis of eyeblink rates while watching media content.

Abstract

This article explores a method to detect differences in visual perception in humans. The method used is based on the psychological (or "cognitive") function of eyeblinks. Participants' eyeblinks are detected and acquired while watching videos specifically created for the investigation. The detection and acquisition of eyeblinks are carried out with the help of a 20-channel electroencephalographic (EEG) wireless device. The international 10-20 system for electrode placement is followed. A high-definition (HD) video camera is used to record participants' facial expressions, for contrast purposes. Instead of using pre-existing media content, purpose-made video content has been created following specific criteria of interest for this investigation, with stimuli enabling researchers to manage the precise parameters of interest. Otherwise, results could be contaminated with uncontrolled variables. The synchronization of the presentation of video stimuli with EEG recordings needs to be done in milliseconds. Analysis of collected data is performed with robust software for working with big matrices. Statistically significant differences in eyeblink rate related to media professionalization and editing style are found with the reported experimental procedures.

Introduction

The Purpose of This Method

This method proposes a dual protocol for detecting eyeblinks. The aim is to analyze viewers' visual perception of media content, specifically created for this investigation, by using EEG recordings and HD video recording systems.

The Rationale Behind the Development and/or Use of This Method

Each eyeblink hides visual flow for 150-400 ms^1,2. Blinking has physiological^3,4,5 and psychological^6,7 functions. The connection between attention and eyeblink rate has been studied and proven in different studies⁸. A higher level of attention decreases eyeblink rate and according to previous studies, humans share a mechanism for controlling the timing of blinks that searches for the best moment to avoid loss of important visual information⁹. Thus, analyzing eyeblink behavior of viewers when watching screens could provide information regarding the level of attention given to media contents.

One method for detecting spontaneous eyeblink rate is by using EEG electrodes to record the electrical activity. Eyeblinks can easily be detected by prefrontal and electrooculogram electrodes connected to an EEG recording system. In most EEG analyses, eyeblinks are considered artifacts. For this reason, many software packages designed for analyzing EEG data have eyeblink detectors¹⁰. The advantage of using EEG for detecting eyeblinks is the high temporal resolution (in the order of milliseconds) and the possibility of registering brain effects of different narratives and cuts in movies synchronized with those eyeblinks - a matter open to further study. Recording participants' faces with an HD camera can also be useful for matching/contrast purposes⁹.

The Advantages over Alternative Methods with References to Relevant Studies

There are multiple methods for counting eye blinks. Some dedicated instruments for detecting blinks are magnetic coils, infrared (IR) light beams, optoelectronic motion detectors with eye movement analysis such as eye-tracking techniques, and several techniques based on bioelectrical signals, e.g., electrooculography (EOG), electromyography (EMG), and EEG. Another more accurate, but time-consuming option is manually counting blinks from a frame-by-frame video recording¹¹. The technologies today can be classified broadly into two groups: a) contact-free recording which includes two modalities, the direct blink detection using computer vision and offline blink detection using eye-tracking, and b) contact-based recording using biological signals through EOG and EEG devices^12,13.

The eye-tracking system is a widely used technology, ranging from traditional image-based passive designs to the active near-infrared-based approaches mainly used today with a high-resolution camera. The latter exploits the reflective properties of the pupil under IR illumination¹⁴. The concept underlying modern eye-tracking methods is Pupil Center Corneal Reflection (PCCR), which involves a camera tracking the center of the pupil, where light reflects from the cornea. However, there is a lack of blink detection algorithms published for eye-tracking protocols. Moreover, although the different models of eye-tracking on the market provide integrated software with blink detection, the source code is not always provided by manufacturers, making it difficult to modify or know how the algorithms work¹². Also, during experiments with eye-tracking there are events that cause data loss, such as tracking delays and significant head or gaze movements. The eye area is very small in video captures, which is a problem for calculating the duration of the blink, and which sometimes introduces various types of artifacts¹⁵.

In this experiment, EEG and EOG methods are used. EEG is not usually used alone to detect eyeblinks. However, analyzing eyeblinks recorded with EEG electrodes is a standard procedure for the study of eyelid displacements. This procedure enables researchers to have information of exactly when eyeblinks take place. The most common signal pattern for detecting blinks is that of peak points, representing vertical movement responses. There are several peak detection algorithms applicable to raw EEG, time-domain, or frequency-domain signals. Processes involved in peak identification are peak detection, feature extraction, and classification. Eyeblinks have a considerable effect on frontal channels of the EEG signal. Typically, eyeblinks are detected in EEG by using a pre-determined amplitude threshold¹⁶. The algorithms in the analysis software used in this experiment are based on the signals' standard deviation and the root mean square (RMS) of the pre-filtered EEG signal; they are open source and available to the scientific community¹⁷. However, some eye movements not involving eyeblinks can provoke electrical activity that may be confusing. For that reason, a second method - recording viewers' faces with an HD video camera - allows researchers to match eyeblinks by manually counting them. With such a double method, the investigator achieves a matrix of eyeblinks that can be easily analyzed with statistical tools.

Therefore, the proposed method performs a data triangulation with two different sources to validate the detected eyeblinks. This method is based on Nakano et al. indications⁹ for confirmation. At the same time, it also enables researchers to collect brain-activity and frequency-band information for further analysis. The experiment described here is part of a wider future investigation into the effects of editing-style cuts on occipital and prefrontal brain areas.

Determine Whether the Method is Appropriate for an Investigation

This experimental protocol enables viewers' eyeblinks while watching video content to be studied under three experimental conditions. First, eyeblink rate is detected by using two complementary techniques: EEG and recorded HD videos. Here, we use a wireless EEG with 20 channels. Second, specific stimuli adapted to the experiment are created, so that the researcher can manage all the variables of the visual content. Here, three videos with the same narrative but different video-editing style were created. The narrative consisted of a man who entered a room, sat at a desk, juggled with three balls, opened a laptop, looked up information in some books, typed something on the laptop, closed it, ate an apple, looked directly into camera, and left the room. The three video stimuli last 198 s each. The first was a one-shot movie; the second was edited according to classical Hollywood-style rules with 33 different shots; and the third was edited following MTV-style rules with 79 shots. A fourth stimulus was also presented in which the narrative was identical, but the format was a real representation with an actor instead of a video. This fourth non-video stimulus was not used in an initial study of editing-style differences but was used in a different investigation to compare eyeblink-rate differences between real representation and screened media⁸. Third, different groups of participants are selected depending on their previous expertise in the visual analysis of videos. The purpose is to determine differences in eyeblink rates of subject groups watching the same visual stimuli. In this case, 40 subjects took part in the investigation. Half of them were media professionals (16 males and 4 females; age 30-56 years, with an average age of 44.15 ±7.15 years) and the rest were non-media professionals (15 males, and 5 females; age 28-56 years, with an average age of 43.25 ± 8.59 years). Media professionals were chosen with the criterion of more than 6 years of experience in making decisions related to media editing in their everyday work.

Protocol

All methods described here have been approved by the Ethics Commission for Research with Animals and Humans (CEEAH) of the Universitat Autònoma de Barcelona.

1. Creation and Presentation of the Visual Stimuli

1. Create the video stimuli according to the desired goals.
   1. Decide which variables are of interest for the video stimuli.
      NOTE: For example, in the present investigation, the main variable of interest is the editing style.
      1. Decide the duration, the content, and the style of the video based on the interest of the experiment. To determine all of these variables, consider how many videos the participant is going to watch and how long the participant will be watching the screen, evaluate if the narrative of the video meets the interest, and decide the visual style of the video.
   2. Write the script, paying attention to the chosen variables.
   3. Record the video with a high-quality video camera, and edit the video as needed. Seek professional help if necessary.
2. Design an appropriate context for the stimuli presentation.
   1. Decide where the data acquisition will take place and design a comfortable space to display the stimuli. Participants should feel at ease.
   2. Select a high-quality screen for the presentation of the video. The higher the quality of the screen, the better.
   3. If there are several video films to be presented, select a method for the randomization of their presentation.
      NOTE: It is convenient to randomize stimuli presentation across the subjects to avoid inconclusive results due to the order in which the participants perceive a stimulus.
   4. Choose a software package to present the videos.
      NOTE: There are various options that allow the researcher to display different stimuli with inter-trial timers and instructions, or even a randomized presentation. When choosing the software, it is critical to consider its capability for synchronization with data acquisition devices. Since this protocol analyzes participants' eyeblink rate, the synchronization between the visual presentation of the visual stimuli and the EEG device is crucial.

2. Selection of Participants

1. Establish a criterion for choosing participants. When a specific variable is going to be analyzed, a control group is needed. Write down the criterion and consult with colleagues to improve upon the study design.
2. Select participants and seek approval from an ethics committee that follows relevant guidelines and regulations.
   NOTE: Most universities and research centers have an ethics committee. To apply for approval, it will be necessary to inform them of the investigation goals. Common requirements include explaining the research methods, the benefits of the investigation to society, how informed consent will be obtained, whether participants will be recompensed financially, and how data collected from participants will be managed. Take time to design this request. It is very important to get the approval of an ethics committee to share the results with the scientific community. Some documents that can be useful in this respect are the Declaration of Helsinki, the Nuremberg Code, and the Belmont Report.
3. Design an informed consent form for participants, and follow the criteria of the ethics committee.
4. Explain the experiment to every potential participant. Let the participant know all the important information, but avoid information that is not important or can affect the results of the experiment.
   NOTE: For example, in an eyeblink detection protocol, the participant cannot be told that eyeblinks are going to be detected, as this could modify the natural eyeblink rate. Ask for their approval to participate. Give each participant the document for informed consent and allow them enough time to read it, to understand the text, and, finally, to sign it.
5. Make an appointment with each participant to conduct the experiment. Give the participant the following instructions: to arrive rested and with recently washed hair; chemical products such as hairspray, etc., should be avoided because they can affect EEG recordings. Give the participant all the necessary contact information.

3. Set-up Preparation

1. Prepare the experimental session well in advance of the participant's arrival. Rehearse the session with a colleague. In case something incorrect is detected, make the corresponding modifications to the experimental session.
   1. Prepare connections between all devices that are going to be used and check their synchronization (visual stimuli presentation and data acquisition).
      NOTE: The connectivity between hardware and software is critical. It is desirable to perform tests to confirm synchronization. Discuss with the manufacturer or software developer, if needed.
2. Make the subjects feel comfortable in the experimental room: welcome the participant, and prepare a space for their things (bags and others).
3. Explain the entire protocol to the participant: where the participant will be seated, how many visual stimuli are going to be shown, and the length of the session. Explain that the participant's cooperation is extremely important.
4. Since a wireless EEG recording system is going to be used, explain how difficult these recordings are, and that in order to obtain proper electrophysiological signals with low artifacts, the participant should not move unnecessarily.

4. Data Acquisition

1. Clean the surface of the skin of the participant to remove any traces of dirt or remnants of hair products.
2. Measure the head of the participant. Measure the distance between the nasion and the inion, and mark the half-way point as a vertex.
3. Prepare the electrodes in the EEG cap using the 10-20 system.
4. Put the cap on the subject following the mark so that the vertex matches the Central zero (Cz) point of the cap.
5. Apply conductive gel with the help of a syringe to the electrodes; make sure the cap fits well on the subject's head.
6. Place the reference electrode and the EOG electrode on the subject's head.
7. Make sure that the participant is comfortably seated and prepared for the session.
8. Focus an HD video camera on the participant's face in close-up. This will be used to manually detect the participants' eyeblinks.
9. Check that all devices are synchronized.
10. Check that the EEG control box is properly connected to the computer registering the data.
11. Check that the signal for each electrode is correct; if not, check for the problematic electrode.
    NOTE: Some common reasons for a bad signal from an electrode are a lack of conductive gel on the electrode, a bad position of the electrode on the head, or a bad connection of the electrode cable with its control box.
12. When everything is ready, start recording the experiment (and playing the video).

5. Post-experimental Session

1. Remove the cap from the participant's head and, if possible, give the participant a questionnaire related to the topics of interest.
2. Save all the data to an external hard drive.
   NOTE: It is recommendable to back up the data. A good option would be to save the data to two external hard drives and store them in different places. In designing the best procedure, it is important to consider the protocol for data storage approved by the ethics committee.
3. Clean the electrodes and cap; most electrodes need to be cleaned immediately after use.
   1. Clean the electrodes and cap with abundant water under a faucet and without chemical product (unless stated otherwise by the device manufacturer). Carefully dry the electrodes to prevent corrosion. If this protocol is insufficient to clean the cap, it can usually be washed in a washing machine (see the manufacturer's instructions).

6. Data Analysis

1. Organize the eyeblink detection data.
   1. Import the EEG file into a suitable software package for data analysis.
   2. Select the EEG area of interest based upon the synchronization triggers used with the visual stimuli presentation software. Filter the data and apply an eyeblink filter to see how many and when eyeblinks are generated.
      NOTE: Most commercial and free software packages have filters to detect eyeblinks in EEG.
   3. Select the HD video of participants' faces and manually match eyeblinks with those found in the EEG data.
   4. Create a spreadsheet matrix with the final list of each participant's eyeblinks, and compute the rate per minute while the subject was watching the stimuli.
2. Perform statistical analysis.
   1. Import the matrix into the corresponding software for statistical analysis.
   2. In a statistical software program, design the desired analysis to contrast the initial hypotheses and complete them.

Representative Results

Using the procedure presented here, the eyeblink rate of 40 participants while watching three different video movies was analyzed. For a comparative analysis on media professionalization, half of the participants were media professionals. They were selected based on the criterion of a minimum 6 years of job experience in making decisions related to media editing and audio-visual cuts. For a comparative analysis on editing styles, the three video stimuli were created with exactly the same narrative but a different editing style. One was a one-shot movie with one open shot and no camera movements. Another was a Hollywood-style movie with 33 shots and an average shot length of 5.9 s, as well as continuity throughout the shots, with relationships of content, rhythm, space, and time, in accordance with classical Hollywood editing-style rules. Finally, the third video was an MTV-style movie with 79 shots and an average shot length of 2.4 s. In the latter case, there was discontinuity of content, rhythm, space, and time between the shots.

According to the collected results (Figure 1), the editing style affected viewers' eyeblink rate (X²(2) = 7.2, p= 0.027, Friedman Test). With shorter lengths of the shots and more unexpected cuts, there was a lower spontaneous eyeblink rate in viewers.

The investigation also showed that media professionalization affects eyeblink rate. We found statistically significant differences between the two groups (Mann-Whitney U = 86, n = 20, p= 0.002, Mann-Whitney Rank Sum Test) when comparing all stimuli together. Non-media professionals have a higher eyeblink rate than media professionals. This difference is seen for every editing style analyzed: in the one-shot movie (Mann-Whitney U = 86.5, n = 20,p= 0.002, Mann-Whitney Rank Sum Test); in the Hollywood-style movie (Mann-Whitney U = 90, n = 20,p= 0.003, Mann-Whitney Rank Sum Test); and in the MTV-editing-style movie (Mann-Whitney U = 94.5, n = 20,p= 0.004, Mann-Whitney Rank Sum Test). Media professionals show greater homogeneity in eyeblink rate than non-media professionals, as the standard deviation of their eyeblink rate is much lower, regardless of the editing style analyzed.

Figure 1. Eyeblink rate in different editing styles and in different groups. Spontaneous blink rate (SBR) per min (min^-1) in viewers while watching different editing styles. The ends of the boxes define the 25^th and 75^th percentiles, with a line at the median and error bars defining the 10^th and 90^th percentiles; unfilled circles represent data points. Viewers' eyeblink rate decreases as the editing of the media content becomes more chaotic. The MTV-editing style provokes a lower eyeblink rate than the Hollywood-editing style or the one-shot movie. Media professionals have a lower eyeblink rate than non-media professionals. This happens for every style of editing. These data were taken and modified from reference ³. Please click here to view a larger version of this figure.

Discussion

A method for analyzing visual perception of media content with purpose-made video creation is described here. Many other studies attempt to analyze the perception of media content in narrative contexts with pre-existing films. The present method proposes to create visual content with a narrative construction following the criteria of interest, and is based on the suggestion that eyeblink rate is connected to the viewer's attention⁹. For that reason, the study detects participants' eyeblinks while watching media content and compares their rate under different conditions (the editing style and the media professionalization).

The idea that eyeblink rate is connected to cognitive tasks has been developed over several studies during recent decades^18,19,20. Eyeblink is also useful as an attentional marker²¹. A decrease of eyeblink rate has been linked with an increase of attention^9,22,23. In this investigation, by using EEG measurements and high-quality video recording for analyzing visual perception of media content, we found that the style of the editing can be used by creators to handle viewers' attention³. This had already been proposed by the cinema editor Walter Murch, who, while editing films, suspected that eyeblinks may have a comprehension and punctuation marker function in films, related to audio-visual cuts²⁴. This should be of interest in audio-visual production contexts. The fact that the editing style is an element that can affect viewers' eyeblink rate complements a previous work, which detected that the narrative of a media content affected viewers' eyeblink rate⁹. A higher need of attention is created by the MTV-editing style, where cuts occur more often than in the other two styles of editing, and provokes a decrease of eyeblink rate. This is in accord with previous works studying eyeblink inhibition and operational memory²⁵ and in which blink inhibition has been proposed as a mechanism to avoid the loss of information by a task with an increased level of demand⁷. It is worthwhile to state that in other non-audio-visual platforms, such as paper, the presentation format does not affect the frequency and placement of blinks²⁶.

Previous investigations had found that professionalization is a variable that gives rise to perceptive differences, even cerebral changes, in, e.g., athletes²⁷, architects²⁸, musicians²⁹, or ballet dancers³⁰. Here, we found that media professionalization results in decreasing eyeblink rate while video content is watched, regardless of the editing style⁸. Since eyeblink also has the physiological function of wetting the eye, the decrease of eyeblink rate in media professionals may be of interest for hazard control and prevention in this professional group.

Critical Steps Within the Protocol

The proposed methodology includes the following critical steps: (i) The selection of the participants since an insufficient description of the criteria used can lead to misunderstandings in possible replications of the experiment. To avoid such problems, it is desirable to explicitly write down in detail which criteria are used in the experiment. (ii) The design and creation of the video stimuli. Traditionally, the analysis of media content has been done with pre-existing narratives (films, advertisements, or TV shows). However, the proposed methodology is based on the preparation of the media content to be analyzed. Since creating professional videos is not easy, researchers may need the help of a professional audiovisual producer. Therefore, this is a critical step within the protocol, where the researcher may need to work with professionals outside the laboratory. It is important to include the variables of interest in the design of the media content. (iii) The synchronization between stimulus presentation and data acquisition. Depending on the hardware and software used by the researcher (such as the software for presenting the stimuli and that used for recording the EEG), the synchronization between them may be confusing. Each manufacturer has their own protocols to manage inputs and outputs for connectivity. In case of doubt, the researcher may need to speak with them.

Significance of the Technique with Respect to Existing/Alternative Methods

Use of the proposed double approach for analyzing eyeblinks helps researchers to contrast the results. While the EEG signal can classify eye movements such as eyeblink, by contrasting results with a recorded HD video of the participants' faces it is possible to avoid errors in the results. This approach also collects electrical brain activity data for further analysis.

Limitations of the Technique

Different groups of subjects may present significant differences in their eyeblink rates. A limitation of this technique is that, despite obtaining detailed questionnaires from the participants, important information can be missed. For that reason, it is necessary to carefully establish the criteria for selecting subjects with regard to how their professional or social situations can affect their eyeblink rates. This limitation could cause erroneous conclusions, such as misinterpretation of eyeblink patterns linked to subject groups.

Materials

Name	Company	Catalog Number	Comments
EEG Device	Neurolectrics	Enobio 20 EEG/EMG system
Ag/AgCl Electrodes	Neuroelectrics	[NE022b] GelTrode
Recording EEG software	Neuroelectrics	NicOffline software
HD-video camera	Sony Corporation	Sony HDR-GW55VE
Syringe	Monoject	Monoject 412, curved tip syringe, 50/box
Saline electrode EMG gel	Signa-Gel	X32-204: Signa Gel
Visual Stimuli Presentation Software	Paradigm Stimulus Presentation	Perception Research System Incorporated
EEG software analysis	Centre National de la Recherche Scientifique and Montreal Neurological Institute	Brainstorm3
EEG software analysis	The MathWorks Inc.	MATLAB 2013b
TV for video presentation	Panasonic Corporation	PanasonicTH- 42PZ70EA  - 50"
PC for presenting stimuli	MacBook Air	Year 2013, running Mac OSX Mountain Lion
PC for recording stimuli	MacBook	Year 2009 running Windows 7	With a bootcamp partition of the disk for providing Windows OS
Statistical Analysis	Systat Software Inc.	Sigmaplot 11.0