eye-tracking-during-visually-situated-language-comprehension

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing ...

Friedrich Schiller University Jena

Pickering & Garrod's (P&G's) framework assumes an efference copy based on the interlocutor's intentions. Yet, elaborate attribution of intentions may not always be necessary for online prediction. Instead, contextual cues such as speaker gaze can provide similar information with a lower demand on processing resources. 

Preparing to be punched: prediction may not always require inference of intentions.

To describe changes of motor and nonmotor disabilities in patient with peripheral facial palsy (FP) during treatment using the patient-reported outcome measures (PROMs) Facial Clinimetric Evaluation (FaCE), Facial Disability Index (FDI), and Short Form 36-Item Questionnaire (SF-36) and to analyze predictors for these changes

Nonmotor disabilities in patients with facial palsy measured by patient-reported outcome measures.

First comparison of two methods of needle insertion: long axis ("in-plane") versus short axis ("out-of-plane") approach, each with and without a prototype needle guidance system (NGS).

Comparison of in-plane and out-of-plane needle insertion with vs. without needle guidance.

Photographic cropping is the act of selecting part of a photograph to enhance its aesthetic appearance or visual impact. It is common practice with both professional (expert) and amateur (non-expert) photographers. In a psychometric study, McManus et al. (2011b) showed that participants cropped photographs confidently and reliably. Experts tended to select details from a wider range of positions than non-experts, but other croppers did not generally prefer details that were selected by experts. It remained unclear, however, on what grounds participants selected particular details from a photograph while avoiding other details. One of the factors contributing to cropping decision may be visual saliency. Indeed, various saliency-based computer algorithms are available for the automatic cropping of photographs. However, careful experimental studies on the relation between saliency and cropping are lacking to date. In the present study, we re-analyzed the data from the studies by McManus et al. (2011a,b), focusing on statistical image properties. We calculated saliency-based measures for details selected and details avoided during cropping. As expected, we found that selected details contain regions of higher saliency than avoided details on average. Moreover, the saliency center-of-mass was closer to the geometrical center in selected details than in avoided details. Results were confirmed in an eye tracking study with the same dataset of images. Interestingly, the observed regularities in cropping behavior were less pronounced for experts than for non-experts. In summary, our results suggest that, during cropping, participants tend to select salient regions and place them in an image composition that is well-balanced with respect to the distribution of saliency. Our study contributes to the knowledge of perceptual bottom-up features that are germane to aesthetic decisions in photography and their variability in non-experts and experts. 

Preference for Well-Balanced Saliency in Details Cropped from Photographs.

Patients with facial palsy (FP) not only suffer from their facial movement disorder, but also from social and psychological disabilities. These can be assessed by patient-reported outcome measures (PROMs) like the quality-of-life Short-Form 36 Item Questionnaire (SF36) or FP-specific instruments like the Facial Clinimetric Evaluation Scale (FaCE) or the Facial Disability Index (FDI). Not much is known about factors influencing PROMs in patients with FP. We identified predictors for baseline SF36, FaCE, and FDI scoring in 256 patients with unilateral peripheral FP using univariate correlation and multivariate linear regression analyses. Mean age was 52 ± 18 years. 153 patients (60 %) were female. 90 patients (31 %) and 176 patients (69 %) were first seen <90 or >90 days after onset, respectively, i.e., with acute or chronic FP. House-Brackmann grading was 3.9 ± 1.4. FaCE subscores varied from 41 ± 28 to 71 ± 26, FDI scores from 65 ± 20 to 70 ± 22, and SF36 domains from 52 ± 20 to 80 ± 24. Older age, female gender, higher House-Brackmann grading, and initial assessment >90 days after onset were independent predictors for lower FaCE subscores and partly for lower FDI subscores (all p < 0.05). Older age and female gender were best predictors for lower results in SF36 domains. Comorbidity was associated with lower SF General health perception and lower SF36 Emotional role (all p < 0.05). Specific PROMs reveal that older and female patients and patients with chronic FP suffer particularly from motor and non-motor disabilities related to FP. Comorbidity unrelated to the FP could additionally impact the quality of life of patients with FP.

Initial severity of motor and non-motor disabilities in patients with facial palsy: an assessment using patient-reported outcome measures.

A speaker's gaze behaviour can provide perceivers with a multitude of cues which are relevant for communication, thus constituting an important non-verbal interaction channel. The present study investigated whether direct eye gaze of a speaker affects the likelihood of listeners believing truth-ambiguous statements. Participants were presented with videos in which a speaker produced such statements with either direct or averted gaze. The statements were selected through a rating study to ensure that participants were unlikely to know a-priori whether they were true or not (e.g., "sniffer dogs cannot smell the difference between identical twins"). Participants indicated in a forced-choice task whether or not they believed each statement. We found that participants were more likely to believe statements by a speaker looking at them directly, compared to a speaker with averted gaze. Moreover, when participants disagreed with a statement, they were slower to do so when the statement was uttered with direct (compared to averted) gaze, suggesting that the process of rejecting a statement as untrue may be inhibited when that statement is accompanied by direct gaze. 

Direct Speaker Gaze Promotes Trust in Truth-Ambiguous Statements.

Variation in referential form has traditionally been accounted for by theoretical frameworks focusing on linguistic and discourse features. Despite the explosion of interest in eye tracking methods in psycholinguistics, the role of visual scanning behaviour in informative reference production is yet to be comprehensively investigated. Here we examine the relationship between speakers' fixations to relevant referents and the form of the referring expressions they produce. Overall, speakers were fully informative across simple and (to a lesser extent) more complex displays, providing appropriately modified referring expressions to enable their addressee to locate the target object. Analysis of contrast fixations revealed that looking at a contrast object boosts but is not essential for full informativeness. Contrast fixations which take place immediately before speaking provide the greatest boost. Informative referring expressions were also associated with later speech onsets than underinformative ones. Based on the finding that fixations during speech planning facilitate but do not fully predict informative referring, direct visual scanning is ruled out as a prerequisite for informativeness. Instead, pragmatic expectations of informativeness may play a more important role. Results are consistent with a goal-based link between eye movements and language processing, here applied for the first time to production processes.

Looking at a contrast object before speaking boosts referential informativeness, but is not essential.

Children's ability to refer is underpinned by their developing cognitive skills. Using a production task (n = 57), we examined pre-articulatory visual fixations to contrast objects (e.g., to a large apple when the target was a small one) to investigate how visual scanning drives informativeness across development. Eye-movements reveal that although four-year-olds fixate contrast objects to a similar extent as seven-year-olds and adults, this does not result in explicit referential informativeness. Instead, four-year-olds frequently omit distinguishing information from their referring expressions regardless of the comprehensiveness of their visual scan. In contrast, older children make greater use of information gleaned from their visual inspections, like adults. Thus, we find a barrier not to the incidence of contrast fixations by younger children, but to their use of them in referential informativeness. We recommend that follow-up work investigates whether younger children's immature executive skills prevent them from describing referents in relation to contrast objects.

Look before you speak: children's integration of visual information into informative referring expressions.

Language-processing accounts are beginning to accommodate different visual context effects, but they remain underspecified regarding differences between cues, both during sentence comprehension and subsequent recall. We monitored participants' eye movements to mentioned characters while they listened to transitive sentences. We varied whether speaker gaze, a depicted action, neither, or both of these visual cues were available, as well as whether both cues were deictic (Experiment 1) or only speaker gaze (Experiment 2). Speaker gaze affected eye movements during comprehension similarly early to a single deictic action depiction, but significantly earlier than non-deictic action depictions; conversely, depicted actions but not speaker gaze positively affected later recall of sentence content. Thus, cue type and cue-language relations must be accommodated in characterising real-time situated language comprehension and subsequent recall of sentence content.

Article	Year
Preparing to be punched: prediction may not always require inference of intentions. The Behavioral and brain sciences\| PubMed ID: 23790095	2013
Nonmotor disabilities in patients with facial palsy measured by patient-reported outcome measures. The Laryngoscope\| PubMed ID: 26421410	2016
Comparison of in-plane and out-of-plane needle insertion with vs. without needle guidance. European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery\| PubMed ID: 26498947	2016
Preference for Well-Balanced Saliency in Details Cropped from Photographs. Frontiers in human neuroscience\| PubMed ID: 26793086	2015
Initial severity of motor and non-motor disabilities in patients with facial palsy: an assessment using patient-reported outcome measures. European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery\| PubMed ID: 27040558	2017
Direct Speaker Gaze Promotes Trust in Truth-Ambiguous Statements. PloS one\| PubMed ID: 27643789	2016
Looking at a contrast object before speaking boosts referential informativeness, but is not essential. Acta psychologica\| PubMed ID: 28628785	2017
Look before you speak: children's integration of visual information into informative referring expressions. Journal of child language\| PubMed ID: 29739488	2018
Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions. Acta psychologica\| PubMed ID: 29858107	2018

Helene Kreysa

Biography

JoVE Journal Publications

Other Publications