the-spatial-memory-game-testing-relationship-between-spatial-language

The (Spatial) Memory Game: Testing the Relationship Between Spatial Language, Object Knowledge, and Spatial Cognition

The memory game paradigm is a behavioral procedure to explore the relationship between language, spatial memory, and object knowledge. Using two different ...

University of East Anglia

When looking at static visual images, people often exhibit mental animation, anticipating visual events that have not yet happened. But what determines when mental animation occurs? Measuring mental animation using localized brain function (visual motion processing in the middle temporal and middle superior temporal areas, MT+), we demonstrated that animating static pictures of objects is dependent both on the functionally relevant spatial arrangement that objects have with one another (e.g., a bottle above a glass vs. a glass above a bottle) and on the linguistic judgment to be made about those objects (e.g., "Is the bottle above the glass?" vs. "Is the bottle bigger than the glass?"). Furthermore, we showed that mental animation is driven by functional relations and language separately in the right hemisphere of the brain but conjointly in the left hemisphere. Mental animation is not a unitary construct; the predictions humans make about the visual world are driven flexibly, with hemispheric asymmetry in the routes to MT+ activation.

Multiple routes to mental animation: language and functional relations drive motion processing for static images.

Spatial demonstratives - terms including this and that - are among the most common words across all languages. Yet, there are considerable differences between languages in how demonstratives carve up space and the object characteristics they can refer to, challenging the idea that the mapping between spatial demonstratives and the vision and action systems is universal. In seven experiments we show direct parallels between spatial demonstrative usage in English and (non-linguistic) memory for object location, indicating close connections between the language of space and non-linguistic spatial representation. Spatial demonstrative choice in English and immediate memory for object location are affected by a range of parameters - distance, ownership, visibility and familiarity - that are lexicalized in the demonstrative systems of some other languages. The results support a common set of constraints on language used to talk about space and on (non-linguistic) spatial representation itself. Differences in demonstrative systems across languages may emerge from basic distinctions in the representation and memory for object location. In turn, these distinctions offer a building block from which non-spatial uses of demonstratives can develop.

Spatial demonstratives and perceptual space: describing and remembering object location.

Typical spatial language sentences consist of describing the location of an object (the located object) in relation to another object (the reference object) as in "The book is above the vase". While it has been suggested that the properties of the located object (the book) are not translated into language because they are irrelevant when exchanging location information, it has been shown that the orientation of the located object affects the production and comprehension of spatial descriptions. In line with the claim that spatial language apprehension involves inferences about relations that hold between objects it has been suggested that during spatial language apprehension people use the orientation of the located object to evaluate whether the logical property of converseness (e.g., if "the book is above the vase" is true, then also "the vase is below the book" must be true) holds across the objects' spatial relation. In three experiments using sentence acceptability rating tasks we tested this hypothesis and demonstrated that when converseness is violated people's acceptability ratings of a scene's description are reduced indicating that people do take into account geometric properties of the located object and use it to infer logical spatial relations.

Spatial language and converseness.

In three experiments, we investigated the influence of two types of language on memory for object location: demonstratives (this, that) and possessives (my, your). Participants first read instructions containing demonstratives/possessives to place objects at different locations, and then had to recall those object locations (following object removal). Experiments 1 and 2 tested contrasting predictions of two possible accounts of language on object location memory: the Expectation Model (Coventry, Griffiths, & Hamilton, 2014) and the congruence account (Bonfiglioli, Finocchiaro, Gesierich, Rositani, & Vescovi, 2009). In Experiment 3, the role of attention allocation as a possible mechanism was investigated. Results across all three experiments show striking effects of language on object location memory, with the pattern of data supporting the Expectation Model. In this model, the expected location cued by language and the actual location are concatenated leading to (mis)memory for object location, consistent with models of predictive coding (Bar, 2009; Friston, 2003).

Language and memory for object location.

This multiple single case study contrasted left hemisphere stroke patients (= 6) to healthy age-matched control participants (= 15) on their understanding of action (e.g., holding, clenching) and motion verbs (e.g., crumbling, flowing). The tasks required participants to correctly identify the matching verb or associated picture. Dissociations on action and motion verb content depending on lesion site were expected. As predicted for verbs containing an action and/or motion content, modified-tests confirmed selective deficits in processing motion verbs in patients with lesions involving posterior parietal and lateral occipitotemporal cortex. In contrast, deficits in verbs describing motionless actions were found in patients with more anterior lesions sparing posterior parietal and lateral occipitotemporal cortex. These findings support the hypotheses that semantic representations for action and motion are behaviorally and neuro-anatomically dissociable. The findings clarify the differential and critical role of perceptual and motor regions in processing modality-specific semantic knowledge as opposed to a supportive but not necessary role. We contextualize these results within theories from both cognitive psychology and cognitive neuroscience that make claims over the role of sensory and motor information in semantic representation.

Dissociation between Semantic Representations for Motion and Action Verbs: Evidence from Patients with Left Hemisphere Lesions.

This paper addresses the impact of dialogue strategies and functional features of spatial arrangements on communicative success. To examine the sharing of cognition between two minds in order to achieve a joint goal, we collected a corpus of 24 extended German-language dialogues in a referential communication task that involved furnishing a dolls' house. Results show how successful communication, as evidenced by correct placement of furniture items, is affected by: (a) functionality of the furniture arrangement; (b) previous task experience; and (c) dialogue features such as description length and orientation information. To enhance research in this area, our 'Dolldialogue' corpus ( www.dolldialogue.space ) is now available as a free resource.

Article	Year
Multiple routes to mental animation: language and functional relations drive motion processing for static images. Psychological science\| PubMed ID: 23774464	2013
Spatial demonstratives and perceptual space: describing and remembering object location. Cognitive psychology\| PubMed ID: 24445332	2014
Spatial language and converseness. Quarterly journal of experimental psychology (2006)\| PubMed ID: 26911205	2016
Language and memory for object location. Cognition\| PubMed ID: 27179309	2016
Dissociation between Semantic Representations for Motion and Action Verbs: Evidence from Patients with Left Hemisphere Lesions. Frontiers in human neuroscience\| PubMed ID: 28261070	2017
Communicative Success in Spatial Dialogue: The Impact of Functional Features and Dialogue Strategies. Language and speech\| PubMed ID: 28697700	2017

Kenny R. Coventry

Biography

JoVE Journal Publications

Other Publications