How language incorporates affordances

Lucia RiggioLucia Riggio

Department of NeuroscienceDepartment of Neuroscience

University of ParmaUniversity of Parma

Cambridge 3 September 2013

Affordances refer to properties of an object or to specific parts of it (typically the handle) that trigger components of actions such as reaching or grasping

Visual perception of objects activates the very system responsible for their actual manipulation, that transforms object pragmatic features into appropriate motor programs for their interaction

Affordances

Handle graspability

[ Buccino et al 2009 ]

Visual presentation of manipulable objects in which the most important feature relevant for action is violated (e.g., a cup with a broken handle) prevents motor activation from occurring.

Object graspability

[ Taylor & Zwaan 2010]

Dynamic responses (i.e. the grip force) to a set of spheres with increasing size were no longer correlated with the apparent weight of the stimuli, if the experimenter referred to them as planets, rather than graspable solids

Language

Embodied theories predict that nouns of graspable objects should also activate affordances since understanding of language is achieved by recruiting the same sensorimotor systems activated when we experience the action or object a word refers to.

Presentation of action verbs associated with different effectors results in somatotopic activation of motor areas (e.g. Pulvermüller et al 2001; Hauk et al 2004; Tettamanti et al 2005)

Modulation in the MEP for a muscle of a given effector associated with the verb expressed in a sentence (e.g. Buccino et al 2005)

Modulation of RTs when the effector used to respond is also involved in the action expressed by the linguistic material (e.g. Buccino et al 2005; Boulenger et al 2006; Sato et al 2008; Dalla Volta et al 2009) or when the response requires the execution of a movement in the same direction of the action described by the sentence (e.g., Glenberg & Kaschak 2002)

Verbs

Verbs : interference and facilitation

Interference effects within 200-250 ms after the presentation of action-related word: slowing–down of the RTs and decrease of the MEPs when either the same effector as that expressed by the action verb or sentence (e.g. Sato et al 2008; Buccino et al 2005) or the same directional movement are involved (de Vega et al 2013).

Facilitation effects in the matching conditions for longer time intervals (e.g. Glenberg & Kaschak 2002; Zwaan & Taylor 2006; de Vega et al 2013).

Nouns

Is the early involvement of the motor system, observed during the understanding of action-related sentences and verbs, also present for nouns?

Nouns: Materials

Hand-related nouns

Foot-related

nouns

Abstract

nouns

Forbici (scissors) Pedale (pedal) Tattica (tactics)

Tazzina (cup) Pantofola (slipper) Superbia (arrogance)

Spazzola (brush) Pedana (footboard) Ritegno (reluctance)

Forchetta (fork) Pattini (skate) Elogio (praise)

Pettine (comb) Gradino (step) Deferenza (deference)

Matita (pencil) Mocassino (moccasin) Gelosia (jealousy)

Accendino (lighter) Scalinata (staircase) Lusinga (allurement)

Martello (hammer) Ciabatta (slipper) Inganno (deceit)

[Marino et al 2013]

Nouns: behavioural experiment

Nouns: behavioural experiment

Nouns: TMS experiment

Nouns: TMS experiment

*

Nouns: conclusions

Both experiments indicate the early involvement of the left motor cortex in the representation of nouns of graspable objects, in line with a causal role of the motor system in language understanding. This modulation is comparable to that previously found with verbs or action related sentences (e.g. Hauk & Pulvermüller 2004; Buccino et al 2005; Boulenger et al 2006; Sato et al 2008).

The differential pattern of results between the two hemispheres is consistent with the dominance of the left hemisphere for language and tool processing.

Nouns, as their external referents, elicit motor information.

[Borghi & Riggio 2009]

Stable and variable affordances

Stable affordances: stable aspects of the interaction with objects, which can be included in an object representation stored in memory

Variable affordances: variable aspects, which change depending on the way the object is presented

Canonical affordances: variable aspects associated with a canonical interaction

The comprehension of words referring to a particular object activates only stable and canonical affordances .

Objects and nouns in space

The activation of motor information associated with visual objects is spatially constrained (e.g. Costantini et al 2010; 2011)

Is motor information elicited by nouns spatially constrained?

[Ferri et al 2011]

Task: Participants had to signal whether an object, presented either within or outside their reachable space, was natural or manufactured by making reach-to-precision or reach-to-power grasp responses.

Experiment 1: objects

Reach-to-grasp response device

Stimuli were presented in a 3-D room created by means a 3-D Studio MaxTM

Experiment 2: nouns

Task: Participants had to signal whether the object (near or far) was the same (congruent) as that indicated by the word, or different, producing reach-to-precision or reach-to-power grasp responses.

An action compatibility effect between the response and the grip evoked by the object, only in the reaching space and only for artefacts

Experiment 1: objects Experiment 2: nouns

An action compatibility effect between the response and the grip evoked by the noun that was not modulated by the spatial position of the visual object

Variable features such as distance and location are not integrated in the motor representation of the objects elicited by language.

Results

Conclusions

Action information relative to the size of an object is spatially constrained for visual objects but not for their nouns.

Space information elicited by a seen object is different from that elicited by nouns.

Examples of stimuli

[ Yoon et al 2010 ]

Converging results

Task: Right-handed participants had to decide whether the paired objects would typically be used together.

Results: Decision times faster when stimuli were in the correct locations for a right-hand action (active member on the right) only when objects were presented.

Do concrete verbs and nouns combined in single sentences have a different weight on the modulation of the motor system?

Combinations of nouns and verbs

Material and ProcedureSensible and non-sensible sentences formed by concrete verbs and nouns were presented in a sensibility judgment task. Verbs could express an action that could be performed in a specific way (low degrees of freedom) or in very many different ways (high degrees of freedom). Nouns referred to graspable and non-graspable objects.

Sensible Non-sensibleTo squeeze the orangeTo squeeze the orange To To

water the handlewater the handle

VLDoF Noun of graspable objectTo recycle the bottleTo recycle the bottle To consult the onionTo consult the onion

To water the flowerbedTo water the flowerbed To squeeze the sunsetTo squeeze the sunset

VHDoF Noun of non-grasp. object To consult the archivesTo consult the archives To recycle the herdTo recycle the herd

Sensorimotor specificity of verbs and nouns

[Marino et al 2012]

Results

With Verbs with LDoF performance is independent on the noun type: activation

of specific features of objects on which the action is usually performed

With Verbs with HDoF performance is affected by the noun type

The motor system is modulated by concrete nouns in a way comparable to concrete verbs

Nouns seem to activate only stable and canonical affordances. Distance and location, related to the reaching of the object, seem not to be represented.

When nouns are combined with verbs the sensorimotor specificity of both words could have a role in the motor activation.

In short