hand-object interaction in perspectivelaral.istc.cnr.it/pubblicazioni/english/journals/bruzzo... ·...

Bruzzo, Borghi & Ghirlanda Hand-object interaction in perspective

1

HAND-OBJECT INTERACTION IN PERSPECTIVE

*Angela Bruzzo,

1 Anna M. Borghi,

1 Stefano Ghirlanda

1

In press in Neuroscience Letters, 2008.

* Angela Bruzzo

1Department of Psychology

University of Bologna

Viale Carlo Berti Pichat, 5

40127 Bologna, Italy

Phone: +39-051-2091838

Fax: +39-051-243086

E-mail [email protected]

Acknowledgements

Study supported by a grant from Italian MIUR. Thanks to Bernhard Hommel, Diane Pecher and

an anonymous reviewer for useful comments.


2

Abstract

We investigated the effect of perspective on the recognition of actions, without using

motor preparation. Photographs of a hand wearing a glove were presented as primes, followed by

photographs of the same hand interacting with an object. Both primes and targets were shown in

ego- or non-egocentric perspective. Participants had to decide whether or not the hand interacted

with the object in a sensible way. In order to increase the similarity between the perceived and

the enacted movement, half of the participants were required to wear a glove while responding.

We found an advantage of the egocentric over the non-egocentric perspective for targets in the

Glove condition. The advantage of the egocentric perspective was present for primes as well,

even though the effect was limited to the No Glove condition. Results are discussed in the

framework of the recent literature on mirror neurons and body schema.

Key words: Body schema; Mirror system; Motor preparation; Perspective; Theory of Event

Coding (TEC); Action.


3

Manuscript

Introduction

To identify the movement of a specific body part, the brain constructs a representation of the

general body structure. In doing this, the brain relies on the so-called body schema [12, 13],

which encompasses the perceptions and dynamics of an individual’s own body in relation to that

of another [2]. Neuro-imaging evidence [1, 10] suggests that a multimodal representation of

one’s own body might be involved in the perception of other people’s bodies [18, 23].

Studies on the mirror system (MS) show that, whenever we look at someone performing an

action, there is, in addition to the activation of various visual areas, a concurrent activation of the

motor circuits recruited when we ourselves perform that action [10, 15]. In other words, seeing

an action is thought to activate a motor simulation [4, 17].

The brain imaging studies on the MS in humans are consistent with predictions advanced by

common coding theories [14, 19], which focus on the link between perception and action.

According to the Theory of Event Coding (TEC), perceived events (perceptions) and events to be

produced (actions) are represented by the same integrated, task-tuned networks of feature codes

in cognitive structures, called “event codes”. One decisive feature of such representations is the

commensurability of contents of perception and action intentions, ultimately stemming from the

fact that both perception and action plans represent “events in the environment”[14]. This theory

generates an important prediction: if perception and action draw on identical representational

structures, the similarity between the seen stimuli and the performed actions facilitates

processing of the seen stimuli.

In keeping with TEC, recent brain-imaging studies have demonstrated that the perspective

through which we perceive others’ actions influences performance [6,16] and that distinct areas


4

of the posterior parietal cortex are specialized for egocentric and non-egocentric perspectives

[24].

Along the same lines, behavioral studies have focused on the information elicited by a hand

photograph presented in different orientations [e.g., 7]. In a study by [25], the authors

manipulated the perspective in which the hand was presented. The hand could either match the

end posture of the observer’s own hand (egocentric perspective) or the end posture of that of

another person (non-egocentric perspective). The authors [25] found a congruency effect for the

egocentric perspective when a neutral hand stimulus was given as a preview and for the non-

egocentric perspective when the prime stimulus was preceded by a fixation dot. They explained

the non-egocentric advantage as a stimulus-driven visuo-motor effect, which rapidly encodes

suddenly appearing conspecifics’ hands. More importantly, the ego- perspective advantage can

be interpreted as a planning-driven motor-visual priming effect that selectively enhances the

visual processing of body parts. Even if the authors do not discuss this point, the reason for the

advantage of the egocentric perspective could lie in the higher similarity between the perceived

stimulus and the performed action.

Our study aims to verify whether the similarity between the execution modalities (how) of the

perceived and the performed action can facilitate processing of motor information related to the

seen stimulus. For this reason we manipulated both the perspective (egocentric and non-

egocentric) of a visually presented hand and the morphological similarity between the seen hand

and the responding hand.

Participants were presented with a prime consisting of a photograph of a human hand wearing a

glove. The hand was presented alone, in four different postures. The hand-prime was followed

by a target consisting of the photos of the hand either correctly grasping an object or performing

an action that did not make sense in relation to the object. The task was to decide whether or not

the illustrated action made sense by pressing the relevant key on the keyboard. For example, it

makes sense to use a power grip to grasp an apple whereas a crossing finger action does not


5

make sense in relation to an apple. Each photograph was presented in an egocentric and non-

egocentric perspective. By egocentric perspective, we refer to the perspective consistent with

looking at one’s own hand, while by non-egocentric perspective we refer to the perspective

consistent with looking at someone else’s hand performing an action [22]. To manipulate the

similarity between the observed stimulus and the performed action, half of the participants wore

a glove while performing the experiment (Glove condition), while the other half did not (No

Glove Condition).

Two main hypotheses derived from TEC were tested.

1. We predicted an advantage of the ego- over the non-egocentric perspective due to the increase

in similarity between the perceived and the performed action. Also, we hypothesized a

compatibility effect between the hand and the hand-object perspective. Hence, a hand in

egocentric perspective followed by an egocentric hand-object interaction should elicit the fastest

response.

2. The presence of the glove should improve performance due to the inferred visuotactile

similarity between the seen stimulus (the hand wearing a glove) and our own body part (our own

hand wearing a glove). In particular, we predicted that the best performance should be found

when participants wore a glove and saw the hand interacting with the object in an egocentric

perspective. In keeping with the TEC, this improvement might be due to the increase of overlap

between the perceived and the acted “events”.


6

Methods

Participants

Forty right-handed students from the University of Bologna, with normal or corrected-to-normal

vision, participated in the experiment, which was carried out according to the ethical guidelines

laid down by the Committee on Human Research of Bologna (Italy). All participants gave their

written informed consent for their participation in the study, and were unaware of the purpose of

the experiment.

Stimuli

Primes consisted of digital photos of a human hand wearing a glove. Presenting a hand

wearing a glove could seem like an artificial step, but it was crucial in order to avoid the

recognition of particular morphological features. We used gloves, in line with previous studies

[e.g., 8], because other strategies, such as painting the seen hand, would have not impeded

recognition of the morphological differences between the subject’s own and the seen hand. The

photos displayed four different hand postures as primes, two prehensile postures (a precision and

a power grip) (e.g., for precision grip Fig.1 A, B), and two postures not related to a prehensile

action (horns and a victory signal) (e.g., for horns Fig. 1 C, D). A hand with a lifted index finger

worked as a catch trial. As targets, we selected sixteen photos of actions performed with

everyday objects. Eight displayed a hand grasping an object with a precision grip (e.g., a tooth-

brush) (Fig. 1 E, F) and eight with a power grip (e.g., an orange, Fig. 1 G, H). Besides the sixteen

photos displaying a correct grasping action performed with the object (referred to from here

onwards as sensible actions), we presented sixteen other photos displaying a hand action that did

not make sense in relation to the object (Fig. 1Q, R, S, T). All photos were presented with the

hand in both an egocentric and a non-egocentric perspective and were equated for luminance.


7

Thus, we had a total of 10 primes and 64 targets, 32 representing a sensible action and 32

displaying an action that did not make sense. Each target-stimulus was presented once with each

different prime action.

Insert Figure 1

Procedure

The experiment was performed in a quiet, dimly-lit laboratory room. Each participant was

randomly assigned to one of two different conditions. In the first condition (Glove condition),

they were asked to wear two blue gloves in order to augment the similarity between the observed

and the performed actions. Our interest was not the effect of the glove per se, but the effects of

the increased similarity between the participant’s effector and the perceived effector. In the other

condition (No glove), participants simply had to perform the task. For both groups the

instructions were the same. Participants were required to wear the gloves both during the initial

training phase, in which participants familiarized themselves with a separate training set of

items, and during the experiment, composed of two blocks, counterbalanced in order across

participants. Half of the participants in each group were asked to respond “yes” with the right

index in the first block and “no” in the second; the other half was asked to do the opposite. A

chin rest was used in the trials, allowing the participants, in principle, to glimpse their own hand.

However, they were invited to focus their attention on the center of the screen, where the visual

stimuli appeared. Each trial began with the 250 ms display of a photo of a prime, followed by the

photo of the target. Participants were instructed to refrain from responding when a photo of a

hand with the lifted finger (catch-trial) appeared. Otherwise, they had to indicate by pressing a

right or a left button on the keyboard whether or not the target represented an action that made

sense. The photo displaying a hand-object interaction remained on the screen until participants


8

responded, with a cut off at 2000 ms. We did not introduce any SOA between prime and target

presentation. Participants received feedback when they provided a correct answer. Both reaction

times and errors were recorded.

Data analysis

A mixed multifactorial ANOVA was performed on errors, with the factors of Prime Perspective

(ego- vs. non-egocentric) and Target Perspective (ego- vs. non-egocentric) manipulated within

participants, and Morphological Similarity manipulated between participants. No effect was

significant. As seen in Table 1, the error pattern indicated an advantage of the egocentric over

the non-egocentric perspective, more marked with Primes than with Targets. The error analysis

revealed that there was no speed-accuracy trade off, so we focused on an analysis of reaction

time. Reaction times more than 2 standard deviations from each participant average were

discarded. No participant was excluded from the analysis. Correct RTs were entered into a mixed

multifactorial ANOVA. The factors of Prime Perspective (ego- vs. non-egocentric) and Target

Perspective (ego- vs. non-egocentric) were manipulated within participants, while Morphological

Similarity (Glove vs. No Glove) was a between participants factor.


9

Results

Our analyses focused on sensible actions. We avoided analyzing non-sensible actions, because

our stimuli were construed and our hypotheses were formulated referring to sensible actions, in

analogy with the standard procedure of psycholinguistic and language grounding studies on

action sentence sensibility [e.g., 11].

The Prime perspective was not significant, F (1, 38) = 0.03, MSe = 1497.31, p =.50. However, we

found a main effect of Target Perspective, F (1, 38) = 10.23, MSe = 3147.87, p < .01, due to the

fact that targets were processed faster in an egocentric (M=712.62 ms) than in a non-egocentric

perspective (M = 732.68 ms). The main effect of Morphological Similarity (F (1, 38) =6.92,

MSe = 10892.2, p <. 0122) might be due to the simple fact that wearing a glove slows down

responses. However, the significant interactions between Morphological Similarity and Target

Perspective, F (1, 38) = 6.92, MSe = 3147.87, p <.01 and Similarity and Prime Perspective, F(1, 38)

= 5.85, MSe = 1620.18, p < .05 (see Fig. 2, panel A and B) are theoretically relevant.

Insert Figure 2

The first interaction was due to the advantage of the Egocentric Target over the Non-egocentric

Target in the Glove condition (Newman-Keuls, p < .001), the second due to the advantage of the

Egocentric Prime over the Non-egocentric Prime in the No Glove condition (Newman-Keuls,

p <.05). Having obtained significant interactions between Morphological Similarity and Prime

Perspective, and between Morphological Similarity and Target Perspective, we decided to

perform two separate ANOVAs for the Glove and No Glove conditions to better understand the

double interactions. The decision to perform separate ANOVAs was also due to the difference in

RTs between the Glove and No Glove conditions. The factors of both ANOVAs, manipulated

within participants, were Prime Perspective and Target Perspective. In the No Glove condition,


10

the factor Prime Perspective reached significance due to the advantage of the ego-over the non-

egocentric perspective, F (1, 19) = 4.33, MSe = 896.93, p = .05. In the analysis of the Glove

condition, Target Perspective was significant, as targets in the egocentric perspective produced

faster RTs than targets in the non-egocentric perspective, F (1, 19) = 10.15, MSe = 2634.90, p <

.01. Therefore, our first hypothesis, that the egocentric perspective has an advantage over the

non-egocentric perspective, was confirmed. Importantly, whereas the egocentric prime

advantage was restricted to the No Glove condition, the egocentric target advantage was more

pronounced in the Glove condition. This may be due to the fact that, if the similarity between the

perceived and the performed action is increased, as was the case in the Glove condition,

participants become more sensitive to the role played by the action goal, that is, by the Target.

Interestingly, the predicted congruency effect between the perspective of Prime and Target

almost reached significance, F (1, 38) = 3.64, MSe = 828.07, p < .06 (Fig. 3).

Insert Figure 3

The results show both the advantage of the egocentric perspective with targets as well as a

prime-target compatibility effect. Namely, faster RTs were obtained when an egocentric Prime

was followed by egocentric Target (M=708.02 ms), whereas a non-egocentric prime followed by

a non-egocentric target had an advantage of only 3 ms compared to pairs composed of a non-

egocentric prime followed by an egocentric target.

In order to be sure that our effects were due to perspective and not to congruency, we analysed

the results a second time removing any of the items that were palm up but could represent

“canonical” views of one’s own hand (horns and victory signals in both perspectives).

Accordingly, a further ANOVA was performed with the factors of Prime Perspective (ego- vs.

non-egocentric) and Target Perspective (ego- vs. non-egocentric) manipulated within

participants, and Glove/No Glove condition manipulated between participants. The results we


11

found matched the results found with all hand postures perfectly. Specifically, the difference

between Target Egocentric and Non-egocentric Perspective was significant, F (1, 38) = 8.93, MSe

= 2150.4, p < .01. In addition, there was a significant interaction between Morphological

Similarity and Target, F(1, 38) = 5.22, MSe = 2150.4, p < .05, due to the fact that the egocentric

Target was processed much faster than all other targets (Newman-Keuls, p <.001). Finally, the

interaction between Morphological Similarity and Prime Perspective also reached significance, F

(1, 38) = 5.9, MSe = 1497.3, p < .05, due to the advantage of the No Glove over the Glove

condition and to the fact that, in the No Glove condition, the Prime was processed slightly faster

in the egocentric perspective than the non-egocentric Prime (Newman-Keuls, p =.07).


12

Discussion

Our results are clearly in keeping with the TEC theory, that is, with the idea that there is a

common framework for the recognition and planning of motor actions.

First, we found a slight advantage in responding when targets were preceded by egocentric rather

than by non-egocentric primes. The fact that this effect was confined to the No Glove condition

could be due to the perceptual salience of the glove in particular for participants who did not

wear a glove.

Second, and more crucially for our predictions, in the Glove condition egocentric Targets were

processed faster than non-egocentric Targets. The results on Targets are particularly relevant

because target pictures displayed hand-object interactions. The effect obtained in the Glove

condition with targets and not with primes indicates that the advantage of the egocentric

perspective occurred when the hand was presented together with the object, that is, when the

goal of actions (“event in the environment”) was displayed. We interpret this result as an

indication of the synergic contribution of both object-related (canonical) and action-related

(mirror) neurons during observation of actions directed towards graspable objects. It can be

objected that the perspective effect should extend to the prime as well, particularly in the Glove

condition. However, in our defense we could advance at least two arguments. The first is that we

believe our results capture the main point of TEC. This theory concerns the planning of actions

and claims that an event is more easily perceived in cases of correspondence between the

planned action and the environment in which the action is executed. Regarding our stimuli, the

event was displayed in the target, where the action was actually presented. The second argument

is that we did find the predicted Prime-Target compatibility effect, even though this result did


13

not reach full significance; participants performed particularly well when egocentric Targets

followed egocentric Primes.

Our study has implications for research on perspective taking. Recent studies provide

evidence of perspective effects. For example, in a fMRI study [16], participants either passively

watched or imitated hand or foot actions depicted in video clips. The video clips depicted actions

filmed either from an egocentric or a non-egocentric perspective. Latency for imitation was

significantly shorter for the egocentric than for the non-egocentric perspective. Functional

imaging results showed more activity in the left sensory motor cortex for the egocentric

perspective, even during simple observation, and in the lingual gyrus for the non-egocentric

perspective. Furthermore, neural evidence confirms our sensitivity to action perspective. For

example, [6] have shown the existence of neurons in the superior temporal sulcus that

differentiate self-produced actions from the actions of others. PET studies [21] have

demonstrated that cortical activations in inferior parietal, precuneus and somatosensory differ

when humans imagine actions in egocentric or non-egocentric person perspective. In addition, a

number of neuropsychological studies on both normal subjects and patients have shown that

shifts in perspective might lead to erroneous attributions of our own actions to others (e.g., 8). In

these studies participants typically saw their own effectors acting according their own or the

experimenter’s perspective.

Our behavioral study both confirms such perspective effects and extends previous results.

Namely, our findings suggest that visually displayed hands interacting with objects activate a

sort of motor simulation, which is sensitive to perspective. Differently from previous studies, we

found that even seeing static stimuli portraying the performance of an action is sufficient to

activate a representation of our body schema and to induce a resonance effect. Importantly, the

perspective effect driven by a motor simulation occurred even in the absence of a motor-

preparation phase and without a task that strongly involves the motor system, such as a simple

key pressure task.


14


15

References

[1] S.V. Astafiev, C.M. Stanley, G.L. Shulman, M Corbetta. Extrastriate body area in human

occipital cortex responds to the performance of motor actions, Nature Neurosci 7 (2004) 542–

548.

[2] G. Berlucchi & Aglioti S. The body in the brain: neural basis of corporeal

awareness. Trends in Neurosciences 20 (1997) 560-564.

[3] A.M. Borghi, C. Bonfiglioli, L. Lugli, P. Ricciardelli, S. Rubichi, R. Nicoletti. Are visual

stimuli sufficient to evoke motor information? Studies with hand primes, Neurosci. Lett. 411

(2007) 17-21.

[4] A. Bruzzo, B., Gesierich, A. Wohlschlaeger. Simulating biological and non biological

motion. Brain and Cogn 66 (2008) 145-149.

[5] D.N. Bub, M.E.J. Masson, C.M. Bukach. Gesturing and naming: the use of functional

knowledge in object identification, Psychol Sci 14 (2003) 467-472.

[6] D.P. Carey, D.I. Perrett, & M.W. Oram. Recognizing, understanding and reproducing action.

In: Jeannerod, M., Grafman, J., (Eds). Handbook of Neuropsychology-Action and Cognition,

1997.

[7] L. Craighero, A. Bello, L. Fadiga, G. Rizzolatti. Hand action preparation influences the

responses to hand pictures, Neuropsychologia 40 (2002) 492-502.


16

[8] E. Daprati , N.FranK, N.Georgieff, et al. Looking for the agent: an investigation into

consciousness of action and self-consciousness in schizophrenic patients. Cognition 65 (1997)

71-86.

[9] G. di Pellegrino, L. Fadiga, L. Fogassi, V.Gallese, G. Rizzolatti. Understanding

motor events: a neurophysiological study. Exp Brain Res 9 (1992) 176-180.

[10] P.E. Downing, Y. Jiang, M. Shuman, N. Kanwisher. A cortical area selective for visual

processing of the human body, Science 293 (2001) 2470–2473.

[11] A. Glenberg & M.P. Kaschak, Grounding language in action. Psychonomic Bull & Rev 9

(2002) 558-565.

[12] P. Haggard, K. Kitadono, C. Press, M. Taylor-Clarke The brain’s finger and hands. Exp

Brain Res 172 (2006)94-102.

[13] H. Head, & H.G. Holmes Sensory disturbances from cerebral lesions. Brain 34 (1911) 102-

254.

[14] B. Hommel, J. Musseler, G. Aschersleben, W. Prinz. The Theory of Event Coding (TEC): a

framework for perception and action planning, Behav Brain Sci 24 (2001) 849-878.

[15] M. Iacoboni, & M. Dapretto. The mirror neuron system and the consequences of its

dysfunction. Nature Review Neuroscience 7 (2006) 942-51.


17

[16] P.L. Jackson, A.N. Meltzoff, & J. Decety. Neural circuits involved in imitation and

perspective taking, Neuroimage 31 (2006) 429-439.

[17] M. Jeannerod. Motor Cognition: What actions tell the self. Oxford University press (2007).

[18] D.T. Leube, G. Knoblich, M. Erb, W. Grodd, M. Bartels, T.T. Kircher. The neural

correlates of perceiving one’s own movements, Neuroimage 20 (2003) 2084-2090.

[19] W. Prinz. Perception and action planning. Eur J Cogn Psychol 9 (1997) 129-154.

[20] C.L. Reed, & M.J.Farah. The psychological reality of the body schema: A test with normal

participants, J Exp Psychol Hum Percept Perform 21 (1995) 334–343.

[21] R. Ruby, & J. Decety. Effect of subjective perspective taking during simulation of action: a

PET investigation of agency, Nat. Neurosci 4 (2001) 546-550.

[22] R. Saxe, N. Jamal, & L. Powell. My Body or Yours? The effect of visual perspective on

cortical body representations, Cereb. Cortex 6 (2006) 178-182.

[23] A.Tessari, & A.M. Borghi (2007). Body image and body schema: The shared

representation of body image and the role of dynamic body schema in perspective and imitation.

Commentary to the target-article 'Somatosensory processes subserving perception and action' by

Dijkerman & de Haan. Behav Brain Sci. 30 (2007) 221-222.


18

[24] K. Vogely & G.R. Fink. Neural correlates of the first-person-perspective, Trends Cogn Sci.

7 (2003) 738-742.

[25] S. Vogt, P. Taylor, B. Hopkins. Visuo-motor priming by pictures of hand postures:

perspective matters, Neuropsychologia 41 (2003) 941-951.


19

FIGURES LEGEND

Figure 1. The stimuli. The photos display: four different hand postures used as primes, in

egocentric (A, C, E, G) or in non-egocentric perspective (B, D, F, H), a catch trial in both

perspectives (I, L), and some targets. The targets display a precision grip in egocentric (fig. 1M:

grasping a tooth-brush) and in non-egocentric perspective (fig. N), and a power grip, in

egocentric (fig. O: grasping an orange) and non-egocentric perspective (fig.P). In fig Q, R, S, T

non-sensible actions are displayed.

Figure 2. The interactions. (A) Targets. In the Glove condition, Targets in an egocentric

perspective produced faster RTs than Targets in a non-egocentric perspective. (B) Primes. In the

No Glove condition, Targets preceded by Primes in egocentric perspective were processed faster

than Targets preceded by Primes in a non-egocentric perspective. Asterisks mark significant

comparisons (p < 0.01). Bars show standard errors.

Figure 3. The congruency effect between the perspective of Prime and Target. The fastest

RTs were obtained when an egocentric Target followed an egocentric Prime.

Table 1. Error Analysis. The values of accuracy are shown (%) for each condition.


20


21


22


23

hand-object interaction in perspectivelaral.istc.cnr.it/pubblicazioni/english/journals/bruzzo... ·...

Documents