Attention Capture by Sudden

and Unexpected Changes: A Multisensory Perspective

Erik Marsja

Department of Psychology

Umeå 2017

This work is protected by the Swedish Copyright Legislation (Act 1960:729) Dissertation for PhD ISBN: 978-91-7601-803-3 Cover art by Olof Marsja (front) and Erik Marsja (back) Electronic version available at: http://umu.diva-portal.org/ Printed by: UmU Print Service, Umeå University Umeå, Sweden 2017

To Angelica and Elliot,

you make me a better person

i

Table of Contents

Abstract ............................................................................................ iii

Abbreviations ................................................................................... iv

List of studies .................................................................................... v

Enkel sammanfattning på svenska ................................................... vi

Introduction ....................................................................................... 1 Attention ........................................................................................................................... 1

Distraction/orienting ................................................................................................ 2 Electrophysiological responses to deviating stimuli ................................................3 Behavioral responses to deviating stimuli ............................................................... 5

Short-term memory ......................................................................................................... 6 Multicomponent ......................................................................................................... 7 Embedded-processes model ....................................................................................... 7 Duplex-mechanism account ..................................................................................... 8 Distraction and short-term memory ....................................................................... 8

Violation of predictions and attention capture .............................................................. 9 Multisensory accounts on attention, distraction, and short-term memory ................. 11

Central or sensory-specific mechanisms? .............................................................. 12 Multisensory neural models? .................................................................................. 14

Aims ................................................................................................. 15 Specific aims: .................................................................................................................. 15

Study I:...................................................................................................................... 15 Study II: .................................................................................................................... 15 Study III: ................................................................................................................... 15

Materials and methods .................................................................... 16 Subjects ........................................................................................................................... 16 Instruments/procedure .................................................................................................. 16

Materials ................................................................................................................... 16 Crossmodal oddball task (Study I & II) .................................................................. 18 Verbal & spatial serial recall task (Study III) ....................................................... 20 Data analysis ............................................................................................................ 21

Results ............................................................................................ 23 Study I: ........................................................................................................................... 23

Aim ........................................................................................................................... 23 Results ...................................................................................................................... 23

Study II: ......................................................................................................................... 25 Aim ........................................................................................................................... 25 Results ...................................................................................................................... 25 Conclusion ................................................................................................................ 26

Study III: ......................................................................................................................... 27 Aim ............................................................................................................................ 27 Results ....................................................................................................................... 27

ii

Conclusion ................................................................................................................ 28

Discussion ....................................................................................... 29 Central or specific mechanisms? ............................................................................ 29 Implications for theories on attention/distraction/prediction ............................ 30 Implications for theories on short-term memory .................................................. 31 Practical implications ............................................................................................. 32 Limitations ............................................................................................................... 32 Conclusions .............................................................................................................. 34

Acknowledgement ........................................................................... 35

References ....................................................................................... 37

Appendix ......................................................................................... 46 Study 1 ...................................................................................................................... 46 Study 2 ..................................................................................................................... 46 Study 3 ..................................................................................................................... 46

iii

Abstract

The main focus for this thesis was cross-modal attention capture by sudden and

unexpected sounds and vibrations, known as deviants, presented in a stream the

same to-be-ignored stimulus. More specifically, the thesis takes a multisensory

perspective and examines the possible similarities and differences in how deviant

vibrations and sounds affect visual task performance (Study I), and whether the

deviant and standard stimuli have to be presented within the same modality to

capture attention away from visual tasks (Study II). Furthermore, by presenting

spatial deviants (changing the source of the stimuli from one side of the body to

the other) in audiotactile (bimodal), tactile, and auditory to-be-ignored, it

explores whether bimodal stimuli are more salient compared to unimodal (Study

III). In addition, Study III tested the claims that short-term memory is domain-

specific.

In line with previous research, Study I found that both auditory and tactile

deviants captured attention away from the visual task. However, the temporal

dynamics between the two modalities seem to differ. That is, it seems like practice

causes the effect of vibratory deviants to reduce, whereas this is not the case for

auditory deviants. This suggests that there are central mechanisms (detection of

the change) and sensory-specific mechanisms.

Study II found that the deviant and standard stimuli must be presented within

the same modality. If attention capture by deviants is produced by a mismatch

within a neural model predicting upcoming stimuli, the neural model is likely

built on stimuli within each modality separately.

The results of Study III revealed that spatial and verbal short-term memory are

negatively affected by a spatial change in to-be-ignored sequences, but only when

the change is within a bimodal sequence. These results can be taken as evidence

for a unitary account of short-term memory (verbal and spatial information

stored in the same storage) and that bimodal stimuli may be integrated into a

unitary percept that make any change in the stream more salient.

iv

Abbreviations

ERP (Event-Related Potentials)

fMRI (functional Magnetic Resonance Imaging)

MMN (Mismatch Negativity)

OOR (Object-Oriented Records)

OR (Orienting Response)

RON (Reorienting Negativity)

SOA (Stimulus-Onset-Asynchrony)

STM (Short-term memory)

ISI (Interstimulus Interval)

ITI (Inter-Trial Interval)

TBI (To-Be-Ignored)

TBR (To-Be-Recalled)

v

List of studies

I Marsja, E., Neely, G., & Ljungberg, J. K. (Under Review). The Effects of

Unexpected and Sudden Vibrations and Sounds Does Not Disrupt

Performanice Similarly Over Time. Manuscript

II Marsja, E., Neely, G., & Ljungberg, J. K. (In Press). Investigating Deviance

Distraction and the Impact of The Modality of the To-Be-Ignored Stimuli.

Experimental Psychology

III Marsja, E., Marsh, J. E., Hansson, P., & Neely, G (Submitted). Examining

the Role of Spatial Changes in Bimodal and Uni-Modal To-Be-Ignored

Stimuli and How They Affect Short-Term Memory Processes. Manuscript

vi

Enkel sammanfattning på svenska

Denna avhandling fokuserade på hur uppmärksamhet fångas mellan sensoriska

modaliteter. Fokus låg på hur plötsliga och oväntade ljud samt vibrationer

presenterade i en ström av upprepande stimuli påverkar prestation (dvs.,

irrelevanta sekvenser). Mer specifikt tar uppsatsen ett multisensoriskt perspektiv

och undersöker möjliga likheter och skillnader i hur plötsliga och oväntade

vibrationer samt ljud påverkar prestation i en visuell uppgift (Studie I), och

huruvida avvikande och upprepande stimuli måste presenteras inom samma

sensoriska modalitet för att det avvikande ljudet ska fånga uppmärksamhet

(Studie II). Vidare undersöktes hur en oväntad spatial förändring (förändring av

presentationskälla från kroppens ena sida till den andra) i en audiotaktil

(bimodal), taktil och auditiv stimuli (unimodal) irrelevant sekvens, påverkar

spatialt och verbalt korttidsminne. Är oväntade förändringar i bimodala stimuli

mer utstickande jämfört med unimodala (Studie III)? Dessutom testade studie

III påståenden att korttidsminnet är domänspecifik (lagras spatial och verbal

information separat).

I linje med tidigare forskning fann denna avhandling att både plötsliga och

oväntade förändringar i auditiva och taktila irrelevanta sekvenser fångade

uppmärksamhet bort från den visuella uppgiften. Men den temporala dynamiken

mellan de två modaliteterna verkar skilja sig. Det verkar som att upprepad

utsättning för plötsliga och oväntade vibrationer gör att den negativa effekten

minskar, medan detta inte är fallet i den auditiva modaliteten. Detta tyder på att

det finns centrala mekanismer (upptäckt av förändringen) och sensoriskt

specifika mekanismer.

Studie II fann att den plötsliga och väntade förändringen måste presenteras i

samma modalitet som det upprepande stimulit. Om det som gör att

uppmärksamheten fångas är en neural modell som förutsäger inkommande

sensorisk information, så byggs den neurala modellen sannolikt av stimuli inom

varje modalitet separat.

Resultaten av Studie III visade att spatialt och verbalt korttidsminne påverkas

negativt av en oväntad spatial förändring, men endast när förändringen sker

inom en bimodal irrelevant sekvens. Dessa resultat är bevis för en enhetlig syn av

korttidsminnet (verbal och spatial information lagras enhetligt) och att bimodala

stimuli kan integreras i en enhetlig percept som gör den plötsliga spatiala

förändringen i den irrelevanta strömmen mer utstickande.

1

Introduction

Imagine that you are sitting at your favorite café at the train station reading a

newspaper. The café is probably an environment packed full of information

coming through the different sensory modalities, such as sight, sound, touch, and

smell. Although the café is filled with a lot of noises in the background (e.g.,

people talking), visual information (e.g., flashing lights from the coffee

machines), vibrations (e.g., someone moving a chair), and more, you can focus on

the task at hand – reading your newspaper. However, when the barista suddenly

and unexpectedly drops a big box of coffee on the floor, your focus is diverted due

to the sound and the vibrations through the floor. This example illustrates the

interplay between the ability to focus attention on the task at hand and, a crucial

aspect for survival, attention capture. On one hand, you can focus on the

newspaper but on the other hand the sounds and vibrations could be a signal that

something dangerous is happening. In the example above, the sudden and

unexpected event was not of importance, but it nonetheless distracted you from

reading your newspaper. This thesis aims to examine the interplay between

focusing of attention and distractibility using a multisensory approach.

Specifically, it examines sudden and unexpected changes, known as deviant or

novel events, in expected to-be-ignored (TBI) streams using a multisensory

perspective. It starts by examining whether deviant sounds and vibrations

capture attention in a functionally similar manner. It then examines whether the

expected events in the TBI stream need to be presented within the same modality

as the deviant event. Finally, it examines whether multisensory models of the

expected sensory stimuli are formed.

Attention

This part defines one of the central cognitive constructs for this thesis, namely

attention. It is a well-used construct in the field of cognitive psychology and there

are many different definitions. One popular definition is that of William James:

“Everyone knows what attention is. It is taking possession of the mind, in clear

and vivid form, of one out of what seems several simultaneously possible objects

or trains of thought. Focalization, concentration of consciousness is of its

essence. It implies a withdrawal from some things to deal effectively with

others.” - William James (1890, pp 403-404)

To navigate through the environment that is full of information from different

sensory modalities, the human brain must create mental representations (i.e.,

neural models; e.g., Sokolov, 1963). To do so, the human brain encodes and filters

the sensory input by through extraction of statistical regularities and attentional

2

filtering. The ability to filter certain stimuli out while obtaining information from

another stimulus is known as selective attention (e.g., Broadbent, 1958; Driver,

2001; Treisman, 1969). Attention can further be divided into exogenous and

endogenous attention. Exogenous attention, also known as stimulus-driven, is

when attentional resources are allocated to something external, such as a salient

event or a sudden change. Endogenous attention, also known as goal-driven, is

when the attentional resources are directed internally such as a goal or directed

behavior (Corbetta & Shulman, 2002). This thesis exmained exogenous attention

by sudden and unexpected changes. Corbetta and Schulman (2002) put forward

two different systems for attention, namely the dorsal and ventral frontoparietal

networks. The dorsal system reflects the goal-directed part of attention (i.e.,

endogenous attention). It thus reflects how cognitive processes such as prior

experiences and current goals can control attention. The ventral attention system,

on the other hand, reflects the stimulus-driven (i.e., exogenous attention) part of

attention. This is the system that is recruited during detection of behaviorally-

relevant sensory information– in particular, when the sensory information is of

relevance, salient and unattended (Corbetta & Schulman, 2002). This thesis

examines exogenous attention by sudden and unexpected changes.

Distraction/orienting

The interplay between attention/focusing of attention and distractibility have

been of interest for a long time. This section will briefly describe some of the early

work that showed that TBI stimuli can affect behavior.

Early studies examining sudden and unexpected changes focused mainly on

audition and used the dichotic listening task developed by Cherry (1953). In this

task, subjects listen to a passage of speech through headphones. They are then

asked to shadow (repeat) the speech presented to one ear and to ignore anything

from the other ear. While participants are shadowing the speech, a message is

presented in the unattended channel. Cherry (1953) found that the unexpected

message presented to the unattended ear disrupted shadowing performance (i.e.,

more errors were produced). A follow-up study by Moray (1959) found that most

of the message presented to the unattended ear is successfully filtered out.

Certain properties of the message in the unattended ear, however, captured the

subject’s attention (i.e., the subjects own name). The effect of presenting the

subject’s own name has been followed up and the results, in line with above

mention studies, showed that presenting the subject’s own name serves as a

distraction from the shadowing task. When the subject’s own name was presented

in the irrelevant channel, one-third of the participants showed increased errors

and response lags to the two words subsequent to the name (Wood & Cowan,

1995).

3

More recent studies have also shown that performance in irrelevant sound

experiments using dichotic measurements may be mediated by individual

differences. For example, Conway, Cowan, & Bunting (2001) followed up the

above-mentioned study with an experiment in which the participant's working

memory capacity was measured. The results revealed that subjects with a low

working memory capacity were more prone to distraction by the irrelevant

message compared to subjects with high working memory capacity. More

recently, it was found that presenting the subject’s own name while doing a visual

task does not affect performance more compared to another name (Ljungberg,

Parmentier, Jones, Marsja, & Neely, 2014). It was suggested that the effect of an

unexpected presentation of the subjects’ own name is only a surprise effect.

Another strand of research of interest to this thesis is relates to how sudden and

unexpected events in an otherwise repetitive stream of stimuli elicits the orient

response (OR). Sokolov (1963) put forward the idea that a repetitive presentation

of the same stimulus built up a mental representation, or neural model, based on

the properties of the stimulus. When a novel, or unexpected, event is presented,

it is compared to the neural model. If this event is salient or of biological

relevance, the OR is elicited (Sokolov, 1963). The OR was defined as a number of

different physiological responses (e.g., slowing of heart rate and increase in skin

conductance; Sokolov, 1963). These responses can, in turn, be accompanied by

attention capture.

Following the ideas of Sokolov (1963), more recent research examined the

underpinnings of the neural model and attention capture using the oddball

paradigm. This strand of work focused for many years on the electrophysiological

responses (ERP) to stimulus deviating from an otherwise repetitive stream of

sounds. The deviating stimulus is known as a deviant or novel stimulus

depending on whether it is the same throughout a task or new at each

presentation. The following paragraphs briefly describe the findings from studies

using auditory-auditory and auditory-visual oddball tasks. Since the research

started off by examining the electrophyisiological markers for the OR, I will start

by briefly describing these markers. After that, I will describe the behavioral

responses that are relevant to the studies. Furthermore, there are other studies

using a unimodal (i.e., using stimuli from only one sensory modality) and a

crossmodal (e.g., to-be-ignored stimuli in one modality and task in another)

method with visual and somatosensory stimuli. These studies will be presented

later in the part examining multisensory aspects.

Electrophysiological responses to deviating stimuli

Early research examining the neural underpinnings of attention capture focused

mainly on deviant stimuli using an auditory-auditory oddball paradigm in which

4

subjects are typically exposed to a stream of repetitive sounds. For instance, in a

seminal study Näätänen, Gaillard, and Mäntysalo (1978) subjects were asked to

perform a dichotic listening task. In this task, subjects were presented with either

a frequent, standard stimulus or an infrequent, deviating, stimulus. Both types of

sounds were presented to either the left or the right ear. The study found that the

deviating stimulus elicited a late negativity in the ERP waveform (i.e., the

mismatch negativity; MMN).

Most oddball studies use a method similar to the study by Näätänen, Gaillard,

and Mäntysalo (1978) whereby the repetitive stream typically consists of two or

more sounds. In most of the trials, the same, standard, stimulus is used. With

sudden and unexpected trials, another, novel or deviant, stimulus is presented.

However, it is worth noting that more recent studies have used methods in which

the deviant stimulus is a deviation in learned perceptual rules (e.g., Schröger,

Bendixen, Trujillo-Barreto, & Roeber, 2007).

In the auditory-visual variant, subjects are typically engaged in a task in the visual

modality (e.g., categorizing visually presented digits as odd or even) while being

exposed to the stream of to-be-ignored (TBI) stimuli. The difference between

most unimodal and crossmodal oddball tasks is that the targets and TBI stimuli

are perceptually decoupled in the crossmodal settings while they are typically part

of the same object in the unimodal settings (see Parmentier, 2014). In most

unimodal studies, the task is to categorize the sound (e.g., as long or short; e.g.,

Berti, 2008). Here, a deviant can be a change of spatial location, or a change in

pitch or frequency of the sound. In the crossmodal variant, however, the TBI

stimuli are typically presented prior to the visual targets (e.g., Schröger & Wolff,

1998).

Research using both unimodal and crossmodal task settings has found that novel

and deviant stimuli elicit a triumvirate of ERPs; MMN/N1, the P3a, and the

reorienting negativity. MMN has been extensively researched and is elicited

whether the subjects are engaged in a task or not (while reading a book; e.g.,

Saarinen, Paavilainen, Schöger, Tervaniemi, & Näätänen, 1992; Schröger, Giard,

& Wolff, 2000). Furthermore, it has been suggested that the MMN is either a

marker for a change-detection mechanism (e.g., the deviant stimulus; Näätänen,

Paavilainen, Rinne, & Alho, 2007) or a marker that the change violated the

predictions of the neural model (e.g., Bendixen, SanMiguel, & Schröger, 2012;

Winkler, 2007). Secondly, it has been suggested that the P3a is a marker for when

attention has been oriented to the change (e.g., Berti, 2008; Friedman, Cycowicz,

& Gaeta, 2001). Finally, if the subjects are engaged in a primary task, the

reorienting negativity (RON) is elicited. This is suggested to be a marker that

attention has been reoriented back to the focal task (e.g., Berti, 2008; Schröger &

Wolff, 1998).

5

Behavioral responses to deviating stimuli

Behaviorally, the introduction of a novel or deviant stimulus has been shown to

prolong response latencies to the focal task. For instance, when subjects are to

categorize sounds by their duration (long or short), responses have been shown

to be prolonged in deviant trials (e.g. Berti, 2008). In the crossmodal settings, the

task is typically categorizing digits as odd or even. Here as well, a prolongation is

typically seen after the introduction of a novel or deviant stimulus (see

Parmentier, 2014 for an extensive review). The TBI stimuli are presented prior to

each target stimuli and a standard stimulus is frequently used (e.g., 80% of trials)

whereas a novel or deviant stimulus is used infrequently (e.g., Andrés,

Parmentier, & Escera, 2006; Escera, Alho, Winkler, & Näätänen, 1998). The

prolongation of response times will henceforth be called deviance distraction. It

has been found that deviance distraction is because the processing of the onset of

the target is delayed (Parmentier, Elford, Escera, Andrés, & SanMiguel, 2008). In

other words, rather than being due to that the processing of the target itself, it

seems that attention has been oriented to the change. Parmentier, Elford, Escera,

Andrés, & SanMiguel (2008) showed this in a study in which they manipulated

the perceptual load in the visual task, increased the number of response choices,

and used a visual cue prior to the visual targets. It was found that the visual cue

recaptured attention to the visual target. The effects of deviant sound have been

reported to be reduced due to practice (Parmentier, 2008; Sörqvist, Nöstl, &

Halin, 2012). For instance, Parmentier (2008) found that the effect of deviant

sounds was smaller at the end of the task compared to the beginning of the task.

Why are sudden changes distracting?

How this attentional shift works has not been fully examined, but Parmentier et

al. (2008) put forward three alternative explanations, namely that deviance

distraction can be due to a spatial shift, modality shift, and a task set shift.

To elaborate, in the crossmodal oddball task the TBI stimuli, whether standard

or deviant, are presented from a different spatial location than the visual targets.

For example, in the audio-visual oddball task, the visual targets are presented in

front of the subjects on the computer screen whereas the TBI stimuli are typically

presented through headphones. When the deviant sound captures attention,

focus shifts from the visual targets to the source of the sounds (i.e., the

headphones). The deviant sound may also shift attention from the task modality

(visual) to the TBI modality (auditory).

However, given that the same mechanisms are underlying deviance distraction in

unimodal and crossmodal settings, the spatial and modality shift cannot fully

explain what is going on. In unimodal oddball tasks, there is only one modality

used (e.g., auditory) and the location of the task and the TBI stimuli are the same.

Thus, the spatial and modality shift can only explain crossmodal task settings.

6

Parmentier et al. (2008) therefore proposed a third possible attentional shift –

task set shifting. According to this account, the time penalty is caused by the

cognitive system trying to find the right response to the deviant or novel stimulus.

This account is based on the assumption that not producing a response is the right

response to a deviant or novel sound. Furthermore, this makes the stimulus-

response mapping distinguishable from cases where the primary tasks lead to

deviant or novel sounds making a person switch task sets (see Parmentier, 2014

for a discussion).

Response latencies are sped up

Depending on the task setting, it has also been found that response latencies can

be sped up by the presentation of deviating stimuli. SanMiguel, Linden, & Escera

(2010) conducted a series of experiments. They found that novel sounds

facilitated responses in terms of both decreased response latencies and increased

accuracy. SanMiguel and colleagues (2010) suggested that this facilitation was

due to a nonspecific alerting signal thought to be part of the orienting response.

Short-term memory

In a closely-related paradigm, attention capture was also studied using a similar

to the oddball paradigm but using a short-term memory task (e.g., the serial recall

task) instead. The main idea is similar to that of the oddball tasks, i.e. subjects are

presented the same auditory TBI stimuli in most of the trials. However, the task

is more mentally taxing and fewer trials are used in the experiments. In the serial

recall task, subjects are typically asked to remember verbal targets (e.g., digits),

called to-be-remembered items (TBR), that are presented one at a time on a

computer screen. The task is to remember the order of the TBR items. For

example, if the digits “7 4 2 1 8 0” are presented in a trial, they are to be

remembered in that precise order.

Before defining and describing short-term memory, it may be worth noting that

the term short-term memory (STM) is often used in reference to the construct of

working memory (WM). A typical STM task is the serial recall task described

previously. In WM tasks, the subjects may be exposed to a set of TBR items.

However, they are also engaged in another task, unrelated to the serial recall task

(e.g., reading sentences). Thus, STM is typically defined as maintenance of TBR

items over a brief time period and WM as maintenance plus manipulation. Note,

however, that a lot of research has focused solely on the maintenance part of WM

(e.g., see a review by D’Esposito, 2007). Other researchers also thoroughly

reviewed and meta-analyzed literature using STM and WM tasks in different

contexts (Unsworth & Engle, 2007). Unsworth and Engle (2007) concluded that

STM and WM reflect the same cognitive process and, therefore, rejected the

notion that they were two different constructs.

7

In this thesis, most literature reviewed use the term STM and simple span tasks

(i.e., the serial recall). Thus, the term STM will be used to describe the

maintenance of information over a brief time period. Note, however, that some of

the models and theories cited use the term WM.

One outstanding issue in relation to short-term memory/working memory relates

to whether different kinds of information re held in different stores or in the same

store. In other words, it has been questioned whether the STM is domain-general

or domain-specific (e.g., Kane et al., 2004; Li, Christ, & Cowan, 2014). Some

researchers argue that information held in the STM is held in different domains

(e.g., visual, auditory, verbal, spatial; Cocchini, Logie, Della Sala, MacPherson, &

Baddeley, 2002), while others acknowledge the existence of domain-specific

stores but argue that information also can be stored in a central, domain-general

manner (e.g., Li et al., 2014; Saults & Cowan, 2007).

Multicomponent

One of the most prominent models of working memory is the multicomponent

view of the working memory that posits that there are several modules

responsible for different aspects of information kept, and manipulated, over brief

periods of time (e.g., see Baddeley, 2012). The components are as follows: the

phonological loop, the visuospatial sketchpad, episodic buffer, and the central

executive (CE). Verbal (linguistic, speech-like) material is thought to be

maintained in the phonological loop, whereas non-verbal material is thought to

be maintained in the visuospatial sketchpad. The episodic buffer, on the other

hand, has been suggested to be a buffer store for processing multidimensional

code or integrated episodes. In addition to these three components, the central

executive guides the behavior related to the maintenance of the two components.

Importantly, the multicomponent view of short-term memory can be viewed as a

domain-specific model of short-term memory.

Embedded-processes model

Another prominent model is the embedded-processes account of short-term

memory (Cowan, 1988, 1995). In this model, the short-term memory is divided

into three hierarchical storages. First, there are subsets of inactivated long-term

memory representations. Second, information can be brought into short-term

memory by a subset of information from the long-term system being temporarily

activated. This activation is limited by time and will decay. Third, a limited focus

of attention can cause further subsets of information to receive even more

activation and be made particularly salient when they fall under the focus of

attention. In this model, a central controller is assumed to provide domain-

general processing capacity. Thus, the embedded processing model predicts that

8

different types of information (verbal, visuospatial) are maintained in a common

storage.

Duplex-mechanism account

The third account, the object-oriented record (OOR), is also a domain-general

model. According to this view, short-term memory representations of verbal,

non-verbal sounds as well as verbal and spatial items are stored within the same

medium (e.g., Jones, Macken, & Murray, 1993). The short-term memory

representations are called objects and are associated with pointers. These

pointers are created through rehearsal and store serial order. More recently, the

OOR has been superseded by the duplex-mechanism account (Hughes, 2014;

Hughes, Vachon, & Jones, 2005).

Distraction and short-term memory

As this thesis deals with distraction by sudden and unexpected changes, the next

section will describe research related to this and then return to the different

short-term memory accounts mentioned in previous sections.

It is well-documented that irrelevant auditory stimuli can affect the ability to

reproduce visually presented sequences of verbal items (e.g., see Hughes, 2014

for a review). This is typically referred to the irrelevant speech effect (Colle &

Welsh, 1976) or the irrelevant sound effect because non-speech sounds can also

produce distraction (Beaman & Jones, 1997) The vast majority of studies (e.g.,

Hughes, Tremblay, & Jones, 2005; Jones et al., 1993; Röer, Bell, Marsh, &

Buchner, 2015; Tremblay, Nicholls, Alford, & Jones, 2000; see Hughes, 2014)

have examined how a changing sound sequence (e.g., "ABABABAB") disrupts

serial recall compared to a repetitive sequence (e.g., "AAAAAAAA"). There are

usually three conditions: the changing-state, the steady-state, and a silence

control condition. The changing-state effect is prominent during both encoding

and retention of the TBR items. Noteworthy, the steady-state sequence does not

typically affect serial recall compared to the silent control condition.

As previously mentioned, there is also an increasing interest in how deviant

sounds affects short-term memory. A similar method is used; the subjects are

exposed to a sequence of TBI sounds while performing the serial-recall task. In

the majority of trials, the TBI sound is the same. Crucially, on some trials there is

a deviant sound presented in the TBI sequence (e.g., at the fourth item). For

instance, in a study by Hughes, Vachon, and Jones, (2005) a deviation in the

inter-stimulus interval between two sounds was shown to have a negative impact

on serial-recall (e.g., fewer items were recalled correctly in deviant trials). It has

further been found that deviants (e.g., changes from car horn sounds to piano

sounds) disrupted performance in verbal but not spatial short-term memory

9

(Lange, 2005). Furthermore, it has also been found that an unexpected and

sudden change from male to female voice have the same negative impact

(Hughes, Vachon, & Jones, 2007). More recently, it has also been found that a

deviation in spatial source of to TBI stream (i.e., from one ear to the other) disrupt

both spatial and verbal short-term memory (Vachon, Labonté, & Marsh, 2017).

Returning to the different models of short-term memory they explain the

changing-state and deviation effects a bit differently. Furthermore, broadly they

can be categorized into either the unitary (e.g., Cowan, 1988, 1995) or the duplex-

mechanism accounts (e.g., Hughes, 2014; Hughes, Hurlstone, Marsh, Vachon, &

Jones, 2013; Hughes, Vachon, & Jones, 2007).

According to the unitary account both the deviation and the changing-state

effects are due to exogenous attentional orienting or attention capture (e.g.,

Cowan, 1995; Lange, 2005). Changing-state sequences are viewed as a succession

of deviants that over, and over, capture attention from TBI items.

The duplex-mechanism account, on the other hand, posit that the deviation effect

is due to attentional capture and the changing-state effect is due to interference-

by-process. According to the duplex-mechanism account changing-state sounds

do not capture attention but that "the preattentive and obligatory processing of

the order of the changing stimuli conflicts with the deliberate serial rehearsal of

the to-be-remembered stimuli" (pp. 1050, Hughes, Vachon, & Jones, 2005).

Violation of predictions and attention capture

In this section I will describe a very recent account of the effect of deviant stimuli.

This account brings together both the studies focusing on response latencies and

short-term memory disruption. Namely, the violation of predictions account of

attention capture.

Recently, researchers have focused on investigating the mechanisms of attention

capture and asked whether deviants capture attention due to that they are novel

or due to that they violate the subjects’ expectations concerning the TBI

sequences. Deviant or novel events are most often presented at a relatively lower

probability compared to standard events and, therefore, it was assumed that

deviants were distracting due to their low base-rate probability. There are,

however, at least two other hypothesis that have been put forward; perceptual

local change and violation of predictions hypothesis (e.g., Nöstl, Marsh, &

Sörqvist, 2012; Parmentier, Elsley, Andrés, & Barceló, 2011). According to

perceptual local change, deviant and novel stimuli capture attention due to that

there is a physical change from the standard to the deviant (e.g., in pitch). The

unexpectedness account, on the other hand, posits that deviant or novel stimuli

is capturing attention due to that they are unexpected. That is, our cognitive

10

system creates feedforward models and is expecting the standard but gets a

deviating stimulus.

Other evidence suggesting that a neural model based on the cognitive systems

prediction on upcoming events comes from research that have found that simply

omitting the standard stimulus elicits the MMN (Horváth, Müller, Weise, &

Schröger, 2010; Yabe et al., 2001; Yabe, Tervaniemi, Reinikainen, & Näätänen,

1997). Yabe and colleagues found that MMN (Yabe et al., 1997) and MMNm (the

MEG equivalence of MMN; Yabe et al., 2001) was elicited when the standard

sound was omitted (e.g., "S-S-S-S- -S-S"). Horváth et al. (2010) reported MMN

for an omission of the sound in a continuous stream of sounds (e.g., "SSSSSSS

SSSS").

Returning to the short-term memory literature and disruption by deviants,

Hughes, Vachon, and Jones (2007) argued that attention is captured because

"foregoing predictions do not hold" (pp. 738, Hughes, Vachon, & Jones, 2005).

The authors put forward an algorithmic model. According to their model,

attention capture is the result of a violation of a pattern or a rule. This account

provides an answer to the fact that a violation of a canonical order (e.g., 1 2 3 4 5

7 8) and deviations from locally constructed rules (e.g., 5 4 3 8 7 7 4 6) have been

shown to capture attention (e.g., Unger, 1964). More recent evidence was

provided by Vachon, Hughes, and Jones (2012). Their study found that a deviant

that was a change from one gender to another only captured attention when the

subjects had to time form a neural model based on expectations of the voice. In

contrast, if the deviant was presented in the beginning of a trial, where the sound

would have been novel, the change could not have been expected. However, when

the subjects started to expect the voice change (across the experiment) the effect

declined. Further, evidence comes from a study by Hughes, Hurlstone, Marsh,

Vachon, and Jones (2013) in which the subjects were warned prior to each trial

that a deviant would be presented.

Violations of prediction may be what underlies deviance distraction, but is this

effect limited to the oddball paradigm? There is a more general theory/framework

that fits quite well with this idea, i.e. the predictive coding framework (e.g.,

Friston, 2010; Quak, London, & Talsma, 2015; Talsma, 2015).

According to the predictive coding framework, there are stochastic models in the

brain that are constantly updated based on processed sensory information.

Higher-order brain areas provide the lower areas with predictions. These

predictions, in turn, have an impact on the processing of the current sensory

information. If the incoming sensory information is a mismatch between what is

predicted and the actual input, the model needs to be updated. For instance, when

unexpected sensory input (like a deviant stimulus) is present, the internal model

11

of the environment may need to be updated to deal with this change in

representations. This continuously updating model has been described as a

multisensory working memory representation that is made up of information

from different modalities (Quak et al., 2015). To clarify, the representations held

in the stochastic models are thought to be amodal (e.g., Talsma, 2015; et al.,

2015).

Interestingly, both the auditory (e.g., Schröger et al., 2014; Winkler & Czigler,

2012) and visual MMN (e.g., Stefanics, Kremláček, & Czigler, 2014; Winkler &

Czigler, 2012) were interpreted within the context of the predictive coding

framework and the MMN is thought to reflect the mismatch between what was

predicted (i.e., the standard stimulus) and the actual stimulus (i.e., the deviant).

Multisensory accounts on attention, distraction, and short-

term memory

Prediction may be something that enables the cognitive system to deal with the

vast amount of sensory processing from the environment. It is also worth

questioning whether the neural model is created based on input from each

sensory modality separately. In this section, I will briefly return to the OR, and

discuss evidence from neuroscience and the oddball paradigm that indicate that

both multisensory aspects (e.g., central mechanisms) and unisensory aspects

(e.g., sensory specific mechanisms) are involved in the detection of unexpected

and sudden changes.

According to Sokolov (1963), the OR is non-specific as regards stimulus quality

because it can be elicited by sound, light, electrical or thermal stimulation of the

skin. Thus, Sokolov (1963) claimed that the OR is modality independent. Sokolov

described the OR as a functional system that becomes inactivated by repeated

exposure to a stimulus and is reactivated by any change in that stimulus.

Furthermore, it has been reported to be elicited by stimuli in one modality

presented after a repetitive stream of stimuli in another modality. Houck &

Mefferd Jr. (1969), and Smith, Dickel, & Deutsch (1978) exposed subjects to a

stream of either standard auditory or standard visual stimuli. After the stream of

standard stimuli, a novel stimulus in the opposing modality was presented. Thus,

these two studies imply that unexpected stimulus in one modality can be detected

even though it is presented in the context of another modality.

More recently, it has been found that certain neural networks are activated by

unexpected changes. Importantly, these networks are activated whether it is a

salient and/or an unexpected change and share underlying neural networks

regardless of which sensory modality the change was presented in. Most relevant

is one network, the ventral attention network, which is suggested to be an alerting

12

system (e.g., Corbetta, Patel, & Shulman, 2008). Furthermore, it is suggested that

the ventral attention network is independent of modality (e.g., Corbetta et al.,

2008; Macaluso, 2010). In a seminal study, Downar, Crawley, Mikulis, & Davis

(2000) used functional magnetic resonance imaging (fMRI) to study which brain

regions respond to deviants in the visual, auditory and tactile modalities. It was

found that visual, tactile, and auditory deviants activated multimodal networks.

More recent evidence comes from a meta-analysis of oddball studies in the

auditory and visual modalities (Kim, 2014). The results showed that regardless

whether the deviant stimuli were visual or auditory, relevant or irrelevant, they

shared common neural networks (i.e., the ventral system). However, there are

three reasons be cautious when generalizing to behavioral deviance distraction in

crossmodal settings using tactile stimuli. First, all results in the aforementioned

studies used unimodal designs (i.e., deviants, standards, and tasks were all

presented within the same modality). Second, no study in the meta-analysis by

Kim (2014) used tactile stimuli. Third, the methodologies of oddball studies

employing fMRI, ERP, as well as behavioral elements differ somewhat. For

instance, the temporal resolution of fMRI is lower than with ERP and behavioral

studies, which leads to slower presentation rates of the stimuli for fMRI (i.e.,

longer interstimulus intervals).

Central or sensory-specific mechanisms?

Of key interest for this thesis is whether there are similar mechanisms underlying

distraction of sudden and unexpected changes. At first glance, if distraction by

deviants both in categorizing and cognitive tasks (e.g., short-term memory) is due

to violations of expectations, one could propose that this distraction should be

independent of modality. Furthermore, and as previously described, research

using the crossmodal oddball paradigm as well as serial recall task clearly indicate

that the TBI stimuli and the task do not have to be in the same modality. This

section will describe research using tactile and visual TBI stimuli.

Deviant or novel presented stimuli in the visual and somatosensory modalities

have been reported to elicit responses similar to those in the auditory modality.

For instance, visual deviants have been found to elicit the MMN and the P3a (e.g.,

Berti, Roeber, & Schröger, 2004; Boll & Berti, 2009), and tactile deviants have

been shown to elicit the P3a (Knight, 1996). Importantly in relation to this thesis,

vibrotactile deviants have also been shown to disrupt performance in visual tasks

(Ljungberg & Parmentier, 2012a; Parmentier, Ljungberg, Elsley, & Lindkvist,

2011). This has led some authors to suggest that there are functional similarities

between the tactile and auditory modalities (e.g., Ljungberg & Parmentier, 2012).

In a study by Ljungberg and Parmentier (2012), subjects were exposed to

vibratory and auditory TBI stimuli while performing a visual categorization task.

In both modalities, deviants prolonged responses in the visual task. Furthermore,

13

Ljungberg & Parmentier (2012) found that the temporal dynamics of deviance

distraction were similar for auditory and vibratory deviants. That is, although the

effect was reduced across the tasks, it did not disappear in either modality.

It is worth noting, however, that there are also some differences between

modalities in terms of deviance distraction. Ljungberg and Parmentier (2012)

reported a significant correlation between auditory and vibratory deviance

distraction concerning accuracy. However, the correlation between response

latencies when comparing the distractive effect of sound and vibration deviants

failed to reach a significant level. Furthermore, Leiva, Parmentier, and Andrés,

(2015) reported a difference between deviance distraction by auditory and visual

deviants. In their study, they used visual-visual, auditory-auditory, auditory-

visual, and visual-auditory oddball tasks. Crucially, the standard and deviant

stimuli were always presented prior to the target in the focal task, whereas in

previous research of visual deviance distraction the standard and deviant stimuli

were part of the same object (e.g., the target). Auditory deviants significantly

affected performance, but the visual deviants did not.

Although the research on how deviations in tactile TBI sequences is sparse, there

is a growing interest in how tactile stimuli, for instance, orient attention to visual

targets or certain spatial locations. In these research paradigms, the focus is on

the facilitation of responses (e.g., operationally defined as faster response

latencies) due to tactile cues. For instance, it has been found that subjects respond

faster when changes in a search display coincide with a tactile cue (e.g., Ngo &

Spence, 2010; Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2009).

Furthermore, from an applied perspective, it has been shown that tactile warning

systems can capture attention and warn for potential upcoming car collisions

(e.g., Ho, Reed, & Spence, 2006; Mohebbi, Gray, & Tan, 2009). Ho, Reed and

Spence (2006) examined whether vibrotactile warnings signals could cue

subjects of potential front-to-rear-end collision. It was found that vibrotactile

cues led to faster breaking responses compared to when there was no cue.

Importantly, vibrotactile distractors have also been found to affect performance

in visual tasks using other experimental paradigms. Using a crossmodal

incongruence task, Spence and Walton, (2005) found that when the vibrotactile

elevation was incongruent with the visual elevation, the subjects responded

slower. In other words, when the visual target (i.e., flash of a LED light) was

presented at a higher location but the vibrotactile target was presented at a lower

location (i.e., on the thumb and not the index finger) the responses were slowed.

Tactile information can, apparently, affect performance in visual tasks.

Altogether, both auditory and tactile stimuli seem to capture attention in a similar

manner. This is true whether the task is disrupted or facilitated.

14

Multisensory neural models?

The fact that there are brain areas activated for changes in the visual, auditory,

and tactile modality (e.g., Corbetta, Patel, & Shulman, 2008; Downar, Crawley,

Mikulis, & Davis, 2000; Macaluso, 2010) as well as the functional similarities

highlighted above (e.g., Ljungberg & Parmentier, 2012) fit well with an idea of a

predictive brain that uses feedforward networks for sensory events (e.g.,

Parmentier et al., 2011; Talsma, 2015; Vachon, Hughes, & Jones, 2012; Winkler

& Czigler, 2012). When a sensory event is a mismatch, it violates the prediction

and attention may be captured (the model has to be updated; e.g., Talsma, 2015).

Such a view transcends sensory distinctions because the computation of

expectation does not have to be specific to any sensory modality. Furthermore,

there are some limitations of the two studies on vibrotactile deviance distraction

(Ljungberg & Parmentier, 2012b; Parmentier, Ljungberg, et al., 2011) that might

make conclusions concerning similarities in the tactile and auditory modality

premature. In the first study, the exposure to the vibrating TBI stimuli was

limited (i.e., for a short period; 3 minutes). Furthermore, in the study by

Ljungberg and Parmentier, (2012), the method used may not have been suitable

for examining the reduction of deviance distraction over time. In their study, the

authors exposed the subjects to auditory and tactile stimuli in blocks of trials.

These blocks were alternated in such a way that, although the total exposure time

for each modality was 12 minutes, any given block lasted only 3 minutes and then

the modality was switched (i.e., three auditory-visual blocks alternated with three

vibrotactile-visual blocks).

Although there has been a lot of work invested in examining the cognitive

underpinnings of attention capture by deviant events, there are still some critical

questions that need to be answered. For one, the evidence of similarities or

differences between the effects in different modalities are few and have some

methodological limitations. Secondly, investigation is required to determine

whether the neural model is built based on sensory information within modalities

or between them.

15

Aims

The overall aim of the empirical work included in this thesis was to investigate

attention capture by sudden and unexpected changes (i.e., deviants) in TBI

stimuli from a multisensory perspective. More specifically, the studies aimed to

explore similarities and differences in how deviant stimuli affect performance

crossmodally, whether the neural model makes predictions based on within-

modality experience or if it is comprised in a multisensory manner (mismatches

in one modality requires the model to be updated).

Specific aims:

Study I:

To examine the possible functional similarities and differences of deviance

distraction by vibrating deviants to the effects in the auditory modality. Are the

temporal dynamics also similar when the subjects are exposed to the TBI

vibrations for a longer and uninterrupted period of time?

Study II:

To test whether deviance distraction can be elicited even when the standard and

deviants are presented in different modalities. Is deviance distraction contingent

on the standard and deviant stimuli being presented within the same modality?

Study III:

Based on multisensory accounts of STM, examine whether a change in spatial

deviants in bimodal TBI sequences is more salient compared to unimodal (i.e.,

vibrating or auditory). Further, to address the claim of domain-specificity of the

STM is addressed; i.e. can a spatial deviant affect both verbal and spatial STM?

16

Materials and methods

This section describes the methodologies used in all three studies included in the

thesis. All experiments included in this thesis used some variation of the

crossmodal oddball paradigm using vibratory and auditory TBI stimuli. All

sounds used in the experiments were presented through headphones (Study I-

III). Vibrations were presented using vibrating handles (Study I-II) or vibrating

motors (Study III).

Subjects

All subjects in the three studies were persons from the city of Umeå. They were

recruited via flyers at Umeå University, an online webpage (Studentkaninen.se),

and through Facebook groups. All subjects reported that they had normal or

corrected-to-normal sight and hearing (or any known somatosensory deficits).

The first study included 20 people (8 females and 12 males) with a mean and SD

age of 25.4. The second study included 30 subjects across 3 experiments. The

mean age was 24.63, 23.82, and 24.61 for experiments 1-3, respectively.

The third study included 50 subjects (mean age 27.2) in Experiment 1, 50 subjects

(mean age 25.9) in Experiment 2, and 50 subjects (mean age 27.1) in Experiment

3.

Instruments/procedure

Materials

Informed consent forms were used to provide brief information on the

overarching aim and procedures, potential benefits and risks of participation,

specification that participation was voluntary, how the results were to be handle,

and contact information for the research. All subjects used Vic Firth sound

attenuating headphones and the experiments were run on computers.

Auditory stimuli

A sinewave tone with a frequency of 600Hz was used in all three studies. In Study

I and III, the sinewave tone was used as a standard TBI stimulus, whereas in

Study II it was used as a deviant TBI stimulus. The duration of the sinewave tone

was 200 ms in Study I and II and 200 ms in Study III. All sounds were generated

with the open-source software Audacity®.

17

Vibrating handles (Study I & II)

A specially-built vibrating device consisting of two handles was used in the first

two studies (see Figure 1). Each handle was made of a transparent plastic tube

with response buttons on top. The vibrations were caused by a motor inside the

tube that spun an eccentric mass on its rotor (See Figure 1).

Figure 1. The specially-built vibrating handles used in Study I & II.

The standard TBI stimulus consisted of a 200 ms vibration with an amplitude

and frequency of 2.3 ms2 (R.M.S), and 33 Hz, respectively. In Study I, the deviant

and standard stimuli consisted of a 200 ms vibration with an amplitude and

frequency of 2.3 m/s2 (R.M.S), and 33 Hz, respectively.

Vibration motors (Study III)

The TBI vibrations were delivered through two brushless coin vibration motors

(Precision Microdrive’s shaftless and brushless vibration motor, Dura VibeTM,

model 910-101, 10 mm, 3 V, 65 mA, 12,500 rpm, 1 g). The vibration motors were

driven by a sinusoidal signal at a frequency and amplitude of 240 Hz and 1.8 g

(peak-to-peak), respectively. Each motor had a surface area (point of contact) of

74.5 mm and reached maximal rotational speed after approximately 52 ms. Each

vibration motor pair was controlled by Arduino UnoTM microcontrollers. The

18

Arduino Uno microcontrollers were programmed using the Arduino IDETM

version 1.66 (www.arduino.cc). See Figure 2 for a photo of one of the vibrating

motors.

Figure 2. The coin-like vibration motors used in Study III.

Crossmodal oddball task (Study I & II)

In this task, subjects had to categorize digits as odd or even while being exposed

to the TBI stimuli. Each trial started with the presentation of the TBI stimuli

shortly followed of the visual target (digits one to eight). In the majority of trials,

the same standard stimulus was used. On sudden and unexpected occasions, the

standard was replaced with a deviant event. The TBI stimuli were presented 100

ms prior to each target stimulus. In Study I, audio-visual and vibrotactile-visual

oddball tasks were blocked, i.e. the subjects were first exposed to either 4 blocks

of auditory-visual or 4 vibratory-visual oddball tasks (counterbalanced across

subjects). The order of the audio-visual and vibrotactile-visual blocks was

counterbalanced across participants (see Figure 3 for a schematic overview of a

typical trial in Study I).

19

Figure 3. A schematic overview of a typical trial in Study I. Depending on the

block modality, each trial started with the presentation of either a vibration or a

sound. In most of the trials, the TBI stimulus (standard) was the same, while on

sudden and unexpected occasions the stimulus was replaced with another

stimulus (deviant).

In Study 2, the proportion of standard and deviant stimuli were the same as in

Study 1, as was the visual task. However, across three experiments the deviant

was either a sinewave tone presented at the same time without the standard

vibration (Experiment 1), a sinewave tone presented at the same time as the

standard vibration (Experiment 2), or an omission of the standard vibration

(Experiment 3). See Figure 4 for a schematic overview of a typical trial for each

experiment.

20

Figure 4. A schematic overview of a typical trial in the crossmodal oddball task

(Study II). Each trial started with the presentation of a fixation cross and a TBI

stimuli. In Experiment 1, it could be either a standard vibration or a deviant

sound. In Experiment 2, it could be either a standard vibration or a standard

vibration and deviant sound. In Experiment 3, it could be either a standard

vibration or the omission of a standard vibration.

Verbal & spatial serial recall task (Study III)

In this task, subjects had to remember the order of items presented visually. Each

trial started with the presentation of the TBI sequence and the first TBR item. In

all three experiments, the TBI sequence consisted of 10 stimuli. Each stimulus

had a duration of 400 ms and the ISI was 450 ms. In Experiment 1, the TBI

stimuli were made up of vibrations and sinewave tones presented simultaneously.

In Experiment 2 and 3, the TBI sequence consisted of only vibrations and sounds,

respectively. A roving standard paradigm was employed (e.g., Lange, 2005), i.e.

the TBI sequence were presented at one side of the body in the majority of trials.

In infrequent trials, the TBI sequence changed side of the body. After this spatial

change, the sequence was presented on this side of the body until the next change.

21

Each TBR item had a duration of 250 ms and was presented on the computer

screen one at a time. The interstimulus interval was 350 ms and subjects were

asked to respond immediately after the last TBI item was presented. In the verbal

task, subjects were presented 7 digits randomly taken from the set 1-9 without

replacement. In the spatial task, the TBR items consisted of dots presented at

locations randomly taken from a 5x5 matrix without replacement. Both tasks

required the subjects to click on the TBR items in the order they were presented.

The verbal TBI items were presented in canonical order and the dots were

presented in the locations in the 5x5 matrix. After a TBR item was clicked, it

changed color from black to green. See Figure 5 for a schematic overview of a

typical trial in both tasks.

Figure 5. A schematic overview of a typical trial in both the spatial and verbal

serial recall task. Each trial started with the presentation of the TBR items and

TBI stimulus. In both tasks, 7 TBR and 10 TBI items were presented. After the

last TBI item was presented, the subjects were to give their answers.

Data analysis

Study I and Study II

Response latencies and the proportion of correct responses (accuracy) were

analyzed. Response latencies were aggregated for correct responses only and

response latencies greater than 200 ms were excluded. Response latencies and

22

the proportion of correct responses were analyzed using repeated measures

ANOVAs.

In Study I, a 2 (modality: auditory, tactile) x 2 (trial type: standard, deviant) x 4

(block: 1, 2, 3, 4) ANOVA was used to explore any differences or similarities in

terms of habituation between the modalities.

In Study II, the response latencies and the proportion of correct responses were

analyzed separately for each experiment using a one-way ANOVA with the levels

of trial type (standard, deviant). In a follow-up analysis using mixed, within and

between, ANOVAs were used to explore any interactions across deviant types

across the experiments.

Study III

The proportion of correctly-recalled items was scored according to a strict

criterion whereby an item it had to be recalled in the correct position (i.e., in the

exact order the items, whether digits or dots, where presented) in order to be

scored as correctly recalled. Each subject’s score was aggregated across trials in

each condition and task. The data was then analyzed by the means of a 2 (task:

verbal, spatial) x 2 (trial type: standard, deviant) ANOVA.

Data presentation

In all experiments presented in this thesis, the mean data, whether response

latencies or proportions, was presented in tables or figures. The confidence

intervals in both types of data presentation techniques were calculated for within-

subject comparisons according to Cousineau (2005) and Morey (2008).

23

Results

Study I:

Aim

The aim was to expand and replicate previous research examining possible

similarities and differences (Ljungberg & Parmentier, 2012b; Parmentier,

Ljungberg, et al., 2011) between deviance distraction in the tactile and auditory

modality. More specifically, the aim was to examine the temporal dynamics of

deviance distraction in the two modalities after uninterrupted, repeated exposure

within each modality separately.

Based on previous research, it was hypothesized that a) both auditory and

vibratory deviants would prolong response latencies, and b) that the temporal

dynamics of deviance distraction would be similar for both auditory and vibratory

deviants.

Results

The subjects performed quite well (M = .87, SD = .34). Subjects responded slower

in the visual task following the presentation of a deviant stimuli. Importantly, this

was true regardless of the modality of the TBI stimuli. Thus, both auditory and

vibratory deviants elicited deviance distraction. Interestingly, the effect of the

auditory deviant was more pronounced compared to that of the vibratory deviant.

As can be seen in Figure 6, there was a difference between the modalities in terms

of the temporal characteristics of deviance distraction. Whereas the effect of the

vibratory deviant was reduced at the end of the task, the same pattern was not

found for the auditory deviant.

24

Figure 6. Results from Study I: Deviance distraction (i.e., the response latencies

in the deviant trials subtracted from the response latencies in the standard trials).

Error bars represent 95% confidence intervals.

To summarize, the results replicated previous research findings (Ljungberg &

Parmentier, 2012b; Parmentier, Ljungberg, et al., 2011). That is, a deviation in

vibrating TBI stimuli disrupts performance in visual tasks – attention has been

captured. However, it also implies that there are more differences between

deviance distraction by auditory and tactile deviants. That is, the effect of the

vibrating deviations reduced over time whereas the effect of auditory deviations

remained relatively stable. The results do not provide compelling evidence for

solely a central mechanism.

25

Study II:

Aim

The aim was to test whether deviance distraction can be elicited even when the

standard and deviants are presented in different modalities. Is deviance

distraction contingent on the standard and deviant stimuli being presented

within the same modality?

Results

Performance across all three experiments was good (see Table 1 for mean and

95% confidence interval).

Table 1. Mean and 95% CI (within brackets) accuracy (proportion of correct

responses) for all three experiments.

Standard Deviant

Experiment 1 0.891 [0.883, 0.899] 0.907 [0.899, 0.915]

Experiment 2 0.9 [0.893, 0.907] 0.901 [0.894, 0.908]

Experiment 3 0.868 [0.857, 0.879] 0.878 [0.867, 0.889]

A neural model scanning the environment for sensory information and updating

sudden and unexpected changes should detect an auditory change even when the

predicted input is a vibration, at least if the model is based on sensory input from

all sensory modalities stimulated. Overall, the results from the experiments of

Study II are not in line with a solely multisensory view of the violation of

prediction. They are further not in line with the predictive coding framework. As

can be seen in Figure 7, the response latencies were slowed in Experiment 1.

Although this may have indicated that the deviant sound captured attention away

from the visual task, the results from Experiment 2 and 3 can be used for another

interpretation.

In Experiment 1, the standard vibration was omitted when the deviant sound was

presented. When the deviant was presented at the same time as the standard

vibration, a different pattern emerged. As can be seen in Figure 7 (middle panel),

the response latencies for deviant and standard trials were very similar in

Experiment 2. The results from Experiment 1 could be explained by the omission

of the vibration, not the deviant sound per se, being what captured attention.

Experiment 3 showed that a simple omission of the standard vibration could also

affect behavior – the response latencies are prolonged.

26

Figure 7. Response latencies in the standard and deviant trial types. Error bars

represent 95% confidence intervals. *p < .00.

Conclusion

The results of Study II are not in line with a view that representations and

predictions of incoming sensory stimulation are constructed in a modality

general manner (e.g., a neural model encompassing information from all

modalities). Deviance distraction may be contingent on the deviant and standard

stimuli being presented within the same modality. Finally, attention can be

captured by a mere omission of a standard stimulus.

27

Study III:

Aim

The aim was to further explore the effects if deviant vibrations in a more complex

task. Specifically, the aim was to examine whether spatial deviants (changes from

one side of the body to the other), bimodal (vibration and sounds), vibrating, and

auditory TBI stimuli can affect short-term memory performance.

Results

As can be seen in Figure 8, the results from the first experiment showed that

performance dropped when the bimodal stream changed from one side of the

body to the other (i.e., the spatial deviant). This implies that a deviation that is a

change in the location of the TBI sequence affects both verbal and spatial short-

term memory performance. Research examining attention capture in relation to

short-term memory is sparse. The first study showed that auditory deviations

(e.g., change from one sound to another) only affected verbal short-term memory

(Lange, 2005). More recently, it has been shown that both verbal and spatial

short-term memory can be disrupted by deviations in the spatial location

(Vachon, Labonté, & Marsh, 2017). Experiment 1 and the methodology from

Vachon, Labonté, and Marsh (2017) are more similar compared to Lange (2005).

28

Figure 8. Results from Experiment 1: mean proportion of items correctly recalled

in trials with and without a spatial deviant. The left panel depicts the proportion

of correctly recalled items in the spatial task and the right panel depicts the

proportion of correctly recalled items in the verbal task. Error bars represent 95%

confidence intervals.

As can be seen in Figure 9 (left panel), the results from Experiment 2 failed to

find an effect when the TBI sequence consisted of only vibrations. This means it

is possible that it was spatial change the auditory TBI sequence that had an

impact on performance, as already shown by Vachon, Labonté, and Marsh (2017).

However, the results from Experiment 3 (see Figure 9, right panel), show that

using only spatial deviations in the auditory TBI sequence was also ineffective in

terms of disrupting STM performance.

Figure 9. Results from Experiment 2 (left panel) and Experiment 3 (right panel):

mean proportion of correctly recalled items with and without a spatial deviant in

the spatial and verbal STM tasks. Error bars represent 95% confidence intervals.

Conclusion

The results from Study III provide evidence against a domain-general view of

short-term memory. The spatial deviant captured attention and thereby recall

were disrupted in both the verbal and the spatial task. Bimodal TBI sequences

make the spatial change more salient and thus are more effective.

29

Discussion

The aim of this thesis was to examine the interplay between focusing of attention

and distractibility using a multisensory approach. Study I found that there are

more differences than previously reported in relation to how deviant vibrations

and deviant sounds affect visual task processing. The results indicated that

although deviance distraction is found in both modalities, the effect of vibration

deviants may be reduced due to practice time whereas sounds are more resilient.

Study II implicated deviance distraction may be contingent on the changes in the

TBI stream being presented within the same modalities. It was found that a mere

omission of a standard vibration captured attention, while the simultaneous

presentation of a deviant sound and a standard vibration did not affect

performance. Finally, in the third study, a spatial change in a bimodal TBI

sequence had a negative impact on both verbal and spatial serial recall. However,

the spatial change in only auditory or vibratory TBI sequences did not to affect

serial recall.

Central or specific mechanisms?

The assumption that there is a central mechanism underlying the detection of

sudden and unexpected changes (e.g., deviants) is not fully supported by the

results of this thesis – at least in a strict sense. The fact that both auditory and

vibratory deviants prolong response latencies suggests that the detection

mechanism may be functionally similar. However, the results from Study I

suggest that the temporal dynamics may be different between the auditory and

tactile modality. This may, in turn, suggest that there are even more sensory-

specific mechanisms than previously suggested (Ljungberg & Parmentier, 2012).

It may further support the hybrid model proposed by Ljungberg and Parmentier

(2012). According to this view, there are both modality-specific and multimodal

mechanisms. These two mechanisms influence behavior differently.

The central mechanism may be that the cognitive system does in fact detect

irregularities in the environment, i.e. there is an amodal interpretation of the

violation of expectation account (e.g., Nöstl et al., 2012; Parmentier, Elsley,

Andrés, & Barceló, 2011). Both tactile and auditory deviants capture attention

because they violate predictions of the cognitive system. Further evidence for this

proposal comes from studies reporting deviance distraction in auditory (e.g.,

Berti, 2008), visual (when target and deviants are presented simultaneously and

are part of the same objects; e.g., Bendixen et al., 2010; Berti & Schröger, 2004;

Boll & Berti, 2009; but see Leiva, Parmentier, & Andrés, 2015 for absence of

distraction by visual deviants) and tactile (Ljungberg & Parmentier, 2012;

Parmentier et al., 2011) modalities. In line with previous research comparing the

effects of auditory and vibratory deviants (Ljungberg & Parmentier, 2012), Study

30

I showed larger effects of deviants in auditory modality compared to tactile

modality (39.69 ms vs. 20.7 ms). This may also suggest that the auditory system

is more fine-tuned to detect sudden changes (see also Leiva et al., 2015 that

suggested that auditory deviants are more potent distractors compared to visual

deviants).

Study I found no evidence of deviance distraction reduction over time with

auditory deviants, while deviance distraction by vibrating deviants was reduced.

If auditory deviants are, in fact, more potent distractors compared to vibrotactile

deviants (e.g., Leiva et al., 2015), it may mean that more time is needed to show

effects of practice for auditory deviants compared to vibratory deviants. Previous

research using auditory TBI stimuli used slightly different methods (Parmentier,

2008; Sörqvist et al., 2012). In both previous studies, the subjects were exposed

to the TBI stimuli for a longer period of time. However, Ljungberg and

Parmentier (2012) found a significant reduction in deviance distraction using the

same total number of trials as in Study I. One explanation could, thus, be that

significant reductions in deviance distraction may start appearing around 500

trials. It might be worth acknowledging that the results in the tactile modality

could be at the sensory receptor level rather than the cognitive level (i.e., due to

practice). The experiment of Study I was not designed to test whether it is a

matter of sensory adaptation or practice effect. However, whether it is due to

adaptation or practice, it suggests that there are sensory-specific mechanisms

underlying deviance distraction. Whether the adaptation occurs on a more local

(i.e., receptors in the hands) or central (e.g., at the level of the brain) level might

have implications for an amodal violation of expectation account. Adaptation on

the level of local receptors would mean that the signals are not sent to the

cognitive system, and deviant vibrations might stop being unexpected due to that.

Evidence contradicting a full central mechanism has further been reported by

Leiva et al. (2015). In their study, they compared behavioral deviance and post-

deviance distraction in the auditory and visual modalities. Visual deviance

distraction was only found when the subjects directed their attention voluntarily

to the TBI images. Auditory deviance distraction, on the other hand, was found

regardless of whether the subjects directed their attention to or ignored the TBI

sounds.

Implications for theories on attention/distraction/prediction

The results from Study II are not in line with an amodal view like that of the

violation of predictions account (e.g., a neural model encompassing information

from all modalities; Nöstl et al., 2012; Parmentier, Elsley, et al., 2011) or the

predictive coding framework (e.g., Friston, 2010; Quak, London, & Talsma, 2015;

Talsma, 2015).

31

Concerning attention and exogenous attention in particular: Do the results mean

that the auditory system is unable to detect sounds presented in silence? If there

is some danger in the surroundings, may it be an oncoming car signaling with the

horn in the silent night, and it were to go undetected, it could have horrible

consequences. There is research, however, that suggests that sounds presented in

silence are detected. In a study by Berti (2013), the crossmodal oddball paradigm

was used with two conditions. In the first condition, he presented a transient

distractor sound that was not embedded by the traditional repetitive standard

sounds. Berti (2013) reported no evidence of behavioral deviance distraction, but

rather that the transient sound elicited the ERP N1 followed by the P3a and the

RON. In a follow-up experiment, Berti (2013) used novel sounds as the transient

distractor. In this experiment, the transient novels prolonged response latencies

and elicited the MMN, the P3a, and the RON. These findings lead Berti (2013) to

propose that there is a mechanism for detecting sounds in an otherwise silent

context. He further argued that the sound probably needs to be a salient and

strong sound (e.g., an environmental sound) to affect behavior.

Implications for theories on short-term memory

The results of Study III put forward some problems for the multicomponent view

of short-term memory (e.g., Baddeley, 2012; Lange, 2005). If spatial and verbal

information are maintained in different storages, a spatial change should only

affect spatial serial recall. Study III found the contrary, i.e. when the TBI sequence

consisted of both auditory and tactile stimuli a sudden change from one side of

the body to the other affected performance negatively. These results could be

taken as evidence of both the OOR/duplex-mechanism and embedded-processes

accounts (e.g., Cowan, 1995; Hughes, 2014). Both of these unitary views predict

that a deviant stimulus, regardless of domain or sensory modality, will interfere

and have negative impact on short-term memory performance due to attention

capture.

It must be pointed out that the results from Study III could, of course, be

explained by the fact that information in the episodic buffer is linked together

(e.g., spatial and verbal). It may, however, be more problematic that Study III

further found that unisensory streams (tactile or auditory) did not capture

attention. These results are somewhat problematic for both the multicomponent

view (if the interference takes place in the episodic buffer) and the unitary view

of short-term memory. In both views, the spatial deviant should capture

attention, regardless of whether it happens within an auditory or a tactile TBI

sequence. Returning to the results from Study III, it may be that the TBI

sequences, when unimodal, were not salient enough and that the spatial deviant,

32

thus, was ineffective. In fact, research in other areas suggests that stimuli

presented in two or more sensory modalities at the same time may be integrated

into one percept (e.g., see Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010).

This percept, in turn, may be more salient compared to either of the two stimuli

presented alone (unisensory). In fact, there is research indicating that the

simultaneous presentation of short-term memoranda is advantageous compared

to unimodal presentation. A comparatively recent study (Botta et al., 2011)

examined the effects of visual, auditory, and audiovisual cues on short-term

memory performance. The auditory cues consisted of a pure tone, the visual cue

was a black outlined square, and the audiovisual was both previously described

cues presented simultaneously. Furthermore, the cues could either be congruent

or incongruent. In the congruent condition, attention was captured towards the

spatial location that contained the TBR items, whereas in the incongruent

condition attention was captured towards the spatial location opposite to the TBR

items. Botta et al. (2011) found that response accuracy increased when the cue

was congruent and decreased when the audiovisual cue was incongruent.

Crucially, both congruent and incongruent cues had a larger effect when they

were bimodal compared to visual cues only.

Practical implications

Although this thesis focused on research questions related to basic research, it

may be worth briefly discussing practical implications of the results. As

mentioned in the introduction, the main focus in relation to tactile attention

capture has been on how it facilitates performance in more applied settings like

collisions (e.g., warning signals for potential collisions; Ho, Reed, & Spence,

2006; Mohebbi, Gray, & Tan, 2009). Even though the focus here is on the

negative impact of unexpected changes in auditory, vibratory, and bimodal

stimuli, the context of driving a car may also be used. To elaborate, when driving

a car one endogenous attention may be occupied with focusing on the road ahead.

An in-car-warning system could warn the driver another car is in the blind spot,

for instance, by vibrating on the left shoulder. Instead of changing to the left lane,

attention may be captured towards the blind spot and the driver may continue

driving in the right lane. Here, sudden vibration was negative for the task

(changing lanes) but effective in terms of alerting for the other car. In situations

like this, reactions in milliseconds can be of importance.

Limitations

The empirical work included in this thesis has several limitations that should be

addressed in future research. All studies reported suffer from some general

methodological issues. The most important one is that of the TBI stimuli chosen

from the different modalities. In relation to the auditory deviant chosen in Study

I (white noise), it has been thoroughly used in many auditory-visual oddball

33

studies (e.g., Ljungberg & Parmentier, 2012; Parmentier, 2016). It can, however,

be discussed whether the white noise used elicits a startle response or if it is

matter of a deviation in the prediction made. It can be both, of course, but it must

be acknowledged that white noise has been reported to elicit the startle response

(e.g., Dawson, Hazlett, Filion, Nuechterlein, & Schell, 1993). The startle response

is an unconscious response and is associated with negative effect. This, in turn,

makes it difficult to draw firm conclusions concerning any similarities or

differences, particularly, if the deviant vibration did not induce a startle response.

Another aspect (also related to Study I and the stimuli used) is that the change

between a sinewave tone and a white noise may be experienced as

phenomenologically “larger” compared to a change between vibrations differing

in frequency and amplitude (e.g., many subjects described the vibrations used in

this thesis as a “slower” and a “faster” vibration).

There are similar concerns in Study II. Here, the change may not have been

salient enough to be deemed, by the cognitive system, as of biological relevance.

This is especially true when the standard vibration and deviant sound (i.e., the

sinewave tone) was presented at the same time. It has been found that if stimuli

from two sensory modalities are presented at the same time they will be

integrated into one percept. In the case of Study II, this may have been enough to

be a mismatch of the neural model. However, if one of the stimuli is more salient

than the other, that stimuli will capture attention. Given that the vibration was

tightly coupled with the visual target, it may have been more biologically relevant

for the subjects and, thus, the effect was overridden. Using another stimulus that

is more salient may have had a different effect. It must be pointed out here that a

pilot study found no significant difference between how subjects perceived the

vibrations and sounds used in Study I in terms of how attention capturing they

were.

Another limitation of the first two studies in this thesis is statistical power. The

first study used as sample of 20 subjects, while the second used a sample of 30

subjects across the three experiments. This, of course, reduced the chance of

detecting a true effect and increased the risk of overestimating the magnitude of

the effect.

An attempt was made to minimize these concerns in Study III. Here, the sudden

and unexpected change was a matter of location (spatial deviant) rather than a

change in the sound’s character. In addition, it used sounds that contained no or

minimal semantic properties (i.e., sinewave tones). Moreover, the deviation was

spatial in both modalities. Although larger sample sizes were used, and the

deviants were more similar in Study III, some limitations must be acknowledged.

The performance in the verbal task was better compared to the spatial task. In the

oddball-like paradigms, where the deviant is presented in far fewer trials

34

compared to the standard, there are fewer trials in the spatial task and the

measure is less reliable. Furthermore, it is hard to know whether the spatial

deviant in both modalities is perceived, or functionally similar, even though it is

a matter of a change in spatial location. Does the subject perceive a spatial change

in an auditory TBI sequence as distracting as the same change in a vibratory TBI

sequence?

Future research should, thus, attempt to equate the crossmodal deviant stimuli

of important psychophysical characteristics to help isolate the mechanisms

behind any distraction effects. Before examining similarities and differences

between deviations in two or more modalities, future research should carefully

test both the standard and the deviants used. For instance, psychophysical

methods such as crossmodal matching could be used to match the stimuli in

terms of intensity or duration. It may be specifically important to match the

deviant stimuli used in the different modalities (e.g., so that the perceived change

from the standard to the deviant is approximately the same).

Conclusions

• One central mechanism underlying attention capture may be to detect

irregularities in the environment.

• In terms of temporal dynamics, detection of sudden and unexpected

changes in auditory and tactile TBI sequences may not be similar.

• The deviant and standard stimuli in the TBI sequence need to be

presented within the same modality. That is, the results from this thesis

do not support the idea that the cognitive system builds a neural model

predicting regular sensory events from many sensory modalities at the

same time.

• A mere omission of an expected vibration can capture attention.

• Sudden and unexpected changes in terms of spatial location of TBI

sequences affect both verbal and spatial serial recall when the sequence

consists of both vibrations and sounds.

• Unisensory TBI sequences, either vibratory or auditory, are not salient

enough to disrupt short-term memory performance.

35

Acknowledgement

Being a Ph.D. student is one of the most rewarding and important projects I've

experienced in my life ... I have had the good fortune to have a good team of

supervisors who made me move on. These tutors clearly deserve recognition for

this.

First and foremost, I would like to take this opportunity to thank my main

supervisor, Jessica K-Ljungberg, whom I met as a Bachelor student. I'm forever

grateful that you saw and still see, the potential in me. Without you, I would never

have gotten the opportunity to become a Ph.d. When I got stupid criticism, you've

reminded me not to take it personally, but you've also involved me in many

projects besides the dissertation project, where I got to my top skills and interests.

I have learned a lot from you throughout these years.

Second, I would like to give my assistant supervisor Greg Neely, who has also

taught me a lot during my time as a Ph.D. student. Sitting on a meeting with you,

without really having any questions, often meant learn from Greg's great

knowledge of how the university world works. Apart from a lot of knowledge

about the scientific part, this has also been invaluable.

In the last years, I also came to include an additional supervisor; Patrik Hansson.

Often knocking on my door, and then sitting in my foothold, he came up with bold

and challenging questions about my studies, but also methodological questions

regarding his own studies. Towards the end when I wanted to give up, Patrik often

knocked on my door and pushed me further. It was needed.

All three supervisors have complemented each other with different types of

knowledge and interests. Method, career, and general knowledge of how the

university world works is probably something that few Ph.D. students get, all in

one package, from their supervisors.

There are clearly a lot of other people who have supported and stimulated me

through these years. The first is my father. You have raised me to a skeptic who

thinks critically but also stimulated my curiosity. The discussions we have had

about the bad pages of the peer review process have, during the periods it felt,

made it much easier. It also fell out well!

I also want to thank those who, at an early stage, took the time to read and review

a draft of this thesis. Peter Bengtsson, Gustaf Wadenholt, and Linnea Karlsson

Wirebring, of your comments and questions, many of them have applied to my

dissertation. Thank you for taking the time to read my work.

36

I was also part of a group of doctoral students in the department of psychology. I

would like to thank everyone that is, or has been, part of this group, but there are

also some worth mentioning by name. Erik and Andreas, a lot of methods and

stats discussion over a beer or two. Gustaf, who came in later, also discussing

stats, methods, and cognitive science in general. All three always posed tricky

questions stimulating my mind. Markus, we shared room almost the 2 first years.

My desk has never been cleaner, and we always had fun! Robert and Hanna, the

roomies that always listened to me when needed to chat (not necessarily both at

the same time but…)

There are of course some other people that I need to acknowledge. Lars, always

positive and never late to join for some good food and beverages.

Last, but not least, I want to thank the rest of my family. Angelica, my girlfriend,

for being such an understanding and supporting person. I could not have done

this without you! My mom, for coming down taking care of us when we’ve been

swamped with work. Invaluable. My brother, Olof, for sharing your art with me.

One of your works ended up at the front page!

Erik Marsja, 2017-11-08

37

References

Andrés, P., Parmentier, F. B. R., & Escera, C. (2006). The effect of age on involuntary capture of attention by irrelevant sounds: A test of the frontal hypothesis of aging. Neuropsychologia, 44(12), 2564–2568. http://doi.org/10.1016/j.neuropsychologia.2006.05.005

Baddeley, A. D. (2012). Working Memory: Theories, Models, and Controversies. Annual Review of Psychology, 63(1), 1–29. http://doi.org/10.1146/annurev-psych-120710-100422

Beaman, C. P., & Jones, D. M. (1997). Role of serial order in the irrelevant speech effect: Tests of the changing-state hypothesis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(2), 459–471. http://doi.org/10.1037/0278-7393.23.2.459

Bendixen, A., Grimm, S., Deouell, L. Y., Wetzel, N., Mädebach, A., & Schröger, E. (2010). The time-course of auditory and visual distraction effects in a new crossmodal paradigm. Neuropsychologia, 48(7), 2130–2139. http://doi.org/10.1016/j.neuropsychologia.2010.04.004

Bendixen, A., SanMiguel, I., & Schröger, E. (2012). Early electrophysiological indicators for predictive processing in audition: A review. International Journal of Psychophysiology, 83(2), 120–131. http://doi.org/10.1016/j.ijpsycho.2011.08.003

Berti, S. (2008). Cognitive control after distraction: Event-related brain potentials (ERPs) dissociate between different processes of attentional allocation. Psychophysiology, 45(4), 608–620. http://doi.org/10.1111/j.1469-8986.2008.00660.x

Berti, S. (2013). The role of auditory transient and deviance processing in distraction of task performance: a combined behavioral and event-related brain potential study. Frontiers in Human Neuroscience, 7(July), 352. http://doi.org/10.3389/fnhum.2013.00352

Berti, S., Roeber, U., & Schröger, E. (2004). Bottom-up influences on working memory: Behavioral and electrophysiological distraction varies with distractor strength. Experimental Psychology, 51(4), 249–257. http://doi.org/10.1027/1618-3169.51.4.249

Berti, S., & Schröger, E. (2004). Distraction effects in vision: behavioral and event-related potential indices. Neuroreport, 15(4), 665–669. http://doi.org/10.1097/00001756-200403220-00018

Boll, S., & Berti, S. (2009). Distraction of task-relevant information processing by irrelevant changes in auditory, visual, and bimodal stimulus features: A

38

behavioral and event-related potential study. Psychophysiology, 46(3), 645–654. http://doi.org/10.1111/j.1469-8986.2009.00803.x

Botta, F., Santangelo, V., Raffone, A., Sanabria, D., Lupiáñez, J., & Belardinelli, M. O. (2011). Multisensory integration affects visuo-spatial working memory. Journal of Experimental Psychology. Human Perception and Performance, 37(4), 1099–1109. http://doi.org/10.1037/a0023513

Broadbent, D. E. (1958). Perception and communication. Elmsford, NY, US: Pergamon Press. http://doi.org/10.1037/10037-000

Cherry, E. C. (1953). Some experiments on The recognition of Speach, with one and with Two Ears. The Journal of the Acoustical Society of America, 25(5), 975–979.

Cocchini, G., Logie, R. H., Della Sala, S., MacPherson, S. E., & Baddeley, A. D. (2002). Concurrent performance of two memory tasks: evidence for domain-specific working memory systems. Memory & Cognition, 30(7), 1086–1095. http://doi.org/10.3758/BF03194326

Colle, H. A., & Welsh, A. (1976). Acoustic masking in primary memory. Journal of Verbal Learning and Verbal Behavior, 15(1), 17–31.

Conway, A. R. a., Cowan, N., & Bunting, M. F. (2001). The cocktail party phenomenon revisited: the importance of working memory capacity. Psychonomic Bulletin & Review, 8(2), 331–5. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11495122

Corbetta, M., Patel, G., & Shulman, G. L. (2008). The reorienting system of the human brain: from environment to theory of mind. Neuron, 58(3), 306–24. http://doi.org/10.1016/j.neuron.2008.04.017

Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews. Neuroscience, 3(3), 201–15. http://doi.org/10.1038/nrn755

Cousineau, D. (2005). Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson’s method. Tutorials in Quantitative Methods for Psychology, 1(1), 42–45. http://doi.org/no DOI found

Cowan, N. (1988). Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. Psychological Bulletin, 104(2), 163–191. http://doi.org/10.1037/0033-2909.104.2.163

Cowan, N. (1995). Attention and memory: an integrated framework. Oxford Psychology.

39

D’Esposito, M. (2007). From cognitive to neural models of working memory. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 761–772. http://doi.org/10.1098/rstb.2007.2086

Dawson, M. E., Hazlett, E. a, Filion, D. L., Nuechterlein, K. H., & Schell, A. M. (1993). Attention and schizophrenia: impaired modulation of the startle reflex. Journal of Abnormal Psychology, 102(4), 633–641. http://doi.org/10.1037/0021-843X.102.4.633

Downar, J., Crawley, a P., Mikulis, D. J., & Davis, K. D. (2000). A multimodal cortical network for the detection of changes in the sensory environment. Nature Neuroscience, 3(3), 277–83. http://doi.org/10.1038/72991

Driver, J. (2001). A selective review of selective attention research from the past century. British Journal of Psychology (London, England  : 1953), 92(Pt 1), 53–78. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11256770

Escera, C., Alho, K., Winkler, I., & Näätänen, R. (1998). Neural mechanisms of involuntary attention to acoustic novelty and change. Journal of Cognitive Neuroscience, 10(5), 590–604. http://doi.org/10.1162/089892998562997

Friedman, D., Cycowicz, Y. M., & Gaeta, H. (2001). The novelty P3: An event-related brain potential (ERP) sign of the brain’s evaluation of novelty. Neuroscience and Biobehavioral Reviews, 25(4), 355–373. http://doi.org/10.1016/S0149-7634(01)00019-7

Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews. Neuroscience, 11(2), 127–138. http://doi.org/10.1038/nrn2787

Ho, C., Reed, N., & Spence, C. J. (2006). Assessing the effectiveness of “intuitive” vibrotactile warning signals in preventing front-to-rear-end collisions in a driving simulator. Accident; Analysis and Prevention, 38(5), 988–96. http://doi.org/10.1016/j.aap.2006.04.002

Horváth, J., Müller, D., Weise, A., & Schröger, E. (2010). Omission mismatch negativity builds up late. Neuroreport, 21(7), 537–541. http://doi.org/10.1097/WNR.0b013e3283398094

Houck, R. L., & Mefferd Jr., R. B. (1969). Generalization of GSR habituation to mild intramodal stimuli. Psychophysiology, 6, 202–206.

Hughes, R. W. (2014). Auditory distraction: A duplex-mechanism account. PsyCh Journal, 3(1), 30–41. http://doi.org/10.1002/pchj.44

Hughes, R. W., Hurlstone, M. J., Marsh, J. E., Vachon, F., & Jones, D. M. (2013). Cognitive control of auditory distraction: impact of task difficulty,

40

foreknowledge, and working memory capacity supports duplex-mechanism account. Journal of Experimental Psychology. Human Perception and Performance, 39(2), 539–553. http://doi.org/10.1037/a0029064

Hughes, R. W., Tremblay, S., & Jones, D. M. (2005). Disruption by speech of serial short-term memory: the role of changing-state vowels. Psychonomic Bulletin & Review, 12(5), 886–90. http://doi.org/10.3758/BF03196781

Hughes, R. W., Vachon, F., & Jones, D. M. (2005). Auditory Attentional Capture During Serial Recall: Violations at Encoding of an Algorithm-Based Neural Model? Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(4), 736–749. http://doi.org/10.1037/0278-7393.31.4.736

Hughes, R. W., Vachon, F., & Jones, D. M. (2007). Disruption of short-term memory by changing and deviant sounds: Support for a duplex-mechanism account of auditory distraction. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(6), 1050–1061. http://doi.org/10.1037/0278-7393.33.6.1050

James, W. (1890). The Principles of Psychology. H. Holt. Retrieved from https://books.google.se/books?id=JLcAAAAAMAAJ

Jones, D. M., Macken, W. J., & Murray, a C. (1993). Disruption of visual short-term memory by changing-state auditory stimuli: the role of segmentation. Memory & Cognition, 21(3), 318–328. http://doi.org/10.3758/BF03208264

Kane, M. J., Hambrick, D. Z., Tuholski, S. W., Wilhelm, O., Payne, T. W., & Engle, R. W. (2004). The generality of working memory capacity: a latent-variable approach to verbal and visuospatial memory span and reasoning. Journal of Experimental Psychology. General, 133(2), 189–217. http://doi.org/10.1037/0096-3445.133.2.189

Kim, H. (2014). Involvement of the dorsal and ventral attention networks in oddball stimulus processing: A meta-analysis. Human Brain Mapping, 35(5), 2265–2284. http://doi.org/10.1002/hbm.22326

Knight, R. T. (1996). Contribution of human hippocampal region to novelty detection. Nature, 383(6597), 256–259. http://doi.org/10.1038/383256a0

Lange, E. B. (2005). Disruption of attention by irrelevant stimuli in serial recall. Journal of Memory and Language, 53(4), 513–531. http://doi.org/10.1016/j.jml.2005.07.002

Leiva, A., Parmentier, F. B. R., & Andrés, P. (2015). Distraction by deviance comparing the effects of auditory and visual deviant stimuli on auditory and visual target processing. Experimental Psychology, 62(1), 54–65.

41

http://doi.org/10.1027/1618-3169/a000273

Li, D., Christ, S. E., & Cowan, N. (2014). Domain-general and domain-specific functional networks in working memory. NeuroImage, 102, 646–656. http://doi.org/10.1016/j.neuroimage.2014.08.028

Ljungberg, J. K., & Parmentier, F. B. R. (2012a). Cross-modal distraction by deviance: Functional similarities between the auditory and tactile modalities. Experimental Psychology, 59(6), 355–363. http://doi.org/10.1027/1618-3169/a000164

Ljungberg, J. K., & Parmentier, F. B. R. (2012b). Cross-modal distraction by deviance: Functional similarities between the auditory and tactile modalities. Experimental Psychology, 59(6), 355–363. http://doi.org/10.1027/1618-3169/a000164

Ljungberg, J. K., Parmentier, F. B. R., Jones, D. M., Marsja, E., & Neely, G. (2014). “What”s in a name?’ “No more than when it”s mine own’. Evidence from auditory oddball distraction. Acta Psychologica, 150, 161–166. http://doi.org/10.1016/j.actpsy.2014.05.009

Macaluso, E. (2010). Orienting of spatial attention and the interplay between the senses. Cortex, 46(3), 282–297. http://doi.org/10.1016/j.cortex.2009.05.010

Mohebbi, R., Gray, R., & Tan, H. Z. (2009). Driver reaction time to tactile and auditory rear-end collision warnings while talking on a cell phone. Human Factors, 51(1), 102–110. http://doi.org/10.1177/0018720809333517

Moray, N. (1959). Attention in dichotic listening: Affective cues and the influence of instructions. The Quarterly Journal of Experimental Psychology, 11(1), 56–60. http://doi.org/10.1080/17470215908416289

Morey, R. D. (2008). Confidence Intervals from Normalized Data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4(2), 61–64. http://doi.org/10.20982/tqmp.04.2.p061

Ngo, M. K., & Spence, C. (2010). Auditory, tactile, and multisensory cues facilitate search for dynamic visual stimuli. Attention, Perception, & Psychophysics, 72(6), 1654–1665. http://doi.org/10.3758/APP.72.6.1654

Näätänen, R., Gaillard, A. W. K., & Mäntysalo, S. (1978). Early selective-attention effect on evoked potential reinterpreted. Acta Psychologica, 42(4), 313–329. http://doi.org/10.1016/0001-6918(78)90006-9

Nöstl, A., Marsh, J. E., & Sörqvist, P. (2012). Expectations Modulate the Magnitude of Attentional Capture by Auditory Events. PLoS ONE, 7(11),

42

e48569. http://doi.org/10.1371/journal.pone.0048569

Parmentier, F. B. R. (2008). Towards a cognitive model of distraction by auditory novelty: The role of involuntary attention capture and semantic processing. Cognition, 109(3), 345–362. http://doi.org/10.1016/j.cognition.2008.09.005

Parmentier, F. B. R. (2014). The cognitive determinants of behavioral distraction by deviant auditory stimuli: A review. Psychological Research, 78(3), 321–338. http://doi.org/10.1007/s00426-013-0534-4

Parmentier, F. B. R. (2016). Deviant sounds yield distraction irrespective of the sounds’ informational value. Journal of Experimental Psychology: Human Perception and Performance, 42(6), 837–846. http://doi.org/10.1037/xhp0000195

Parmentier, F. B. R., Elford, G., Escera, C., Andrés, P., & SanMiguel, I. (2008). The cognitive locus of distraction by acoustic novelty in the cross-modal oddball task. Cognition, 106(1), 408–432. http://doi.org/10.1016/j.cognition.2007.03.008

Parmentier, F. B. R., Elsley, J. V., Andrés, P., & Barceló, F. (2011). Why are auditory novels distracting? Contrasting the roles of novelty, violation of expectation and stimulus change. Cognition, 119(3), 374–380. http://doi.org/10.1016/j.cognition.2011.02.001

Parmentier, F. B. R., Ljungberg, J. K., Elsley, J. V., & Lindkvist, M. (2011). A behavioral study of distraction by vibrotactile novelty. Journal of Experimental Psychology. Human Perception and Performance, 37(4), 1134–1139. http://doi.org/10.1037/a0021931

Quak, M., London, R. E., & Talsma, D. (2015). A multisensory perspective of working memory. Frontiers in Human Neuroscience, 9(April), 1–11. http://doi.org/10.3389/fnhum.2015.00197

Röer, J. P., Bell, R., Marsh, J. E., & Buchner, A. (2015). Age equivalence in auditory distraction by changing and deviant speech sounds. Psychology and Aging, 30(4), 849–855. http://doi.org/10.1037/pag0000055

Saarinen, J., Paavilainen, P., Schöger, E., Tervaniemi, M., & Näätänen, R. (1992). Representation of abstract attributes of auditory stimuli in the human brain. Neuroreport. http://doi.org/10.1097/00001756-199212000-00030

SanMiguel, I., Linden, D., & Escera, C. (2010). Attention capture by novel sounds: Distraction versus facilitation. European Journal of Cognitive Psychology, 22(4), 481–515. http://doi.org/10.1080/09541440902930994

43

Saults, J. S., & Cowan, N. (2007). A central capacity limit to the simultaneous storage of visual and auditory arrays in working memory. Journal of Experimental Psychology. General, 136(4), 663–684. http://doi.org/10.1037/0096-3445.136.4.663

Schröger, E., Bendixen, A., Denham, S. L., Mill, R. W., Bohm, T. M., & Winkler, I. (2014). Predictive regularity representations in violation detection and auditory stream segregation: From conceptual to computational models. Brain Topography, 27(4), 565–577. http://doi.org/10.1007/s10548-013-0334-6

Schröger, E., Bendixen, A., Trujillo-Barreto, N. J., & Roeber, U. (2007). Processing of abstract rule violations in audition. PloS One, 2(11), e1131. http://doi.org/10.1371/journal.pone.0001131

Schröger, E., Giard, M. H., & Wolff, C. (2000). Auditory distraction: Event-related potential and behavioral indices. Clinical Neurophysiology, 111(8), 1450–1460. http://doi.org/10.1016/S1388-2457(00)00337-0

Schröger, E., & Wolff, C. (1998). Behavioral and electrophysiological effects of task-irrelevant sound change: a new distraction paradigm. Brain Research. Cognitive Brain Research, 7(1), 71–87. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/9714745

Smith, B. D., Dickel, M., & Deutsch, S. (1978). Overextinction and Test Stimulus Modality Determinants of Dishabituation. Psychophysiology, 15(4), 324–329. http://doi.org/10.1111/j.1469-8986.1978.tb01388.x

Sokolov, E. (1963). Higher nervous functions; the orienting reflex. Annual Review of Physiology, 25(September), 545–580. http://doi.org/10.1146/annurev.ph.25.030163.002553

Spence, C., & Walton, M. (2005). On the inability to ignore touch when responding to vision in the crossmodal congruency task. Acta Psychologica, 118(1–2 SPEC. ISS.), 47–70. http://doi.org/10.1016/j.actpsy.2004.10.003

Stefanics, G., Kremláček, J., & Czigler, I. (2014). Visual mismatch negativity: a predictive coding view. Frontiers in Human Neuroscience, 8(September), 666. http://doi.org/10.3389/fnhum.2014.00666

Sörqvist, P., Nöstl, A., & Halin, N. (2012). Working memory capacity modulates habituation rate: Evidence from a cross-modal auditory distraction paradigm. Psychonomic Bulletin & Review, 19(2), 245–250. http://doi.org/10.3758/s13423-011-0203-9

Talsma, D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind. Frontiers in Integrative Neuroscience,

44

9(March), 19. http://doi.org/10.3389/fnint.2015.00019

Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14(9), 400–410. http://doi.org/10.1016/j.tics.2010.06.008

Treisman, a M. (1969). Strategies and models of selective attention. Psychological Review, 76(3), 282–299. http://doi.org/10.1037/h0027242

Tremblay, S., Nicholls, A. P., Alford, D., & Jones, D. M. (2000). The irrelevant sound effect: Does speech play a special role? Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(6), 1750–1754. http://doi.org/10.1037/0278-7393.26.6.1750

Unger, S. M. (1964). Habituation of the vasoconstrictive orienting reaction. Journal of Experimental Psychology, 67(1), 11–18. http://doi.org/10.1037/h0044510

Unsworth, N., & Engle, R. W. (2007). On the division of short-term and working memory: an examination of simple and complex span and their relation to higher order abilities. Psychological Bulletin, 133(6), 1038–66. http://doi.org/10.1037/0033-2909.133.6.1038

Vachon, F., Hughes, R. W., & Jones, D. M. (2012). Broken expectations: Violation of expectancies, not novelty, captures auditory attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(1), 164–177. http://doi.org/10.1037/a0025054

Vachon, F., Labonté, K., & Marsh, J. E. (2017). Attentional capture by deviant sounds: A noncontingent form of auditory distraction? Journal of Experimental Psychology: Learning, Memory, and Cognition, 43(4), 622–634. http://doi.org/10.1037/xlm0000330

Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2009). Poke and pop: Tactile-visual synchrony increases visual saliency. Neuroscience Letters, 450(1), 60–64. http://doi.org/10.1016/j.neulet.2008.11.002

Winkler, I. (2007). Interpreting the mismatch negativity. Journal of Psychophysiology, 21(3–4), 147–163. http://doi.org/10.1027/0269-8803.21.34.147

Winkler, I., & Czigler, I. (2012). Evidence from auditory and visual event-related potential (ERP) studies of deviance detection (MMN and vMMN) linking predictive coding theories and perceptual object representations. International Journal of Psychophysiology, 83(2), 132–143.

45

http://doi.org/10.1016/j.ijpsycho.2011.10.001

Wood, N. L., & Cowan, N. (1995). The cocktail party phenomenon revisited: attention and memory in the classic selective listening procedure of Cherry (1953). Journal of Experimental Psychology. General, 124(3), 243–62. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7673862

Yabe, H., Tervaniemi, M., Reinikainen, K., & Näätänen, R. (1997). Temporal window of integration revealed by MMN to sound omission. NeuroReport, 8(8), 1971–1974. http://doi.org/10.1097/00001756-199705260-00035

Yabe, H., Winkler, I., Czigler, I., Koyama, S., Kakigi, R., Sutoh, T., … Kaneko, S. (2001). Organizing sound sequences in the human brain: The interplay of auditory streaming and temporal integration. Brain Research, 897(1–2), 222–227. http://doi.org/10.1016/S0006-8993(01)02224-7

46

Appendix

Study 1

Study 2

Study 3