workshop report for the enterface workshop ......alpha power before ssvep stimulation was...

WORKSHOP REPORT FOR THE ENTERFACE WORKSHOP IN GENOVA, ITALY 2009 1

Bacteria Hunt:A multimodal, multiparadigm BCI game

C. Mühl (1), H. Gürkök (1), D. Plass-Oude Bos (1), M. E. Thurlings (2,3),L. Scherffig (4), M. Duvinage (5), A. A. Elbakyan (6), S. Kang (7),

M. Poel (1), D. Heylen (1)(1) University of Twente, The Netherlands, (2) TNO Human Factors, The Netherlands,

(3) Utrecht University, The Netherlands, (4) Academy of Media Arts Cologne, Germany,(5) Faculté Polytechnique de Mons, Belgium, (6) Kazakh National Technical University,

Kazakhstan, (7) Gwangju Institute of Science and Technology, South Korea

Abstract—Brain-Computer Interfaces (BCIs) allow users to control applications by brain activity. Among their possibleapplications for non-disabled people, games are promising candidates. BCIs can enrich game play by the mental andaffective state information they contain. During the eNTERFACE’09 workshop we developed the Bacteria Hunt gamewhich can be played by keyboard and BCI, using SSVEP and relative alpha power. We conducted experiments in orderto investigate what difference positive vs. negative neurofeedback would have on subjects’ relaxation states and howwell the different BCI paradigms can be used together. We observed no significant difference in mean alpha band power,thus relaxation, and in user experience between the games applying positive and negative feedback. We also found thatalpha power before SSVEP stimulation was significantly higher than alpha power during SSVEP stimulation indicatingthat there is some interference between the two BCI paradigms.

Index Terms—brain-computer interfaces, computer games, multimodal interaction.

F

1 INTRODUCTION

The study of Brain-Computer Interfaces (BCI)is a multidisciplinary field which combines en-gineering, cognitive neuroscience, psychology,machine learning, human-computer interactionand others. Applications using this direct com-munication channel from brain to machine arejust as diverse, from rehabilitation to affectivecomputing.

With methods like electroencephalography(EEG) it is possible to measure voltage dif-ferences over the scalp, which are the resultof brain activity in the neocortex. With thismethod neuroscience has identified several pat-terns of activity that are associated with dis-tinct cognitive functions. EEG opens thereforea window into the mind, which can be usedfor a direct communication between brain andcomputer [17]. It has a number of advantagesover other methods, as it is non-invasive, hasa high temporal resolution, does not require a

laboratory setting, is relatively cheap, and it iseven possible to create wireless EEG head-sets.Downsides of EEG are a low spatial resolutionand its sensitivity to noise and artifacts [28].The hardware also requires some time to set up,and afterwards everything needs to be cleaned.Dry caps which do not need conductive gel andare just as easy to set up as head phones are indevelopment, which will solve this problem.

Where originally BCI research has been fo-cused on paralyzed patients, new develop-ments as affordable and wireless dry cap tech-nology make BCI viable for healthy users. Be-sides the novelty factor, and the magic of beingable to use your brain directly for control, BCIalso provides private, hands-free interaction.It increases the information users can provideapplications, and applications in turn can reactmore appropriately, for example by also takingthe user’s mental state into account.

A large potential target group are gamers, asgames are an area where novelty is appreciated


and learning new skills is often part of the chal-lenge [19]. For research with patients, gamescan be a very interesting option as well. Virtualworlds can provide a safe environment to learnto produce specific brain activity: Accidentallysteering your wheel chair off the stairs is lesspainful in a virtual environment. Additionally,the game element can keep tedious training funand motivating.

Unfortunately, there are still issues that causeproblems when trying to use a BCI. There arelarge differences between the brain activity be-tween people, and even within one person thebrain activity changes quite quickly over time[5]. This makes it difficult to create a systemthat will understand what the user is trying todo, especially for a longer duration. Dependingon the BCI paradigm used a lot of training maybe required (ranging from minutes to months),for the user to be able to generate the correctsignal for the system. Alternatively, the systemmay be trained with user specific data to recog-nize the user’s brain activity associated with acertain (mental) action. However, it is possiblethat the person trying to use the system fallsinto the category of the so-called BCI illiteratelike 20% of the population [20]. This means thatthis particular user may not be able to generatethe signal in a way that is measurable to thesystem, and hence will not be able to control it.As a result from using EEG as a measurementmethod, the recorded brain activity has a lowsignal-to-noise ratio and is susceptible to arti-facts stemming from eye or muscle movements.These problems cause a high level of uncer-tainty when analyzing and interpreting brainsignals. There are also issues with speed andtiming, as relevant brain activity needs to beinduced, recorded, analyzed, and interpreted,before the corresponding action can be per-formed in the application.

Although paralyzed patients for which BCIis the only interaction modality left may bewilling to accept resulting problems with ro-bustness and speed, healthy subjects will beless forgiving. For them many other inputmodalities are available. Therefore, now othermore traditional usability and user experiencechallenges have to be solved as well, in orderto exploit the full potential of this innovative

technology and increase its acceptance amongthe general population.

Current BCI research applications are oftenvery limited, restricting themselves to the useof one modality (BCI) and one BCI paradigm,to control one type of interaction in a verysimplified game [22]. It is a big step from thissituation to interaction found in current com-mercial video games. Besides the large amountof actions that can be taken in game worldsnowadays, gamers will not behave accordingto the restrictions often applied in current BCIresearch in order to reduce artifacts in EEG.Gamers will move, multi-task, and use multi-ple interaction modalities. Because of the prob-lems with BCI at the moment, applying BCI incombination with traditional control modalitiesenables the use of its advantages, while itsdownsides can be avoided by other modes ofinput. The same is true for the use of multipleBCI paradigms.

The goal of this project has been the devel-opment of a game that combines traditionalgame control (e.g. mouse or keyboard) withBCI. This game is a platform for the studyof the use of BCI in a game environment,to learn more about the situations mentionedabove: using BCI in an efficient and naturalway considering usability and user experience,using it in combination with other modalities,using multiple BCI paradigms within one ap-plication, allowing for a realistic setting, andpossible multiplayer interaction. Besides thesespecific interests, this platform can also help inyielding new insights into more general issuesof applying BCI.

To explore some of these previous issues, anexperiment has been conducted using a mul-timodal, multiparadigm, single-player varietyof the game. This functions also an illustrationof how this platform can be used for suchresearch. This investigation is focused on onehand on the influence of using a certain BCIparadigm on the user experience, and on theother hand on possible interference effects ofusing two BCI paradigms simultaneously.

Applying neurophysiological input in gamesgives rise to specific considerations in the de-sign space. Hence, to start off, this new gamedesign space for BCI is clarified in Section 2.


The third section provides more informationon those BCI paradigms that were initiallythought suitable for game control, consider-ing the zero amount of training they requiredand the different types of input they could beapplied to. Pilot studies were run to explorethe most appropriate paradigm and parame-ters for the game. The full system consistingof the game and the pipeline for BCI controlhas been designed according to our specificrequirements, as described in Section 4. Thisis followed by Section 5, which focuses onthe conducted experiment and the obtainedresults. The paper concludes with a discussionand conclusions about the design space, thedeveloped research platform, and the results ofthe conducted experiment.

2 A BCI GAME DESIGN SPACEDuring the last decade, computer games havereceived increased attention from the scientificcommunity. The academic field of game studies,while still in formation, already has given birthto a number of conferences and journals andthe design of computer games is taught atuniversities around the world. These develop-ments created techniques that may support thedesign of computer games and terminologiesthat may standardize the description and anal-ysis of games and game design efforts. Withinthe project, game studies terminology was usedto create a coherent description of what formsof BCI are possible in different forms of com-puter gaming. This “BCI game design space”first of all was needed to enable a precise defi-nition of the couplings of neurophysiologicalinput and game mechanics within the gameand its various levels. In addition, it suggests aterminology that may be used to standardizethe hitherto incoherent descriptions of thesefactors in BCI gaming projects.

2.1 Game studies terminologyComputer games, according to [1], consist of agame world, rules and gameplay. The former twocomprise the entities that are present withina game as well the rules that connect theseentities and define their behavior. Gameplay in-stead results from interaction. It emerges when

a player applies the rules to the world byplaying the game.

In order to be able to elicit gameplay, acoupling of the player to the game world mustbe established. This possibility of interactionyields a subjective experience of causal agencythat results from a player’s activity and itsdirectly perceivable results in the game world.This experience has been named effectance [14].While effectance is rooted in feedback on thelevel of in- and output, a player will alsoexperience a game on the level of its rules:She will, for instance, not only move entitiesin the game world but eventually also win orloose the game. The experience of successfullyinteracting with a game’s rules has been calledthe perception of control [14]. Both, effectanceand control, are thought to be crucial factorsfor the induction of computer game enjoyment[14].

Computer games that not only rely on tradi-tional forms of interaction, but that use phys-iological sensors pose new challenges for thedefinition of both: the direct application ofinputs to the game world – and therewiththe influence of these sensors on the level ofeffectance – as well as the role of the sensors forthe game rules, on the level of control. Noticethat neurophysiology is a part of physiologythat deals with the nervous system. Therefore,while a BCI component (like an EEG sensor)is better classified as a neurophysiological one,it is also not wrong to refer to it simply as aphysiological one.

2.2 Classification of BCI paradigms

A number of BCI paradigms exist and eachof those may be suitable to support differentforms of effectance. In the following, a two-dimensional classification of this BCI inputspace is suggested. While being formulatedfor BCI input, this classification generalizes toall forms of physiological input to computergames.

The dimensions of this classification are de-fined by (i) the dependence on external stimuliand (ii) the dependence on an intention tocreate a neural activity pattern as illustrated inFigure 1.


Fig. 1: A classification of BCI paradigms, spanning vol-untariness vs. stimulus dependency.

Axis (i) stretches from exogenous (or evoked)to endogenous (or induced) input. The for-mer covers all forms of BCI which necessarilypresuppose an external stimulus. SSVEPs asneural correlates of stimulus frequencies, forinstance, may be detected if and only if evokedby a stimulus. They hence are a clear exampleof exogenous input. Endogenous input insteaddoes not presuppose an external stimulus. Oneprominent example is the usage of alpha bandpower in neurofeedback applications. Whilealpha activity may be influenced by externalstimuli, it in principle can be measured whenno stimulus is present and hence classifies asendogenous. Another way of separating bothpoles of the axis is the possibility of buildinga self-paced (or asynchronous) BCI [25]: Onlyendogenous input can be used to build a sys-tem that is self-paced, in the sense that it candecide if a user initiates an action by brainactivity, whenever she does so. On the otherhand, all forms of exogenous BCI necessarilyare synchronous.

Axis (ii) stretches from active to passiveinput. Active input presupposes an intentionto control brain activity while passive inputdoes not. Imagined movements, for instance,can only be detected if subjects intend to per-form these, making the paradigm a prototypi-cal application of active BCI. Alpha and othermeasures of band power instead can also bemeasured if subjects exhibit no intention to

produce it.A summary of BCI Games by Reuderink

[22] distinguishes between the application ofband power feedback (F), P300, VEP and theneural correlates of imagined, planned and realmovements (M). According to the scheme pro-posed here, all cases of F listed in [22] clas-sify as passive-endogenous, all applications ofP300 and VEP classify as active-exogenous andall cases of M classify as active-endogenous.However, other applications are possible or –outside the context of gaming – even are inuse: P300 lie-detectors, for instance, classify aspassive-exogenous. Using the changing VEPstrength evoked by a stimulus that constantlyis present as a correlate of attention would alsoclassify as passive-exogenous.

2.3 Physiological effectance and control

If a BCI input directly affects the game world,the whole BCI feedback cycle is found on thelevel of effectance. In this case we speak ofdirect interaction. Direct interaction is usuallyassociated with active BCI in which the usermakes an effort to create a control signal inorder to succeed at an aim. To the contrary, ifnot the game world but its rules are affected,parts of the interaction take place on the levelof control. Here, BCI may be used to changeoverall parameters of the game, such as itsspeed or difficulty. Such forms of interactionare less directly perceived and hence can benamed indirect. Indirect interaction is usuallyrealized through passive BCI in which the userdoes not put much of his effort on using theBCI but rather on a more important controlmodality.

Moreover, if physiological activity does affectthe game world or the rules, but if influencingthese is not necessary in order to win the game,it constitutes a form of interaction that merelyis auxiliary. An example for auxiliary interac-tion would be a game in which physiologicalactivity changes the game’s appearance or isused to gain bonus points. This sort of interac-tion is used mostly for non-critical features, andthus is very suitable to employ BCI paradigmswhich are hard or time consuming to detect.


2.4 Multiplayer BCIIn multi-player games the number of pos-sibilities of affecting the game world or itsrules increase: First, the game may be com-petitive, cooperative or generative. While com-puter gaming started with competitive gamesmostly (such as Spacewar! and Pong) in theage of online multi-player games cooperativegaming has gained much importance. Genera-tive games are comparably rare, one prominentexample is Electroplankton by Toshio Iwai.

Second, the mapping from physiologicalmeasurements to the game may either takeplace separately for each player or in con-junction. In BrainBall for instance, alpha bandpower of two players is combined to alterthe position of one single ball, making it anexample for conjunct control of the game [10].Of course, it also would be possible to controlthe positions of two balls by one player each,creating a game using separate controls.

3 THE BCI PARADIGMSExisting BCI paradigms in BCI applications canbe divided in several ways. The previous sec-tion distinguishes the classification into passivevs. active, and evoked vs. induced paradigms.To control an active BCI, one has to activelyperform mental tasks such as motor imagery.For example by imagining left or right handmovement a cursor can be controlled to moveup or down. This principle was applied inthe game brainpong, where a bat is controlledto move up or down to block an approach-ing ball, using motor imagery [16]. Typically,active BCIs demand the most intensive usertraining. Evoked BCIs require probe stimuli,stimuli presented by the system. By attendingto one of the stimuli, an interpretable brainsignal can be elicited. In an evoked BCI thisinterpretation is translated into a command ofthe system. The type of brain signal that iselicited is dependent on the characteristics ofthe stimuli provided. The Steady State VisuallyEvoked Potential (SSVEP) and the P300 areboth features in the EEG that can be elicitedby certain probe stimuli and are explained inmore detail later on. These features are inter-esting for BCI applications, because the signal-

to-noise ratio is relatively high and they donot require user training. Passive BCIs detectthe changes in cognitive and affective states,and do not require the user’s active attentionor the performance of cognitive mental tasks.In principle BCIs that belong to this categoryare not used for direct control (e.g. of cursormovement), but are more suitable for indirectcontrol. Alpha band power can be used in thispassive context, as it is related to a relaxedmental state [3]. However, the users can alsolearn to control their alpha activity, thus com-plicating the location of this approach along thepassive/active dimension.

For the current game, it was decided toforgo paradigms that need extensive user train-ing or machine learning, while at the sametime a variety of different types of paradigmswas wanted. This resulted in the selection ofSSVEP, P300, and neurofeedback based on thepower in the alpha band. This section de-scribes these BCI paradigms. Two pilot studieswere carried out to check the feasibility ofthe exogenous BCI paradigms in this gameimplementation, explore appropriate parame-ters and test classification algorithms for theBCI paradigms intended for the game control.The first study explored the neurophysiologicalresponses to stimuli flickering with a fixedfrequency (SSVEP). The second study investi-gated the responses to slower changing stimuli(P300). The final subsection provides informa-tion about alpha activity, based on literature.

3.1 SSVEP

SSVEPs can be induced if a person is attend-ing to a flickering visual stimulus, such as anLED. The frequency of the attended stimulus[21], as well as its harmonics [18], can thenbe found in the EEG. SSVEPs are interestingfor BCI-applications, because multiple stimulican be provided simultaneously in contrast tothe P300 for which stimuli have to be pre-sented sequentially (see subsection 3.2). If thestimuli are all flickering with a different fre-quency, then the attended frequency will dom-inate over the other presented frequencies inthe observers EEG. A commonly used methodto detect SSVEPs is to apply a Fast Fourier


Transformation on the EEG and compare theamplitudes in the frequency bins correspond-ing to the frequencies of the stimuli provided. Ifonly one stimulus is used, the amplitude of thecorresponding frequency bin is compared to aset threshold. Frequencies of stimuli between5-20 Hz elicit the highest SSVEPs [9]. SSVEPsare recorded over the visual cortex; O1, Oz andO2 according to the 10-20 system.

So SSVEPs are dependent on provided flick-ering stimuli, which need to flicker very con-stantly. The goal of this offline study was toinvestigate if it is possible to create such stimulithat are able to elicit the SSVEP response withGame MakerTM. Furthermore, the parametersfor this BCI paradigm were investigated interms of appropriate frequency and the size,shape and pattern used for the stimulus.

3.1.1 Method

Stimuli: For the creation of visual flicker-ing stimuli, several factors have to be taken inaccount. Firstly, is the bandwidth of frequenciesthat in principle can elicit robust SSVEPs, asmentioned above this is between 5 and 20 Hz.Secondly, in the current study it is desiredthat the stimuli are offered on the screen, sothe possible frequencies are dependent on thescreen refresh rate and characteristics. Previousexperience showed that the SSVEP inductioncan not done reliably when the stimulus ap-pearance is changed in every frame. At leasttwo frames after each other need to show thesame stimulus (color). Accordingly, the maxi-mum stimulus frequency that can be obtainedwith a screen refresh rate of 60 Hz is 15 Hz(see Table 1). Therefore, a simple flickeringstimulus is created by the alternation of twoframes presenting a black stimulus and twoframes presenting a white stimulus. Similarly,more complex stimuli are created, for exampleflickering checkerboards, which are often usedto elicit SSVEPs. To investigate the robustnessof the stimuli created with Game MakerTM,such flickering checkerboards were presentedwith the frequencies 7.5, 8.57, 10, 12 and 15Hz. To this end, one color of the stimulus waspresented for two images, while the other colorwas presented for 2-6 images (see Table 1),

resulting in four to eight images that wereconnected in one stimulation period.

Framesper period

Frames on (1)and off (0)

Frequency of stimulus(at 60 frames per second)

4 ...1100... 15 Hz5 ...11000... 12 Hz6 ...110000... 10 Hz7 ...1100000... 8.57 Hz8 ...11000000... 7.5 Hz

TABLE 1: Possible frequencies of a flickering stimuluson a LCD with a screen refresh rate of 60 Hz are shown.

As imaginable, flickering checkerboards arevisually not appealing and are annoying tolook at. Therefore besides the standard checker-board, also other shapes and appearances ofstimuli were investigated. The appearance andcharacteristics are shown in Table 2. To limitthe number of experimental sessions, the diskstimuli were only presented in one frequency,namely 12 Hz.

Experimental Setup: For this pilot studyonly two participants were involved. Theywere seated in front of a laptop, approximately60cm in front of the screen. Each stimulus waspresented for 60 seconds, and appeared in themiddle of the screen. After each stimulus pre-sentation, a pause screen appeared. For detailson the recording of EEG, see Subsection 5.1.The sample frequency for this EEG recordingwas 2048 Hz.

Task: Participants had to sit still in frontof the laptop and attend to the stimulus pre-sented. They were instructed to blink as fewas possible and to avoid any other movementcompletely during stimulus presentation. Par-ticipants controlled the start of each condition.

3.1.2 ResultsOffline Analysis: The EEG recorded at O1,

Oz and O2 was first cut out for each stimu-lus presentation from 40.000 until 100.000 datapoints after the start of the stimulus (60.000data points recorded with a sample frequencyof 2048 Hz corresponds to approximately 30seconds). These extracted signals were firstcommon-average referenced (CAR, see Subsec-tion 4.2.2), and then transformed to the fre-quency domain with a Fast Fourier Transfor-mation.


Checkerboard Small disk Large disk Large disk withhorizontal stripes

Large disk withvertical stripes

Large disk withhorizontal andvertical stripes

Stimulus freq. of0, 7.5, 8.57, 10, 12,

15Hz

12Hz 12Hz 12Hz 12Hz 12Hz

Inner diameter of4o

1o 4o 4o 4o 4o

TABLE 2: The table shows the presented stimuli and the presentation parameters frequency and size. The imageson top were altered with the bottom images to create the flickering stimuli. The third row shows the frequencieseach flickering stimulus was presented with. The last row relates the size of the stimuli to the smallest, namely thesmall disk (o).

Fig. 2: The frequency spectrum of a participants EEGrecorded at Oz during: no stimulus presentation (left)and checkerboard presentation of 12 Hz (right). Here themain response of the SSVEP and its first harmonic areclearly visible.

For the checkerboard stimuli, clear SSVEP re-sponses were found in the EEG of both partici-pants for 7.5, 8.57 and 10 Hz (see also Figure 2).In one participant also the stimuli flickering ata frequency of 12 and 15 Hz evoked SSVEPresponses, although from the latter one onlythe second harmonics was found. From thealternative stimuli, the large disk and the largedisk with horizontal and vertical stripes were

able to evoke SSVEP responses in both partic-ipants. For one participant the large disk withhorizontal stripes and for the other participantthe large disk with vertical stripes also inducedSSVEP responses. No significant differenceswere found for responses recorded at O1, Ozand O2.

Classification: In order to develop a BCIthat uses the SSVEP paradigm, an online classi-fication algorithm is needed. To simulate an on-line situation, the recording of 60 seconds wassliced into four-second windows, with steps ofone second between the start of each window.After CAR and FFT is applied, the SSVEPdetection consists of comparing the target fre-quency power to the maximum peak in thefrequency bin around the target, with a sizeof 3 Hz. If the ratio between these two energypeaks exceeds an experimentally determinedthreshold of 0.8, the window was said to con-tain SSVEP. This SSVEP detection method isdescribed in detail in Subsection 4.2.2, and isdepicted in Figure 7.

After applying the method described aboveon the OZ channel the results presented inTable 3 were obtained. The table shows the true


Stimulationfrequency

Checkerboard Small disk Large disk Large disk withhorizontal

stripes

Large diskwith vertical

stripes

Large disk withhorizontal andvertical stripes

Subject 017.5Hz 84% - - - - -

8.57Hz 93% - - - - -10Hz 98% - - - - -12Hz 90% 56% 92% 79% 76% 97%15Hz 90% - - - - -

Subject 027.5Hz 87% - - - - -

8.57Hz 68% - - - - -10Hz 82% - - - - -12Hz 52% 44% 84% 48% 63% 73%15Hz 80% - - - - -

TABLE 3: The true detection rates of SSVEPs for the different stimuli. The perfect rate is 100 % in the data usedhere.

detection rates (the recognition of SSVEP dur-ing actual SSVEP stimulation) for the differentobjects and stimulation frequencies. Please notethat the optimal performance is 100%, as thetrue detection rate was calculated only on datawith the stimulation present.

Assuming that the ability of the stimulus toelicit SSVEP is the main reason for the obtainedtrue detection rates, those stimuli that yieldedthe best results qualify for further exploration.The stimuli that resulted in a good classifica-tion result for both subjects were:

• Checkerboard pattern flickering at fre-quencies of 7.5, 10 and 15 Hz

• Large disk and the large disk with bothvertical and horizontal stripes, flickering ata frequency of 12 Hz

From these the checkerboard at 7.5 Hz andthe large disk at 12 Hz were selected for furtherevaluation. The large disk was selected be-cause it seemed less obtrusive than a flickeringcheckerboard, which is relevant for applicationin the game. The checkerboard flickering at7.5Hz was selected for further analysis becauseof the potential interferences of the frequenciesfrom 8 to 12 Hz with the other BCI paradigmused in the game. At all frequencies, except7.5 and 15 Hz, the SSVEP would be in thealpha frequency range used in the game for theneurofeedback BCI paradigm. Between the truedetection rates for 7.5 Hz and 15 Hz stimulationonly a minor difference was observed.

For these selected stimuli, the optimal win-

dow length was explored. For this, for eachwindow duration, besides the true detectionrates, also the false detection rates (detectionof the stimulus when no stimulus was pre-sented) and the classification accuracies (totalnumber of correct classifications divided bythe total number of classifications) were calcu-lated. The results are presented in Figure 3. Asexpected, the classification accuracy improveswith longer window durations, as it containsmore data. The figure shows that windows ofat least 3 seconds are required to obtain properclassification results. For shorter windows thetrue detection rate is still high, but the false de-tection rates increase dramatically, decreasingthe classification accuracies to below 70%.

3.1.3 Conclusion

The offline analysis of EEG recorded whileattending to flickering stimuli created withGame MakerTM, shows that SSVEPs can beelicited with these stimuli. The offline analy-sis and the offline classification both showedwell detectable SSVEPs for both subjects forthe stimuli: checkerboards flickering at 7.5, 10,and 15 Hz, and the plain large disk and largedisk with both horizontal and vertical stripesflickering at 12 Hz. Of these stimuli, the plainlarge disk is thought to be the least obtrusive,and therefore is implemented in the game. Thedeveloped classification algorithm seems to berobust enough for implementation in an onlineclassification structure. Analysis showed that


Fig. 3: The classification accuracy, true and false detection rates for the checkerboard and large disk stimuli flickeringat 7.5 and 12 Hz, respectively.

windows of at least 3 seconds are required. Sothe game requires the user to focus for at least3 seconds at flickering stimuli if presented.

3.2 P300

A P300 is an event-related potential (ERP). TheP300 (also referred to as the P3) is a positivepeak in the EEG that occurs approximately300ms after the presentation of a target stimulusthat stands out from other standard stimuli ordistractors. One reason why the stimulus standsout can be due to differences in appearance,e.g. when one black sheep is presented afterten white ones. The experimental paradigmis called an oddball paradigm [12]. The P300amplitude decreases if the target and stimuliare more similar [6]. Another reason is becausea person is attending to a certain stimulus. TheP300 is thought to be related to a higher levelattentional process or orienting response. Ingeneral P300s are detected at Fz, Cz and Pz asdefined by the 10-20 system. To elicit a P300,probe stimuli can be presented in the visual,auditory and tactile modality [2].

The first BCI application that used the P300paradigm is the P300-matrix speller [8]. In this

application letter and numbers are placed ina 6 by 6 matrix. The rows and columns flashup sequentially. Every time the symbol of one’schoice flashes up and the user is attendingto it, a P300 is (potentially) elicited. In thisway, users (e.g. patients with neuromusculardiseases) can spell words and communicatewith their environment, although very slowly(approximately 1-2 words per minute). As theP300 can be easily modulated with attention, itis an interesting component to use in games.

To test the feasibility of using P300 withina game, an offline experiment has been con-ducted that tested visual stimuli created inGame MakerTM. The stimuli were shaped likebacteria, testing the potential applicability ofsuch a stimulus in a game. Additionally, thepossibilities to let a stimulus stand out (insteadof flashing up) were investigated by changingstimulus’ color, size and angle.

3.2.1 MethodsExperimental Setup: The general setup was

similar to the previous pilot experiment (seeSubsection 3.1). Comparable to the spelling ma-trix, stimuli are grouped, similar to rows andcolumns. However, visually, the positioning of


the stimuli is unstructured, to simulate a gamesituation. At the start of each trial in this offlineexperiment, a target stimulus was indicated.Each group was highlighted for 150 ms fol-lowed by an inter-stimulus interval of 150ms.The presentation of the groups was done inrandom order and repeated nine times.

Stimuli: Three different possibilities to leta stimulus stand out were tested. See Table 4for the visuals.

Task: A target was indicated at the startof each trial, by highlighting it for 2 seconds.The participants were instructed to count thenumber of times this particular target stimuluswas changed, to keep their attention focused onthe target. Furthermore they were instructed tosit still, move and blink as few as possible.

Analysis: The P300 potentials were ana-lyzed by averaging the signal of a certain elec-trode (the most important ones were Cz andPz) that was recorded from 200 ms before until800 ms after the stimulus-onset. The signalswere baseline-corrected by the subtraction ofthe average of the 200 ms before stimulus-onset. After that the signals that were related tothe target-stimuli were averaged (two signalsper trial), and the signals that were relatedto the non-target-stimuli were averaged (foursignals per trial).

3.2.2 ResultsThe results of the analysis for EEG responsesand also the average over the nine trials areshown in the graph below (see Figure 4). Thegraph shows the P300 elicited by the stimulithat changed color (orange stimuli were neu-tral and black stimuli were highlighted), asa highlight-effect. The other conditions used,’change size’ and ’rotation’, also elicited P300s.However they were lower in amplitude.

3.2.3 ConclusionBased on this offline analyses, it is expectedthat the P300 paradigm may be usefull in thegame, and good results can be expected if atleast two trials are used. This would require3.6 seconds of data before a classification canbe done. The highlight effect that gave the bestresults was changing the color of the stimu-lus from orange to black. Unfortunately there

Fig. 4: EEG response of one participant to target stimuli(red) and non-target stimuli (blue) recorded at Cz. Thestimulus was present during the grey area. The two thicklines represent the averages over nine trials: the EEGresponse to targets is clearly different from the responseto non-targets. The averaged target line shows a peak at285 ms after stimulus onset, indicating the presence of aP300.

was not enough time during the eNTERFACEworkshop to also complete the online P300-pipeline. Therefore this is recommended forfuture work.

3.3 AlphaThe term ‘alpha rhythm’ is restricted to brainactivity in the alpha range occurring at the backof the head, which is related to a relaxed wake-fulness and best recorded with eyes closed,according to the definition of the InternationalFederation of Societies for Electroencephalog-raphy and Clinical Neurophysiology [4]. In ourreport, alpha band power is used in the moregeneral sense of energy in the frequency bandof the alpha rhythm: 8 to 12 Hz, which can beobserved over all cortical regions.

Alpha activity used to be considered to beinversely related to neuronal activity. This canbe interpreted either as cortical inhibition, orcortical idling. This view is shifting, as alphais now seen to have functional correlates tomovement and memory [26].

As a correlate to relaxed wakefulness, alphaactivity could be an interesting measure forpassive BCIs. Alpha activity is also consideredan important tool in neurofeedback. Besides


Standard stimulus Size change Color change Rotation

TABLE 4: The stimuli for the P300 paradigm. The standard stimulus, and the changes in size, color, and rotationused to elicit the P300.

helping patients, it might also be used to im-prove our mental capabilities, as research hasindicated relations between alpha activity andintelligence [7], and the ability to cope withstress [27]. This is another reason why alphaneurofeedback and neurofeedback in generalcould be interesting BCI paradigms to use ingames.

3.4 ConclusionsFor this project, three potential paradigms werelooked into, which were selected based on theircharacteristics and the minimal amount of usertraining and machine learning they required.These paradigms were looked up in literature,and for the two evoked paradigms small pilotstudies have been conducted.

SSVEPs can be elicited with a stimulus look-ing like a plain large disk, which is consid-ered to be less obtrusive then for examplethe checkerboard stimulus often used for thisparadigm. Although this disk was only testedat 12 Hz, the frequencies of 7.5 and 15 Hzobtained good results for the checkerboardstimuli. To minimize interference with the al-pha frequency range used in the neurofeedbackparadigm, the flickering of the disk with 7.5 Hzseemed a good stimulus choice for the SSVEPparadigm. To obtain an acceptable accuracy ofdetection, windows of at least three seconds arerecommended.

Alpha could be an interesting correlate forthe mental state of relaxed wakefulness, to beused as a passive BCI modality, or in a moreactive way with potential mental benefits as aresult of the training.

For P300, the highlight effect of changing thecolor of the stimulus from orange to black gavethe best results. At least two trials would berecommended for classification, which wouldrequire 3.6 seconds of data. An online pipelinefor this paradigm has not been implementedduring this workshop, but is recommended asfuture work.

4 THE GAME AND BCI DESIGNThe application was built in guidance of prin-ciples and requirements defined in §4.1.1 and§4.2.1. For the ease of maintainability and opti-mal performance the system was decomposedinto subsystems, namely the game and the dataprocessing. The game subsystem is responsiblefor running the game and sending the markersdepending on the input received from EEGanalysis results and other traditional modali-ties. On the other hand, the data processingsubsystem returns to the game the results itcomputed according to the markers and theEEG signals received. The overview of the sys-tem architecture can be seen in Figure 5.

4.1 The GameThis subsection describes how the game world,game levels and game rules are implementedin accordance to the requirements defined.

4.1.1 Requirements1. Scientific: The game is intended to be abasis for investigation of (1.1) the possibilityof combining BCI with conventional controlsand (1.2) the possibility of using multiple


Fig. 5: Overview of the system architecture.

BCI paradigms. In addition, (1.3) it shouldbe possible to easily extend the game withmultiplayer functionality.

2. Usability: The game should be easy tounderstand and uncomplicated in use, andthus (2.1) be playable without training. Inorder to elicit subjective perception of control,(2.2) the game should provide persistent visualinformation indicating progress in terms of itsrules. Finally, if players are meant to perceivecausal agency based on neurophysiologicalrecordings, (2.3) the game must provide directfeedback on the neurophysiological inputsused.

The game should equally fulfill both scien-tific and usability requirements.

4.1.2 Game worldThe game was built using the Game MakerTMdevelopment platform. The game world con-sists of a small number of entities: playeravatar(s) (the amoeba), targets (bacteria), a nu-meric representation of the points obtained sofar, a graph depicting the recent history ofalpha band power and SSVEP classification andSSVEP stimulus (which always is associatedwith one of the target items) when it is trig-gered. The numeric representation of pointsfunctions as a high-level indicator of progress

Fig. 6: The game world. Nine targets (orange “bacteria”)and one player (blue “amoeba”) are present. The player’sscore is shown at the top left. The histogram above theline depicts her recent alpha band power, below the linethe SSVEP classification results are marked.

in terms of the game rules (req. 2.2) while thegraph depicting alpha band power and SSVEPclassification results functions as a low-levelindicator of the neurophysiological coupling ofthe player and the game (req. 2.3).

4.1.3 Game levelsThe game comprises various levels, each usingthe same game world but slightly differing interms of rules. The levels Keyboard only (K), Key-board+Alpha (KA) and Keyboard+Alpha+SSVEP


(KAS) have been implemented and tested. Fur-thermore, a Keyboard+Alpha+P300 (KAP) levelhas been realized, but not tested. The decisionto use alpha band power, SSVEP and P300was guided by the requirement that it shouldbe possible to play the game without training(req. 2.1). Imagined movements, for instance,presuppose a training phase and thus wouldviolate this requirement. The decision to com-bine alpha band power with SSVEPs or P300respectively was taken because it should bepossible to analyze the combination (and possi-ble interactions) of multiple BCI paradigms inone game (req. 1.2).

4.1.4 Game rules

Some general rules are the same for all levels,others vary with the specific level which willbe explained in this subsection.

General rules: The game world always containsnine target items, which never may overlap. Ifthe distance d between the center of a playeravatar and a that of a target is below the radiusof dmin, the target can be “eaten”. Eaten targetsdisappear and are replaced by a new target,randomly placed on a free spot on screen. Thus,visual disappearance of target items, in addi-tion to the numeric display of score, indicatesthe progress in terms of game rules (req. 2.2).Target successfully eaten results in points forthe player. Eating failures result in negativepoints. The rules for eating are defined differ-ently in each level. The ultimate goal of thegame is to obtain as much points as possible.

Movement is performed using the keys, re-sulting in direct feedback through changedavatar position. Avatar position also jitters bysome random noise and the effect of pressinga key depends on some external values, i.e.game controllability varies. Since the influenceof key presses on the position is variable, thechange of position per key press also providesdirect feedback on how the player currentlyaffects game controllability. If controllability isaffected by neurophysiological measures (asin KA, KAS, and KAP), this implies that thefeedback provided is direct feedback on neuro-physiological activity (req. 2.3). In addition, by

affecting controllability with neurophysiologi-cal measures, keyboard input (a conventionalcontrol) is combined with neurophysiologicalinput (req. 1.1).

Let x and y denote a player avatar’s position,s its speed , c ∈ [0, 1] the external factordetermining controllability and random(x) afunction that returns a real value from [1, x].Pressing the “right” key results in the followingcalculation:

xnew = xold + (s+ (random(s)−s

2))× c

ynew = yold + (random(s)−s

2)× c

In addition, even if no key is pressed, the avatarposition changes from frame f to frame f + 1as follows:

xf+1 = xf + (random(s)−s

2)× c

yf+1 = yf + (random(s)−s

2)× c

Level K rules: This level uses no EEG data atall. Hence the factor c is set to a random valuein [0.4, 0.6]. In this way controllability will varybut stay within a range from which large de-viations of controllability are possible for theother levels. Eating is triggered by pressing akey. It is a success if the nearest target meets thedistance condition described above (d ≤ dmin).Points p are calculated by:

p = 100× (1− ddmin

)

If the eating attempt is triggered while thedistance is above the threshold, p is assignedto −50.

Level KA rules: Eating is performed and evalu-ated as in level K. However, c is linked to theplayer’s relative alpha band power (α ∈ [0, 1]).In a pilot study, relative alpha power wasfound to be in [0.15, 0.23] for most subjects.Thus, an ad hoc scaling [0.15, 0.23] → [0, 1] wasintroduced, creating a scaled αs. For futureversions of the game, subject dependent scaling


methods or adaptive scaling are to be consid-ered.

αs =(α− 0.15)

0.08

c = αs

Therefore, controllability is to be affected bythe player’s mental state, while the influenceon controllability is visualized by the jitterexhibited by the avatar (req. 2.2) and bothkeyboard and BCI input are combined (req.1.1).

Level KAS rules: The factor c is linked to alphaband power as in level KA. The process ofeating is changed as follows: approaching atarget closer than the eating distance dmin trig-gers a SSVEP stimulus appearing next to thetarget. The association of stimulus and targetis visualized by a line connecting both. Thestimulus consists of a circle with a diameterof 64 pixels and flickers with a frequency of7.5 Hz, changing from black to white. It isdisplayed for 6 seconds on the screen. Betweenthe seconds 3 − 6, SSVEP classification resultsare recorded (this specific interval depends onthe window size of 4 seconds used in theanalysis pipeline). If the mean output of theclassifier is above 0.5, the target gets eaten.Points are calculated using the mean alphapower measured between the seconds 3− 6.

p = 100×mean(αs)

Thus, for players, it is of benefit to control bothalpha band power and SSVEP simultaneously(req. 1.3). If SSVEP classification fails, -50points are given, the target “escapes” and ismoved away from its current position.

Level KAP rules: Instead of triggering a SSVEPstimulus, approaching a target now triggers re-peated highlighting of groups of targets. To doso, the nine targets on screen are organized insix groups. Each group consists of three targetsand each target is part of two groups. Groupsare highlighted by replacing the image of itsmember objects by a bigger version of the sameimage for 150ms. Between each flash, an inter-stimulus interval of 150ms is used. After eachgroup has flashed once, a P300 response for

the selected target is measured and, if classifiedcorrectly, the target is eaten successfully. Again,alpha activity during that process is used toscale points received for eating. If no target ora non-target is classified using P300, the targetescapes.

4.1.5 Effectance and controlOn the level of effectance, players interact withthe game world by controlling their avatar’sposition using the keys. In addition, theycontrol eating and target “escape” behaviorby focusing SSVEP or P300 stimuli (in anexogenous-active condition).

On the level of control, players change thegame rules based on their alpha band power:a high alpha is to enhance the controllability ofthe game using keys (in an endogenous-passivecondition).

4.1.6 Single- and multi-playerThe initial prototype of the game is single-player only. But in principle it can easily bescaled to larger number of players (req. 1.3). Ina competitive mode, players compete for points.In a cooperative mode, players try to clear a levelfrom all targets as soon as possible. Both modesemploy separate BCI control. Other ways ofcooperative game play are planned, such as amode in which player avatars are merged forconjunct control.

4.2 Data acquisition and processingThis subsection defines the requirements andsteps for data acquisition and processing.

4.2.1 Requirements1. Hardware considerations: The EEG signalsshould to be acquired by the BioSemiActiveTwo system. For this purpose, the gameshould run on a computer with a parallel portto be able to send the signals and markers.The game and analysis pipeline should be ableto communicate with each other via TCP. Thecomputer running the game should possessa monitor with a resolution of 1024x768 anda minimum refresh rate of 60 Hz to be ableto correctly display the SSVEP frequency. Theanalysis pipeline should be able to receive


the signals from the ActiveTwo system viaUSB. BioSemi active electrodes should be usedin measurements as advised by the BioSemicompany.

2. Performance: The time lag between theanalysis pipeline, game engine, and ActiveTwosystem should be kept at minimum for the sakeof analysis accuracy and game amusement.Both the gaming and pipeline computersshould be fast enough to run continuously,without any halts or delays.

3. Physical environment: All the equipmentshould be operated, preferably, in a room freeof electrical noise. Users should be able toaccess the gaming computer but better be keptaway from the analysis computer to avoid dis-traction.

4.2.2 DesignFor signal processing and machine learningpurposes, Golem and Psychic libraries by BorisReuderink were employed in the pipeline [23],[24]. The EEG signals are continuously readas overlapping 4-second-sliding-windows, theinterval between window onsets being 1 sec-ond. The steps for processing a window isdisplayed in Figure 7. Brain potentials have tobe measured with respect to a reference. Thisreference can be based on some electrodes (e.g.placed on ear lobes or mastoids) but also beelectrode-free by re-referencing, i.e. the refer-ence potential is created from a computationbased on a set of electrodes. Common averagereferencing (CAR) is such a spatial filter usedfor re-referencing. It consists of computing themean of the whole set of electrodes per sampleand then subtracting it from each EEG channel.It was shown to be superior to the ear-referenceand other re-referencing methods [15]. There-fore a data window is first re-referenced byCAR.

Regarding the relative alpha power compu-tation, the data in channel Fz was extractedfrom the window. Applying fast Fourier trans-form (FFT) on the data the power within thealpha band [8 Hz, 12 Hz] was calculated anddivided by the total power within the frequen-cies [4 Hz, 40 Hz].

For the detection of SSVEP presence, thedata in channel O2 was extracted from thewindow. Channel O2 was used since no sig-nificant difference was found in the responserecorded at O1, Oz and O2 (see §3.1) andduring experiments recording sites O1 and Ozwere problematic. Then the power in the fre-quency domain was computed by FFT. Thepower spectrum is expected to contain a peakat the flickering frequency of the circle objecton the screen. The flickering frequency (fflicker)of 7.5 Hz was used as it was one of the bestperforming frequencies (see §3.1) and does notinterfere with the alpha band. But detectionof this peak is a bit tricky task. Sometimesthe amplitude of the flickering frequency maynot be present in the spectrum but, instead, inthe frequencies that are close to it. It can stillbe approximated by increasing the resolutionof FFT, or calculated from known values forneighboring frequencies, but result can be in-accurate. Another point is that the flickeringfrequency may be unstable on custom comput-ers and LC displays. Therefore, SSVEP in somecases might not be detected correctly. Taking allthese challenges into consideration, followingprocedure was defined for detecting the peaksin the spectrum:

1) Look for the frequency with the maximalamplitude (fmax) between the range of[fflicker - 1.5 Hz, fflicker + 1.5 Hz].

2) If fflicker is not represented in the spec-trum obtained by FFT, look for the nearestrepresented frequency (fnearest) and setamp(fflicker) = amp(fnearest).

3) If fmax is next to fflicker in the spectrum,set amp(fflicker) = amp(fmax).

4) If amp(fflicker)/amp(fmax) > thresholdthen conclude that a peak is present. Dur-ing experiments the optimum thresholdwas found to be 0.8.

where amp(F ) is the amplitude of the fre-quency F obtained by FFT.

The relative alpha power value and presenceof SSVEP were continuously sent to the gamePC via TCP.


Fig. 7: The signal processing pipeline of Bacteria Hunt.

5 EXPERIMENTSThe overall goal of the project was to developa game that would be suitable for the useas experimental platform to answer researchquestions related to BCI. From the underlyingresearch questions on applicability andcombinability of BCI control in a game specifichypotheses were posed:

Hypothesis I: The use of a positiveneurofeedback paradigm, based on thereward of high alpha values, as a controlmodality results in an increase in alpha powerand in a state of higher relaxation and lesstension.

Hypothesis II: The combination of positive alphaneurofeedback and SSVEPs causes no detri-mental effects. As alpha feedback should resultin a state of relaxed attentiveness, we assumedthat the attention on the flickering SSVEP stim-uli should not necessitate a change of overallrelaxation state.

5.1 MethodologyIn order to test the hypotheses stated above,an experimental protocol was devised. Partic-ipants were asked to play two versions of agame. Each version was constructed from threelevels, namely K, KA, and KAS (see §4.1.3). Inkeyboard only level K the participant had time

to get acquainted with the game. Furthermore,it can be used as an alpha baseline conditionlater on. The difference between both gameswas in the way the alpha neurofeedback wasapplied. In the positive feedback version (pF)an increase of alpha yielded an increase in con-trollability of the game. In the negative feed-back version (nF) we inverted the relationshipbetween alpha level and controllability.

Five participants (2 female, mean age: 26)took part in the experiment. The participantswere seated in front of the notebook runningthe game (1.8 GHz Pentium M). They readand signed an informed consent and filledin a questionnaire assessing prior drug con-sumption and amount of sleep. After that theelectrode cap was placed and the electrodesconnected. 32 Ag/AgCl Active electrodes wereplaced according to the 10-20 system [13]. TheEEG signals were recorded with 512 Hz sam-ple rate via a BioSemi ActiveTwo EEG systemand processed and saved on a separated datarecording and processing notebook (2.53 GHzQuadcore) running BioSemi ActiView software.

The blocks contained the 3 games in thefollowing order: K, KA, and KAS. Each gamelasted for 4 minutes. That made a total durationof 12 minutes per feedback session, excludingsmall breaks between the games. After eachfeedback session the participants were givena questionnaire to evaluate their experience


during the game session.

5.2 AnalysisTo test the first hypothesis the alpha powerwas analyzed as an objective indicator andthe user experience as a subjective indicator.Specifically a higher level of alpha power forthe positive feedback condition compared tothe negative feedback condition was predicted,as alpha power increases were rewarded in theformer and punished in the latter.

The Game Experience Questionnaire GEQ[11] was used to measure user experience ac-cording to the concepts of competence, im-mersion, flow, tension, challenge, negative, andpositive affect in separate scales. We added acontrol scale that assessed the felt controllabil-ity of the player in each feedback condition.It was predicted that positive neurofeedbackwould lead to a decrease of tension and subse-quently an increase of positive affect.

To test the second hypothesis we analyzedonly the objective indicator of relaxation, i.e.alpha power. We compared the power in thealpha band between the KA and the KAS con-dition of the positive feedback session. Weexpected an equal level of alpha power, thusno interference from the SSVEP stimulationin KAS. Additionally, to check for tempo-rally limited interactions between the two BCIparadigms, we compared the level of alphapower before and during SSVEP stimulationand expected also there an equal amount ofalpha.

5.3 ResultsThe analyses on the data collected have twosorts of implications. They provide conclusionsabout applicability and combinability of BCIswhile answering the hypotheses posed previ-ously.

5.3.1 Applicability of BCIAs result of the first hypothesis, concerningthe applicability of a BCI paradigm in a game,an effect of the applied neurofeedback on al-pha power and user experience was predicted.However, neither objective nor subjective indi-cators for an effect of the feedback were found

(see table 5). No significant difference in meanalpha band power could be shown betweenthe games applying positive and negative feed-back. Accordingly, there was no significant dif-ference in the user experience between the pos-itive and negative alpha-feedback conditions.

The failure to find an effect of the feedbackon alpha power and user experience mighthave different causes. In general, it might bedue to the small sample size of 5 participants.Furthermore, for both feedback conditions theparticipants were instructed to relax, whichmight have lead them to relax in both condi-tions despite the difference in feedback. Possi-ble differences might be covert then by a flooreffect. Complying with this possible caveat,table 5 shows very low scores for the tensionscale. This lack of excitement was also reportedby the participants in informal interviews afterthe experiment. To exclude this possibility thestress level during gaming would have to beincreased, possible by a higher difficulty and amore dynamic and faster nature of the game.

Interestingly, we found a marked differenceof the pre-SSVEP alpha power between nF andpF conditions, with higher alpha power forthe pF condition (p ≤ 0.006). In the absenceof an overall difference of alpha power thiseffect seems contradictory. However, it couldindicate the effectiveness of alpha feedback forconditions with higher difficulty, as the KASconditions were potentially more stressful dueto the more difficult SSVEP-based scoring. Thiseffect might be too small though to be reflectedin overall alpha power and the conscious expe-rience of the players assessed via the GEQ.

5.3.2 Combinability of BCI

As a result of the second hypothesis, concern-ing the combinability of two BCI paradigmsin an application, we expected no interactionof both paradigms, that is no effect of SSVEPon alpha power. Accordingly, we found nosignificant difference in mean relative alphapower between KA and KAS.

Unexpectedly, an influence of the SSVEPstimulation on alpha power was found for thealpha feedback condition pF. Specifically, alphapower before SSVEP stimulation was signifi-


negative F positive F ttest(nF,pF)mean std mean std H P

Questionnaire datacompetence 2,03 0,81 2,13 0,81 0 0,850immersion 1,67 1,10 1,33 0,75 0 0,230flow 2,17 0,72 1,93 0,85 0 0,535tension 0,97 0,69 0,87 0,62 0 0,591challenge 1,47 0,55 1,40 0,25 0 0,688negative 1,20 0,74 1,37 0,59 0 0,561positive 1,90 0,74 1,73 0,51 0 0,430control 2,35 1,05 2,15 0,68 0 0,767

EEG dataα KA 0,18 0,02 0,18 0,01 0 0,723α KAS 0,18 0,02 0,18 0,02 0 0,593preSSVEP α 0,18 0,01 0,19 0,01 1 0,006SSVEP α 0,18 0,02 0,18 0,01 0 0,938preSSVEP α - 0,00 0,01 0,01 0,00 1 0,006SSVEP α

Behavioral dataScore K 4348,34 1118,47 4343,24 1295,33 0 0,991Score KA 4040,04 1038,16 2909,89 1491,15 0 0,176Score KAS 496,70 798,15 205,53 740,25 0 0,621

TABLE 5: The questionnaire results for positive (pF) and negative (nF) feedback sessions. The items were assessedon a scale from 0 to 4, with 0 indicating the least, and 4 the most agreement to the concepts assessed. The EEGwas analyzed in terms of mean relative alpha band power on the Fz electrode during the game 2(α KA) and game3 α KAS), the mean alpha power in the 4 seconds before SSVEP stimulation (preSSVEP α) and during SSVEPstimulation (SSVEP α). The minimum observed and the maximum were 0.15 and 0.23, respectively. The behaviorwas analyzed as points scored in the game.

cantly higher than alpha power during SSVEPstimulation (p ≤ 0.0001).

The contrast of the pre-SSVEP alpha be-tween the different feedback sessions discussedbefore, showed that this effect resulted fromthe higher alpha power during the pre-SSVEPepochs of the pF condition. This suggeststhat positive feedback might work in moredifficult conditions, leading to higher alpha.Consequently, the SSVEP stimulation interfereswith this state of higher alpha and potentiallygreater relaxation. This interference could bedue to several mechanisms. For example, itcould be caused by purely dynamic character-istics of the stimuli. As we compute the relativealpha power, an increase of power in SSVEP-related frequency bands would automaticallylead to a decrease of the alpha power. On theother hand, it could be a result of the task thatis executed during the stimulus presence.

5.4 DiscussionIt was hypothesized that the positive feedbackwould lead to higher levels of alpha power,lower tension, and more positive affect. No

general alpha increase for positive feedbackcould be shown. Similarly, no effect of thealpha feedback was found in terms of userexperience.

Furthermore, we hypothesized that SSVEPstimulation would have no side effect on thealpha power. This was true for the compari-son of alpha level between the levels KA andKAS. However, a difference between the alphapower before SSVEP stimulation to the alphapower during stimulation was observed forthe positive feedback condition. This differencewas due to a higher alpha power before stim-ulation. In the negative feedback condition nosuch difference was found. Hence, we arguethat at least for the KAS condition the positivealpha feedback led to higher alpha, which wasthen attenuated by the SSVEP stimulation.

To explore the applicability and value ofneurofeedback in computer games the manip-ulation of the difficulty level of the applicationmight be interesting. Further studies of theinfluence of SSVEP on alpha power could focuson the origin of the decrease of alpha power.


6 DISCUSSION

We have developed a computer game that usestwo BCI paradigms in addition to a conven-tional control via a keyboard. In pilot studiesdifferent BCI paradigms were explored to de-termine the viability within the game environ-ment and to develop algorithms.

Despite the efforts to develop a BCI pipelinethat could be used without prior training of thesubject or the classifier, the results of our anal-yses and interviews with participants suggestthat a subject-dependent classifier would over-come the shortcomings of the general classifiersapplied. For example, the power in the SSVEPfrequency band varied among subjects. Thus asubject-dependent classifier, applying a thresh-old based on a prior training session, couldincrease the control of the subjects over theavatar. Similarly every individual has his/herown range for the alpha power measured. Inpilot experiments, a generic range determinedby observing the lowest and highest alphapower values of the majority of the subjectswas used. A more elegant approach could beadapting this range per subject. This can beaccomplished via a training session run beforethe game or by dynamically adjusting the val-ues during the game play.

The SSVEP detection method was developedso as to use signals from only one EEG chan-nel (O2) but as shown in §3.1 SSVEP can bedetected in all three occipital channels. Thus,information from these channels can be com-bined to make detection more robust. Also,instead of relying on peak detection solelyin the stimulation frequency, the peaks at thesecond or later harmonics can be employed. Inaddition, the parameters like the bin size andthe reference amplitude can also be tailored foroptimized performance.

The original intention of the project was tobuild a multiplayer game. For this purpose,during the workshop, multiplayer versions ofthe game and the analysis pipeline were im-plemented. However, during the workshop, itwas discovered that the proprietary softwaredelivered with the EEG hardware could notsend information from multiple systems as nec-essary for online multiplayer BCI. When this

restriction is revoked, the multiplayer versioncan also be tested.

The results reported in this paper are basedon 5 participants and especially the non-findings concerning the first hypothesis mightbe due to the low number of samples. Despitethe missing overall effect of neurofeedback, adifference between alpha power in the positiveand negative feedback condition was foundin the pre-SSVEP epochs of the KAS games.This effect might indicate that the failure tofind significant differences in alpha power anduser experience might be due to a floor effectcaused by the high degree of relaxation inboth feedback conditions. To avoid this effectin future studies, it is suggested to increase thedifficulty level and thereby decrease the overallstate of relaxation during the game.

Furthermore, negative feedback was em-ployed to examine the effect that feedback hason the player. While this method is in principlevalid to determine the existence of an effect, itdoes not enable the differentiation of the effectsof positive and negative feedback. To delin-eate these, an additional no-feedback conditioncould be employed.

A possible problem of the interpretationof the results could also result from thegame instructions given before the experiment.The participants were informed that relax-ation would increase controllability. This mighthave caused confusion or other negative effectswhen the opposite effect was realized in thenegative feedback condition. Differences be-tween feedback conditions could thus be dueto this clashing expectations rather than due tothe effect of feedback directly.

Finally, the application of small user expe-rience questionnaires after each game wouldenable the delineation of the effect of the dif-ferent BCI controls on user experience. How-ever, the difficulty lies with the avoidance ofconfounds due to entirely different game me-chanics, which could also be implemented byother techniques.

7 CONCLUSIONWith the new branch of the BCI research whichnow considers non-disabled people also as the


potential users, BCIs started to be incorpo-rated into everyday applications, like computergames. BCI is an invaluable communicationchannel that can directly convey informationno other modality can: the state and intentionof the user. This information can be used tocontrol, modify or adapt a game tailored tothe player, and could thereby provide addedvalue. However, the extent that BCIs can tol-erate the dynamic environment of games orthe existence of other modalities is still underinvestigation.

In this paper we proposed a design spacefor physiological game design and describeda multimodal, multiparadigm BCI game calledBacteria Hunt which is controlled by keyboard,SSVEP, and relative alpha power. We inves-tigated how well different paradigms can beused together and what effect positive vs. neg-ative neurofeedback creates. The experimentsconducted on the initial prototype game devel-oped revealed that in the tested setup no sig-nificant difference in mean alpha band powerand in user experience between the gamesapplying positive and negative feedback werefound. Furthermore alpha power before SSVEPstimulation was significantly higher than alphapower during SSVEP stimulation. Thereforeone needs to consider this when combiningthese two paradigms for control.

Bacteria Hunt is suitable for use as an ex-perimental platform for BCI research. We havethe intention to continue experimenting on thisplatform, especially within multimodality andinteraction domains.

ACKNOWLEDGMENTSThe authors would like to thank Boris Reud-erink for his help with the BCI pipeline.

This research has been supported by theGATE project, funded by the Netherlands Or-ganization for Scientific Research (NWO) andthe Netherlands ICT Research and InnovationAuthority (ICT Regie), by the BrainGain SmartMix Programme of the Netherlands Ministryof Economic Affairs and the Netherlands Min-istry of Education, Culture and Science, andby the European Community’s Seventh Frame-work Programme (FP7/2007-2013) under grantagreement no. 231287 (SSPNet).

REFERENCES[1] E. Aarseth. Playing research: Methodological approaches

to game analysis. In Proceedings of DAC, Melbourne,Australia, 2003.

[2] A. Brouwer and J. Van Erp. A tactile P300 BCI andthe optimal number of tactors: Effects of target proba-bility and discriminability. In Proceedings of the 4th In-ternational Brain-Computer Interface Workshop and TrainingCourse 2008, pages 280–285, Graz, Austria, 2008.

[3] J. Cantero, M. Atienza, C. Gómez, and R. Salas. Spectralstructure and brain mapping of human alpha activities indifferent arousal states. Neuropsychobiology, 39(2):110–116,1999.

[4] G. Chatrian, L. Bergamini, M. Dondey, D. Klass,M. Lennox-Buchthal, and I. Petersen. A glossary of termsmost commonly used by clinical electroencephalogra-phers. Electroencephalography and Clinical Neurophysiology,37:538–548, 1974.

[5] B. A. Cohen and A. Sances, Jr. Stationarity of the humanelectroencephalogram. Medical and Biological Engineeringand Computing, 15(5):513–518, 1977.

[6] M. Comercho and J. Polich. P3a and p3b from typicalauditory and visual stimuli. Clinical Neurophysiology,110(1):24–30, January 1999.

[7] M. Doppelmayr, W. Klimesch, W. Stadler, D. Pöllhuber,and C. Heine. EEG alpha power and intelligence. Intelli-gence, 30(3):289–302, 2002.

[8] L. A. Farwell and E. Donchin. Talking off the top ofyour head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and ClinicalNeurophysiology, 70(6):510–523, December 1988.

[9] C. Herrmann. Human EEG responses to 1–100 Hz flicker:resonance phenomena in visual cortex and their potentialcorrelation to cognitive phenomena. Experimental BrainResearch, 137(3):346–353, 2001.

[10] I. S. Hjelm and C. Browall. Brainball - using brain activityfor cool competition. In Proceedings of the first nordicconference on HCI, Stockholm, Sweden, October 2000.

[11] W. IJsselsteijn, Y. de Kort, K. Poels, A. Jurgelionis, andF. Bellotti. Characterising and measuring user experiencesin digital games. In International Conference on Advancesin Computer Entertainment Technology, Salzburg, Austria,2007.

[12] B. Jansen, A. Allam, P. Kota, K. Lachance, A. Osho, andK. Sundaresan. An exploratory study of factors affectingsingle trial P300 detection. IEEE Transactions on BiomedicalEngineering, 51:975–978, 2004.

[13] H. H. Jasper. The ten-twenty electrode system of theinternational federation. Electroencephalography and clinicalneurophysiology, 10:371–375, 1958.

[14] C. Klimmt, T. Hartmann, and A. Frey. Effectance andcontrol as determinants of video game enjoyment. Cy-berPsychology & Behavior, 10(6):845–848, 2007.

[15] D. J. McFarland, L. M. McCane, S. V. David, and J. R.Wolpaw. Spatial filter selection for EEG-based communi-cation. Electroencephalography and Clinical Neurophysiology,103(3):386–394, 1997.

[16] K. Müller and B. Blankertz. Toward noninvasive brain-computer interfaces. IEEE Signal Processing Magazine,23(5):125–128, 2006.

[17] K. Müller, M. Krauledat, G. Dornhege, G. Curio, andB. Blankertz. Machine learning techniques for brain-computer interfaces. Biomedical Engineering, 49(1):11–22,2004.

[18] G. Müller-Putz, R. Scherer, C. Brauneis, andG. Pfurtscheller. Steady-state visual evoked potential(SSVEP)-based communication: impact of harmonic


frequency components. Journal of neural engineering,2(4):123–130, 2005.

[19] A. Nijholt, B. Reuderink, and D. Plass-Oude Bos. Turningshortcomings into challenges: Brain-computer interfacesfor games. In Intelligent Technologies for Interactive Enter-tainment, pages 153–168, May 2009.

[20] A. Nijholt, D. Tan, G. Pfurtscheller, C. Brunner, J. D. R.Millán, B. Allison, B. Graimann, F. Popescu, B. Blankertz,and K.-R. Müller. Brain-computer interfacing for intelli-gent systems. IEEE Intelligent Systems, pages 76–83, 2008.

[21] D. Regan. Human brain electrophysiology: Evoked potentialsand evoked magnetic fields in science and medicine. Appleton& Lange, 1989.

[22] B. Reuderink. Games and brain-computer interfaces: Thestate of the art. Technical report, HMI, University ofTwente, 2008.

[23] B. Reuderink. golemml: Python machinelearning library for EEG processing, October 2009.http://code.google.com/p/golemml.

[24] B. Reuderink. psychicml: Python signal processinglibrary for EEG processing, October 2009.http://code.google.com/p/psychicml.

[25] R. Scherer, A. Schloegl, F. Lee, H. Bischof, J. Janša, andG. Pfurtscheller. The self-paced graz brain-computerinterface: Methods and applications. Computational Intel-ligence and Neuroscience, 2007.

[26] M. Schürmann and E. Başar. Functional aspects of alphaoscillations in the EEG. International Journal of Psychophys-iology, 39(2-3):151–158, 2001.

[27] P. D. Tyson. Task-related stress and EEG alpha biofeed-back. Applied Psychophysiology and Biofeedback, 12(2):105–119, 1987.

[28] B. L. A. van de Laar, D. Oude Bos, B. Reuderink, andD. K. J. Heylen. Actual and imagined movement in BCIgaming. In Proceedings of the International Conference onArtificial Intellingence and Simulation of Behaviour (AISB2009), Aberdeen, Scotland, 2009.

Christian Mühl earned his Master’s de-gree in the Cognitive Sciences in 2007 atthe University of Osnabrück, Germany. Hiseducation was focused on neuroscientificmethods, specifically electroencephalogra-phy. Since then he is working as a researchassistant in the Human Media InteractionGroup at the University of Twente, TheNetherlands. In his PhD thesis he searches

for neurophysiological and physiological correlates of affectivestates in the context of brain-computer interaction.

Hayrettin Gürkök received his B.S. andM.S. degrees in Computer Engineeringfrom Bilkent University, Turkey. His M.S.thesis was on linguistics and text retrieval.He is currently a member of the HumanMedia Interaction group at the University ofTwente. He is conducting research on mul-timodality and realistic settings for BCIs.His research interests also include human-

computer interaction, social computing and information re-trieval.

Danny Plass-Oude Bos got her first ex-perience with online BCI during an intern-ship at the University of Nijmegen in 2007.There she implemented physiological arti-fact detection in an online EEG-based BCIsystem. In 2008 she obtained her masterdegree in Human Computer Interaction,looking into the user experience of usingBCI for games. At the moment she is work-

ing as a PhD student at the University of Twente, still attemptingto merge BCI with HCI by researching how BCI can be made amore intuitive means of interaction.

Marieke E. Thurlings started as a PhDcandidate at TNO (The Netherlands Or-ganization for Applied Scientific Research)and at the Utrecht University, in 2008.She currently investigates navigating inVirtual Environments using brain signalsas an input, involving the fields of Brain-Computer Interfacing, Games and VirtualWorlds. Marieke is especially interested in

using event-related potentials, such as P300s and SSEPs, andexploring covert attention to probe stimuli to evoke these, e.g. bypresenting them in unconventional modalities. Marieke obtainedher Master of Science degree in Design for Interaction from theDelft University of Technology.

Lasse Scherffig is a PhD student at theLab3, Laboratory for Experimental Com-puter Science of the Academy of MediaArts Cologne (Germany). Since July 2006he also is a member of the artistic/scientificstaff of the academy. He graduated in cog-nitive science at the university of Osnabrck(Germany) and received a M.Sc. in digitalmedia from the University of Bremen (Ger-

many). His dissertation project focuses on the relation of actionand perception in Human-Computer Interaction.


Matthieu Duvinage graduated as an elec-trical engineer in signal processing andtelecommunications from the Facult Poly-technique de Mons (Belgium) and SUP-ELEC (France) in june 2009. He also got amaster of fundamental and applied physicsfrom the University of Orsay (France) injune 2009. He has just started a PhD atthe Signal Processing and Circuit Theory

Lab at the University of Mons (Belgium) on the development ofa neuroprothesis to control an artificial leg via BCI.

Alexandra A. Elbakyan graduated fromKazNTU with a Bachelor degree in IT inJune 2009. She conducted a study regard-ing person identification by EEG in her finalyear thesis. She is going to continue herresearch in brain-computer interfaces andbrain implants.

SungWook Kang received the B.S. de-gree in electronic communication engineer-ing from the Kwangwoon University, Seoul,Korea, in 2009. He is currently workingtowards the M.S. and Ph.D degree atBiocomputing Lab., Dept. Information andCommunication, Gwangju Institute Of Sci-ence and Technology. His research inter-ests include bio-signal processing espe-

cially EEG and MEG, feature extraction, information theory andbrain-computer interface.

Mannes Poel is assistant professor in theHuman Media Interaction group at the Uni-versity of Twente. His main research in-volves applied machine learning for visionbased detection and interpretation of hu-man behavior and the analysis and classi-fication EEG based brain signals.

Dirk Heylen is associate professor in theHuman Media Interaction group at theUniversity of Twente where his researchinvolves the automatic interpretation ofverbal and nonverbal communication andthe of modeling conversational and cogni-tive functions of embodied conversationalagents. His work on the analysis and syn-thesis of nonverbal communication in (mul-

tiparty) conversations has been concerned with gaze, and headmovements in particular. His research interests extend to thestudy of physiological signals that can be used to interpret themental state of user.

workshop report for the enterface workshop ......alpha power before ssvep stimulation was...

Documents