School of Biomedical Engineering, Science, and Health Systems

Drexel E-Repository and Archive (iDEA)

http://idea.library.drexel.edu/

Drexel University Libraries www.library.drexel.edu

The following item is made available as a courtesy to scholars by the author(s) and Drexel University Library and may contain materials and content, including computer code and tags, artwork, text, graphics, images, and illustrations (Material) which may be protected by copyright law. Unless otherwise noted, the Material is made available for non profit and educational purposes, such as research, teaching and private study. For these limited purposes, you may reproduce (print, download or make copies) the Material without prior permission. All copies must include any copyright notice originally included with the Material. You must seek permission from the authors or copyright owners for all uses that are not allowed by fair use and other provisions of the U.S. Copyright Law. The responsibility for making an independent legal assessment and securing any necessary permission rests with persons desiring to reproduce or use the Material.

Please direct questions to archives@drexel.edu

38 IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

OPT

ICA

L IM

AG

ING

0739-5175/07/$25.00©2007IEEE

In the last decade, functional near-infrared spectroscopy(fNIR) has been introduced as a new neuroimaging modali-ty with which to conduct functional brain imaging studies[1]–[24]. fNIR technology uses specific wavelengths of

light, irradiated through the scalp, to enable the noninvasivemeasurement of changes in the relative ratios of deoxygenatedhemoglobin (deoxy-Hb) and oxygenated hemoglobin (oxy-Hb)during brain activity. This technology allows the design ofportable, safe, affordable, noninvasive, and minimally intru-sive monitoring systems. These qualities make fNIR suitablefor the study of hemodynamic changes due to cognitive andemotional brain activity under many working and educationalconditions, as well as in the field.

Functional imaging is typically conducted in an effort tounderstand the activity in a given brain region in terms of its rela-tionship to a particular behavioral state or its interactions withinputs from another region’s activity. The program of research incognitive neuroscience conducted by our optical brain imaginggroup has utilized the current-generation fNIR system to investi-gate brain activity, primarily in the dorsolateral and inferiorfrontal cortex [20]–[24]. To date, the fNIR studies of cognitionand emotion have focused on functions associated withBrodman’s areas BA9, BA10, BA46, BA45, BA47, and BA44.Recent positron emission tomography (PET) and functionalmagnetic resonance (fMRI) studies have shown that these areasplay a critical role in sustained attention, both the short-term stor-age and the executive process components of working memory,episodic memory, problem solving, response inhibition, and theperception of smell (for a recent review, see [25] and [26]). Inaddition, word recognition and the storage of verbal materialsactivate Broca’s area and left hemisphere supplementary andpremotor areas [25], [27], [28]. To date, studies utilizing fNIRhave shown results consistent with fMRI and PET findings forworking memory and sustained attention [21]–[23].

In this article, we will describe the working principles offNIR and how the hemodynamic signals are extracted fromthe raw fNIR measurements using the modified Beer-LambertLaw. We will also introduce the fNIR system that we havedeveloped and used in our studies. Current results from theaugmented cognition research conducted in our laboratory arealso presented, and the merits of optical imaging in augmentedcognition are summarized.

Working PrinciplesTypically, an optical apparatus consists of a light source bywhich the tissue is radiated and a light detector that receiveslight after it has interacted with the tissue. Photons that entertissue undergo two different types of interaction, namelyabsorption (loss of energy to the medium) and scattering [4],[5], [19]. Most biological tissues are relatively transparent tolight in the near-infrared range between 700 to 900 nm, whichis usually called the “optical window.” This is mainly due tothe fact that within this optical window, the absorbance of themain constituents in the human tissue (i.e., water, oxy-Hb, anddeoxy-Hb) is small, allowing the light to penetrate the tissue(see Figure 1).

Among the main absorbers (chromophores) in the tissue,oxy- and deoxy-Hb are strongly linked to tissue oxygenationand metabolism. Fortunately, in the optical window, theabsorption spectra of oxy- and deoxy-Hb remain significantlydifferent than each other, allowing spectroscopic separation ofthese compounds to be possible using only a few samplewavelengths.

fNIR technology employs specified wavelengths of lightwithin the optical window. Once the photons are introducedinto the human head, they are either scattered by extra- andintracellular boundaries of different layers of the head (skin,skull, cerebrospinal fluid, brain, etc.) or absorbed mainly byoxy- and deoxy-Hb. A photodetector placed a certain distanceaway from the light source can collect the photons that are notabsorbed and those that traveled along the “banana shapedpath” between the source and detector due to scattering [9],[29] as shown Figure 2.

In functional optical brain imaging studies, the attenuation(reduction in the amount of photons detected by the photode-tectors) due to scattering is assumed to be constant since theamount of scatterers within different layers of the head doesnot change due to cognitive activity. The change in the attenu-ation measured as a result of cognitive activity is hence due tothe changes in absorption resulting from the variation in theconcentrations of oxy- and deoxy-Hb in the brain tissue. Thisrelationship is not surprising, since cerebral hemodynamicchanges are related to functional brain activity through amechanism that is called neurovascular coupling [8], [30]. Infact, this physiological relationship and the ability of fNIR

Functional Brain ImagingUsing Near-InfraredTechnologyAssessing Cognitive Activity in Real-Life Situations

BY MELTEM IZZETOGLU, SCOTT C. BUNCE,KURTULUS IZZETOGLU, BANU ONARAL,AND KAMBIZ POURREZAEI

© BRAND X PICTURES

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

technology to measure the concentration changes in the oxy-gen-related chromophores make the functional optical brainimaging possible.

According to the modified Beer-Lambert Law [2]–[4], [19],the light intensity after absorption and scattering by the bio-logical tissue is expressed as:

I = GEoe−(αHBCHB+αHBO2 CHBO2)L (1)

where G is a factor that accounts for the measurement geome-try and is assumed constant when concentration changes. Io isinput light intensity; αHB and αHBO2 are the molar extinctioncoefficients; CHB and CHBO2 are the concentrations of chro-mophores deoxy-Hb and oxy-Hb, respectively; and L is thephoton path that is a function of absorption and scatteringcoefficients µa and µb.

By measuring optical density (OD) changes at two wave-lengths, the relative change of oxy- and deoxy-hemoglobinversus time can be obtained. If the intensity measurementat the initial time (baseline) is Ib, and at another time is I,the OD change due to variation in CHB and CHBO2 duringthat period is:

�OD = log

(Ib

I

)= αHB�CHB + αHBO2�CHBO2 . (2)

Measurements performed at two different wavelengths allowthe calculation of �CHB and �CHBO2 . Oxygenation and bloodvolume can then be deduced:

Oxygenation = �CHBO2 − �CHB (3)

BloodVolume = �CHBO2 + �CHB. (4)

Using this technique and these measures, several types ofbrain functions have been assessed, including motor [10], [11]and visual activation [15]; auditory stimulation [17]; andperformance of various cognitive tasks [21]–[23]. In our stud-ies described in this article, we used the oxygenation data forthe assessment of different cognitive functions.

fNIR SystemThree distinct types of fNIR implementation have been devel-oped: time-domain, frequency-domain, and continuous wave(CW) spectroscopy systems [3]–[5]. In time-domain systems,also referred to as time-resolved spectroscopy (TRS), extreme-ly short incident pulses of light are applied to the tissue andthe temporal distribution of photons that carry the informationabout tissue scattering and absorption is measured. In frequen-cy-domain systems, the light source is amplitude-modulated tothe frequencies in the order of tens to hundreds of megahertz.The amplitude decay and phase shift of the detected signalwith respect to the incident light are measured to characterizethe optical properties of tissue [31].

In CW systems, light is applied to tissue at constant ampli-tude. The CW systems are limited to measuring the amplitudeattenuation of the incident light [31]. However, CW systemspossess a number of advantageous properties that have result-ed in wide use among researchers interested in brain imagingrelative to other near-infrared systems: it is minimally intru-

sive and portable, affordable, and easy to engineer relative tofrequency- and time-domain systems [31], [32]. Our researchteam has been developing a CW fNIR system that lends itselfto both portable and wireless designs to monitor brain func-tion under both laboratory and field conditions.

In the studies described throughout this article, we used theportable CW-fNIR system that was originally described byChance et al. [32]. The main components of the system are: 1)the sensor that covers the entire forehead, 2) a control box fordata acquisition (current sampling rate is 1.6 Hz), and 3) acomputer for the data analysis software [23], [24]. The wire-less CW-fNIR system operating with different sampling ratesis currently under investigation.

The current flexible sensor consists of four light sources thatcontain three built-in LEDs having peak wavelengths at 730,805, and 850 nm and ten detectors designed to image corticalareas underlying the forehead (dorsolateral and inferior frontalcortices). With a fixed source-detector separation of 2.5 cm,this configuration results in a total of 16 signal channels (vox-els). Communication between the data analysis computer andthe task presentation computer is established via a serial portconnection to time-lock fNIR measurement to the task events.

The flexible sensor is a modular design consisting of twoparts: a reusable, flexible circuit board that carries the necessary

Fig. 1. Absorption spectrum in near-infrared (NIR) window.

Fig. 2. Photon path inside the human head.

Optical Window

0.30

0.25

0.20

0.15

0.10

0.05

0600 700 800 900 1,000 1,100

Wavelength (nm)

Aes

orpt

ion

Fac

tor

(cm

−1)

Deoxy-Hb

Oxy-Hb

Water

Source

Detector

39

40

infrared sources and detectors, and a disposable, single-usecushioning material that serves to attach the sensor to the partic-ipant (see Figure 3) [21]. The flexible circuit provides a reliable,integrated wiring solution, as well as consistent and repro-ducible component spacing and alignment. Because the circuitboard and cushioning materials are flexible, the componentsmove and adapt to the contours of the participant’s head, allow-ing the sensor elements to maintain an orthogonal orientation tothe skin surface, improving light coupling efficiency and signalstrength. We are currently working on a modular sensor designthat will be scalable to any forehead shape and size, including

adult and infants, allowing the adjustment of the sources anddetectors according to the international 10–20 system.

fNIR Studies on Augmented CognitionIn all our fNIR studies presented in this section, participantshave signed informed consent statements approved by theHuman Subjects Institutional Review Board at DrexelUniversity.

Cognitive Performance Measurement Study In this study, we present the deployment and statistical analy-sis of fNIR for the purpose of cognitive state assessment whilethe user performs a complex task [21]. This work is based ondata collected during the Defense Advanced Research ProjectsAgency (DARPA) Augmented Cognition-TechnicalIntegration Experiment session participated by a total of eighthealthy subjects (three females and five males), ranging in agefrom 18 to 50.

The experimental protocol for this session used a complextask resembling a videogame called the Warship CommanderTask (WCT). The WCT was designed and developed byPacific Science and Engineering Group under the direction ofthe Space and Naval Warfare Systems Center to simulatenaval air warfare management [33]. A sample screen shot dur-ing WCT is shown in Figure 4.

Task load and task difficulty were manipulated by changingthe number of airplanes that had to be managed at a given time(six, 12, 18, and 24 plane “waves”), the number of unknownversus known airplane identities (two levels of difficulty, low:33% of the planes were unknown, and high: 67% of the planeswere unknown), and the presence or absence of a verbal mem-ory task (a secondary task causing divided attention).

For this study, each participant completed four sets ofWCT. Each set was comprised of three repetitions of each ofthe four wave sizes (in the order of six, 18, 12, and 24 planes),where each wave lasted 75 s. The factors of four differentwave sizes, two different task difficulties (high versus low per-centage of unknown airplanes), and full versus divided atten-tion (secondary verbal memory task “on” or “off”) werecrossed to create a 4 × 2 × 2 repeated-measures design.

The fNIR measurements are first cleaned from motion arti-facts [21], then for each wave of 75 s, the rate of change inthe oxygenation was calculated from the fNIR measurementsrelative to the baseline collected during the rest period beforethe protocol had started. Finally, blood oxygenation valueswere averaged across eight voxels covering left and righthemispheres.

The fNIR data analysis explored the relationships amongcognitive workload, the participant’s performance andchanges in blood oxygenation levels of the dorsolateral pre-frontal cortex, and the effect of divided attention as elicited bythe secondary component of the WCT (the auditory task). Ourprimary hypothesis was that blood oxygenation in the pre-frontal cortex, as assessed by fNIR, would rise with increasingtask load and would exhibit a positive correlation with perfor-mance measures. In support of our primary hypothesis, theresults indicated that the rate of change in blood oxygenationwas significantly sensitive across both hemispheres(F = 16.24, p < 0.001) to task load (wave size) changes [seeFigure 5(a) when secondary verbal was off].

When attention is divided by the secondary verbal task, the pri-mary effect occurred in the 24-plane wave (the most difficult

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

Fig. 3. (a) Flexible sensor. (b) Participant wearing flexible sensor.

Fig. 4. A snapshot during WCT where air warfare manage-ment required the user to monitor “waves” of incoming air-planes, to identify the identity of the unknown planes(yellow) as friendly (blue) or hostile (red), and to warn andthen destroy hostile airplanes using rules of engagement.

(a)

(b)

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

condition) causing the mean oxygenation for this case to dropbelow that of the 18-plane wave [see Figure 5(a)]. In line with thestated hypothesis, a preliminary interpretation of this finding wasthat a number of participants had reached their maximal level ofperformance in this most difficult task level and lost their concen-tration/effort, resulting in a drop in blood oxygenation.

The hypothesis also predicts that individuals who were ableto stay on task and continue to perform in this difficult condi-tion should demonstrate increased oxygenation relative to boththeir own oxygenation levels in the 18-plane wave and indi-viduals who became overwhelmed and disengaged. Becausesustained concentration and engagement in the task shouldresult in increased performance, a positive correlation betweenperformance and blood oxygenation would provide supportfor this interpretation. A Pearson’s product-moment correla-tion indicated a very strong positive relationship betweenblood oxygenation and performance in the 24-plane condition[Pearson’s r = 0.89, p = 0.003; see Figure 5(b)].

A median split on the Percentage Game Score provided fur-ther evidence of the hypothesized relationship between cogni-tive effort and the blood oxygenation response. As can be seen

in Figure 5(c), the mean levels of oxygenation were higher forboth high and low performers in the 24-plane wave than the18-plane wave when the secondary verbal task was off.However, when the secondary verbal task was on, for themore difficult condition, the individuals who performed wellon the 24-plane wave showed a higher mean level of oxygena-tion for the 24-plane wave than for the 18-plane wave, where-as those who performed poorly showed a decrease inoxygenation relative to the 18-plane wave.

Working Memory Assessment StudyIn order to assess the working memory, we used the n-backtask, which is a sequential letter task with varied workloadconditions that has frequently been used in working memorystudies by cognitive psychologists and neuroscientists [26],[27]. The stimuli are single consonants presented centrally, inpseudorandom sequences, on a computer monitor. Stimulusduration is 500 ms, with a 2,500-ms interstimulus interval.Four conditions were used to incrementally vary workingmemory load from zero to three items. In the 0-back condi-tion, subjects respond to a single prespecified target letter

Fig. 5. (a) Mean oxygenation change versus wavesize (n=8) by secondary verbal task. (b) Pearson’s correlation betweenperformance and oxygenation change. (c) Mean oxygenation change as a function of wavesize, secondary verbal task andaverage Percentage of Game Score (from [21]).

Oxy

gena

tion

Cha

nge

2nd Verbal On2nd Verbal Off

121086420

−2−4−6−8

Low Performance High

6 12 18 24 6 12 18 24

86420

−2−4−6−8O

xyge

natio

n C

hang

e

Wavesize

6 12 18 24

2nd VerbalHigh2nd VerbalOff

(a)

Oxy

gena

tion

Cha

nge

20

10

0

−1008 10 12 14 16 18 20 22

% of Total Possible Score

Rsq − 0.7899

(b)

(c)

These studies indicate that human performance

and cognitive activities such as attention,

working memory, problem solving, etc.,

can be assessed by fNIR technology.

41

42

(e.g., “X”) with their dominant hand (pressing a button toidentify the stimulus). In the 1-back condition, the target isdefined as any letter identical to the one immediately preced-ing it (i.e., one trial back). In the 2-back and 3-back condi-tions, the targets were defined as any letter that was identicalto the one presented two or three trials back, respectively.Subjects pressed one button for targets (approximately 33% oftrials) and another for nontargets. This strategy incrementallyincreased working memory load from the 0-back to the 3-backcondition. Each n-back block contained 20 letters, whether tar-get or nontarget, and lasted for 60 s with 15 s of rest periodsbetween n-back blocks. The total test included seven trials ofeach of the four n-back conditions (hence, a total of 28 n-backblocks) ordered in such a way that within one trial all four ofthe n-back conditions are presented; however, their order ischanged randomly from trial to trial.

In the analysis performed on nine subjects (with agebetween 18 and 25), filtering to eliminate physiologicallyirrelevant data (such as respiration and heart pulsation effects)and equipment noise, etc., is carried out on the raw fNIRmeasurements. Then, for each n-back condition out of seventrials, outliers are eliminated and the resulting trials are aver-aged for each voxel. Once the oxygenation data is obtainedusing the modified Beer-Lambert Law on these averaged rawdata, the overall mean for each n-back block is calculated andused as a feature for comparison purposes.

Statistically significant differences between the n-back con-ditions are obtained on the fourth-voxel fNIR measurements.The location of the fourth-voxel measurement registered onthe brain surface is as shown in Figure 6(a). (Interested readerscan find a more detailed explanation of our fNIR data registra-tion and visualization scheme in [34].) This result is in agree-ment with the fMRI literature [28].

Statistical analysis revealed that the 0-back condition dif-fered from 1-, and 2-back conditions; 1-back > 0-back,

t = 3.21, p = 0.012; 2-back > 0-back, t = 2.58, p = 0.032.The 1- to 2-back did not differ from each other at first.However, we noticed that in the 2-back condition subject 5had performed the worst compared to the others [as shown inFigure 6(b)], which was more than 1.5 standard deviationaway from the overall mean. When the data of subject 5 aretreated as an outlier and eliminated from the analysis, 1- and2-back differed from each other; 2-back > 1-back, t = 2.77,p = 0.0275. No difference was found between 2- to 3-backconditions.

Mean oxygenation data for the nine subjects’ individualand averaged mean oxygenation data across nine subjects foreach workload condition are presented in Figure 6(c) andFigure 6(d), respectively. A positive relationship betweenincreasing workload and the oxygenation is observed in dor-solateral prefrontal cortex, again in agreement with fMRIstudies [28]. The drop in the oxygenation values in the mostdifficult condition (3-back) can be interpreted using a hypoth-esis similar to the human performance study discussed previ-ously, where subjects get overwhelmed and lose theirconcentration.

Problem Solving Study Using a Novel Single-TrialHemodynamic Response Extraction MethodOngoing studies in problem solving of graded difficulties(anagram solution) using both block and event-related (ER)anagram protocols reveal that fNIR measurement of metabolicactivation and blood flow can be valuable as an educationalaid [35], [36]. In ER anagram study, subjects are presented ananagram for 1 s and given 15 s to solve it until the next presen-tation. This procedure allows the hemodynamic response tofully evolve, which has been shown in the literature to take10–12 s [36], [37]. ER studies provide insights to model thehemodynamic response and are used widely in the assessmentof cognitive activation in different regions of the brain for

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

Fig. 6. N-back test results: (a) Imaging area of the brain. (b) Performance of all subjects on 2-back condition. (c) Mean oxy-genation across subjects. (d) Averaged mean oxygenation for all subjects.

(a)

Performance on 2-Back Condition Across Subjects 1.200

1.000

0.800

0.600

0.400

0.200

0.0001 2 3 4 5 6 7 8 9

(b)

Mean Oxygenation Across Subjects (n=9)0.5000.4000.3000.2000.100

−0.100−0.200−0.300−0.400−0.500−0.600

0.0001 2 3 4 5 6 7 8 9

0-Back1-Back2-Back3-Back

(c)

Averaged Mean Oxygenation for Nine Subjects0.150

0.100

0.050

0.000

−0.050

−0.100

−0.150

−0.200

10-Back1-Back2-Back3-Back

(d)

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

different task loads. However, in such studies, the protocoltime is long and they do not reflect real-world situations.

In block anagram study, subjects are shown as many ana-grams as they can solve within 1-minute periods. Wheneversubjects solve the anagram, they press a certain button, whichresults in immediate presentation of the next anagram. Sincemost subjects solve the anagram within 2–5 s, the hemody-namic responses overlap in time, which present challenges fordata analysis. Until now, in block anagram studies, it was notpossible to evaluate the subject’s response times or brain acti-vation for single anagram presentation within a block forgraded difficulty analysis.

We developed a novel single-trial hemodynamic response esti-mation algorithm and applied it to the block anagram solutionstudy to extract evoked responses to single anagram presentationswithin each block [36]. Each ER hemodynamic response wasestimated on the basis of two postulates: 1) that each single-trialhemodynamic response follows a gamma function,hfi = Ait

αii eβi ti as given in Figure 7(a), and 2) that the total oxy-

genation data can be modeled by the summation of individualhemodynamic responses evoked by rapidly presented stimuli,Oxy = ∑N

i=1 hfi. Each single trial was estimated by optimizing

the error between the total oxygenation data from fNIR measure-ments and the linear model: ε = minA,α,β (Oxy − ∑N

i=1 hfi)2 . The protocol we have used in this study involved presenta-

tion of anagram blocks on a computer screen that containssequences of three-letter (3L), four-letter (4L), and five-letter(5L) anagrams starting from minimal (3L anagrams) proceed-ing to the maximal level of difficulty (5L anagrams) and thenback down again to the starting point of 3L anagrams.Between each anagram block session, there is a rest period of30 s. Each anagram block is displayed for approximately oneminute, containing as many anagrams within it depending onthe number of processed anagrams by the subject. The deci-sion of the subjects on each anagram processed and its timingis recorded on a text file for further analysis.

All calculations are applied to the data gathered from theleft hemisphere of the prefrontal cortex on voxel 5 [locationshown in Figure 7(b)]. In a block anagram study based on 14participants (age 18 to 23), the averaged recorded behavioralresponse times, the extracted rise times or time to peak (min),and the maximum amplitudes from the estimated evokedhemodynamic responses with respect to the 3L, 4L, and 5Lanagram sets are presented in Figure 7(c).

Fig. 7. (a) A typical gamma function. (b) Imaging area of the brain for this study. (c) Subject averages of rise and responsetimes (min) and maximum amplitudes. (d) Scatter plot of rise time versus response time averages. (e) Scatter plot of maximumamplitude versus response time averages for all anagram sets for all subjects (from [36]).

876543210

−1−2−3

0 10 20 30 40 60

FWHM Amplitude

Time to Peak

(a)

All Anagram Data, R=0.73

Max

imum

Am

plitu

de

0.400

0.350

0.300

0.250

0.200

0.150

0.100

0.050

0.0000.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000

Response Time (s)

(e)

All Anagram Data, R=0.9410.0009.0008.0007.0006.0005.0004.0003.0002.0001.0000.000

0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000

Ris

e Ti

me

(s)

Response Time (s)

(d)(c)

0.35

0.25

0.15

0.05

0.1

0.3

0.2

03L 1st Set 4L 1st Set 4L 2nd Set 3L 2nd Set5L

Average Response and Rise Times (min) and Maximum Amplitude of theSingle Trial Hemodynamic Response Estimates

Ave Resp Time

Ave Rise TimeAve Max Time

(b)

43

44

It can be clearly seen that the estimated rise time, which isthe time required for the evoked hemodynamic response toreach its maximum amplitude, follows the same pattern as thebehavioral (true) response time of the subjects having a corre-lation of R = 0.94 as presented in the scatter plot of the risetime versus response time in Figure 7(d). Also, the estimatedmaximum amplitudes are correlated with the true responsetimes (R = 0.73) as given in Figure 7(e).

The rise times and the maximum amplitude values increaseas the difficulty level of the anagram solution increases, mean-ing that subjects need more time and more oxygen to solvedifficult anagrams. Estimation of the ER signals in a blockdesign allows more precise analysis of the brain’s functionduring a cognitive/problem solving task.

Attention Measurement Study:A Combined EEG and fNIR StudyIn this last study we demonstrate the utility of the combinedEEG-fNIR system for studies of ER designs that tap into ubiq-uitous cognitive functions such as attention [38]. The protocolwe have used to measure attention is a common visual oddballparadigm modified for use with fMRI by McCarthy et al. [39].The stimuli were two strings of white letters (XXXXX andOOOOO) presented against the center of a dark background.A total of 516 stimuli were presented, 480 context stimuli

(OOOOO) and 36 targets (XXXXX). Stimulus duration was500 ms, with an interstimulus interval of 1,500 ms. Targetstimuli were presented randomly with respect to context stim-uli with a minimum of 12 context stimuli between successivetargets to allow the hemodynamic response an opportunity toreturn to baseline between target presentations.

Fifteen right-handed participants (four females and 11males, age 20.8 ± 4.2) were required to press one of two but-tons on a response pad after each stimulus while both fNIRand EEG were recorded simultaneously. One button waspressed in response to targets (Xs), and another button waspressed in response to context stimuli (Os).

The results for the event-related brain potentials (ERPs)were consistent with the literature [40]; targets elicited a pro-nounced P3 component with an average peak at 365 ms forboth electrodes Cz and Pz [see Figure 8(a) and Figure 8(b)].The peak amplitude response to target stimuli was larger thanthe response to context stimuli at both Cz (t(14) = 7.58;p < 0.001) and Pz (t(14) = 7.81; p < 0.001). These ERPresults confirm that the task parameters and participantresponses were comparable to other ERP studies.

Raw fNIR data was low-pass filtered at a frequency of0.14 Hz in order to eliminate respiration and heart pulsationeffects. The continuous oxygenation data were then segmentedinto 24-s-long epochs, with a prestimulus window of 9 s (six

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007

Fig. 8. Averaged ERP and fNIR data for targets and contexts: (a) and (b) ERP’s from Cz and Pz electrode, respectively. (c) fNIRdata on voxel 11. (d) Location of the significant difference in fNIR measurements between target and context from ([38]).

25

20

15

10

5

0

−5

−10−200 0 200 400 600 800 1,000

Time (ms)

Vol

tage

(µV

)

TargetContext

(a)

25

20

15

10

5

0

−5

−10−200 0 200 400 600 800 1,000

Time (ms)

Vol

tage

(µV

)

TargetContext

(b)

0.015

0.01

0.005

0

−0.005

−0.01

TargetContext

Oxy

gena

tion

(m)

−2 0 2 4 6 8 10 12Time (s)

(c) (d)

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007 45

stimuli) and a poststimulus window of 15 s (nine stimuli).These epochs were baseline corrected by subtracting the meanof the baseline from the waveform and then outliers wereeliminated for each voxel. The remaining epochs were aver-aged for the target and the context stimuli separately.

Repeated-measure ANOVA computed on the fNIR oxy-genation data revealed that oxygenation values were greaterin response to targets than to controls in voxel 11, locatedover the middle frontal gyrus of the right hemisphere [seeFigure 8(c)]. Differentiation occurred between 6 and 9 s post-stimulus [see Figure 8(d)]. These results are consistent withthe fMRI literature for visual target categorization withrespect to increased oxygenation in response to targets, corti-cal location, and time course [39], [41].

ConclusionsThe use of fNIR technology has increased in recent years as ameans to measure hemodynamic changes in the cortex inresponse to cognitive activity. Moreover, it is a noninvasiveand negligibly intrusive optical imaging modality. fNIRinstrumentation allows for safe, portable, and low-cost corticalmonitoring that can be applied in the laboratory as well asfield conditions. This article provided an overview of cogni-tive studies carried out in our laboratory. These studiesindicate that human performance and cognitive activities suchas attention, working memory, problem solving, etc., can beassessed by fNIR technology. Our findings are in agreementwith the results in current EEG and fMRI literature. We havedemonstrated that fNIR technology can be integrated with theERPs collected simultaneously with fNIR for better data clas-sification. Such integration benefits from the high temporalresolution characteristic of ERPs and better spatial resolutioncharacteristic of fNIR. We have automated signal processingalgorithms to solve problems that arise due to the time-scaledisparity between EEG and fNIR signals. Our studies suggestthat fNIR is a promising new technology for the study of cog-nitive activity.

AcknowledgmentsWe would like to thank Dr. Shoko Nioka and Dr. BrittonChance for their valuable research guidance in general andfor supplying in particular the anagram data. This work hasbeen sponsored in part by funds from the DARPAAugmented Cognition Program, the Office of NavalResearch, and the Office of Homeland Security under agree-ment numbers N00014-02-1-0524, N00014-01-1-0986, andN00014-04-1-0119.

Meltem Izzetoglu was born in Samsun,Turkey in 1971. She received the B.S.and the M.S. both in electrical and elec-tronics engineering from Middle EastTechnical University, Ankara, Turkey, in1992 and 1995, respectively. She re-ceived the Ph.D. in electrical and comput-er engineering from Drexel University,

Philadelphia, Pennsylvania, in 2002. She is currently aresearch assistant professor in School of BiomedicalEngineering, Science, and Health Systems at DrexelUniversity. Her research interests include biomedical signalanalysis, adaptive and optimal signal processing, biomed-ical optics, and scale-space processing tools.

Scott C. Bunce is an assistant professor ofpsychiatry and director of the ClinicalNeuroscience Research Unit at the DrexelUniversity College of Medicine. He hasconsiderable experience in both clinicaland individual differences research. Hisareas of expertise are in affective neuro-science, theory of mind, and the effects of

psychological trauma on information processing. A majorfocus in recent years has been on using neurophysiologicalmeasures (EEG, ERPs, ERD, EMG) to assess informationand emotional processing that cannot be reported by thepatient/participant. He has also played an integral role in thedevelopment of a safe, portable, near-infrared optical imag-ing device for the assessment of hemodynamic changes dur-ing cognitive and emotional tasks.

Kurtulus Izzetoglu gained his profession-al software development and medical imag-ing experience as a member of consultingcompanies in the United States and theNetherlands, respectively. In these posi-tions, he worked as a senior analyst as wellas a software and analytical applicationsdeveloper. His experiences include the

development of professional medical imaging software pack-ages, implementation of quantitative analysis, and imagingtechniques. Subsequent to five years of industrial experience,he joined the functional optical imaging research team atDrexel University where he currently serves as the projectengineer. His technical management responsibilities includedevelopment of the Cognitive Workload Assessment Testingand Analysis Platform and signal processing and experimentalprotocol design and implementation. He received his M.S.E.E.from Middle East Technical University in Ankara, Turkey.

Banu Onaral, H.H. Sun Professor ofBiomedical and Electrical Engineering,received her Ph.D. from the University ofPennsylvania in 1978 and her B.S.E.E. andM.S.E.E. from Bogazici University inIstanbul, Turkey. Her academic focus, bothin research and teaching, is centered on bio-medical signals and systems engineering.

She has been a founding member of the BiomedicalInformation Technology Laboratory, Scaling Signals andSystems Laboratory, and the Bio-Electrode ResearchLaboratory. She has led several curriculum development ini-tiatives including the undergraduate telecommunication andbiomedical engineering programs. She has developed severalsignals and systems engineering software products and wasrecognized by the EDUCOM/NCRIPTAL Best EducationalTool Award. She is the recipient of a number of faculty excel-lence awards including the 1990 Lindback DistinguishedTeaching Award of Drexel University. Her professional ser-vices include chair and membership on advisory boards andstrategic planning bodies of several universities and fundingagencies, including service on the NSF Engineering AdvisoryBoard (1997–1999) and on the proposal review panels andstudy sections. Her editorial responsibilities have included ser-vice on the editorial board of journals and the CRCBiomedical Engineering Handbook as section editor for the

46

“Biomedical Signal Analysis” section. She has been active inprofessional society leadership, in particular national andinternational technical meeting organization; she served asvice president for conferences and as president of the IEEEEngineering in Medicine and Biology. She also served on theinaugural board of the American Institute for Medical andBiological Engineering. She is a Fellow of the IEEE; foundingFellow of the American Institute for Medical and BiologicalEngineering (AIMBE), Fellow of the American Associationfor the Advancement of Science (AAAS), senior member ofthe Society of Women Engineers (SWE), member of theAmerican Society for Engineering Education (ASEE), and theSigma Xi scientific research society.

Kambiz Pourrezaei received his B.S.from Tehran University and M.S. fromTufts University. He earned his Ph.D.from Rensselaer Polytechnic Institute in1982. He is currently a professor with theSchool of Biomedical Engineering,Science, and Health Systems at DrexelUniversity. He has active research pro-

grams in the areas of bio-nanotechnology and bio-optics.Currently he is the co-director of the NanotechnologyInstitute in Philadelphia, Pennsylvania.

Address for Correspondence: M. Izzetoglu, DrexelUniversity, School of Biomedical Engineering, Science andHealth Systems, 3141 Chestnut Street, Philadelphia, PA19104. E-mail: meltem@cbis.ece.drexel.edu.

References[1] F.F. Jobsis, “Noninvasive infrared monitoring of cerebral and myocardial suffi-ciency and circulatory parameters,” Science, vol. 198, no. 4323, pp. 1264–1267,1977.[2] M. Cope and D.T. Delpy, “System for long-term measurement of cerebralblood flow and tissue oxygenation on newborn infants by infra-red transillumina-tion,” Med. Biol. Eng. Comput., vol. 26, no. 3, pp. 289–294, 1988.[3] Q. Luo, S. Zeng, B. Chance, and S. Nioka, “Monitoring of brain activation withnear infrared spectroscopy,” in Handbook of Optical Biomedical Diagnostics, V.V.Tuchin, Ed. SPIE Press Monograph PM107. [4] P. Rolfe, “In vivo near-infrared spectroscopy,” Annu. Rev. Biomed. Eng., vol. 2,pp. 715–754, 2000.[5] G. Strangman, D.A. Boas, and J.P. Sutton, “Non-invasive neuroimaging usingnear-infrared light,” Biol. Psychiatry, vol. 52, no. 7, pp. 679–693, 2002.[6] Y. Hoshi and M. Tamura, “Dynamic multichannel near-infrared optical imag-ing of human brain activity,” J. Applied Physiol., vol. 75, no. 4, pp. 1842–1846,1993.[7] Y. Hoshi, “Functional near-infrared optical imaging: Utility and limitations inhuman brain mapping,” Psychophysiology, vol. 40, no. 4, pp. 511–520, 2003.[8] A. Villringer, J. Planck, C. Hock, L. Schleinkofer, and U. Dirnagl, “Nearinfrared spectroscopy (NIRS): A new tool to study hemodynamic changes duringactivation of brain function in human adults,” Neurosci. Lett., vol. 154, no. 1–2,pp. 101–104, 1993.[9] A. Villringer and B. Chance, “Non-invasive optical spectroscopy and imagingof human brain function,” Trends Neurosci., vol. 20, no. 10, pp. 435–442, 1997.[10] T. Suto, M. Ito, T. Uehara, I. Ida, M. Fukuda, and M. Mikuni, “Temporalcharacteristics of cerebral blood volume change in motor and somatosensory cor-tices revealed by multichannel near infrared spectroscopy,” Int. Congress Series,vol. 1232, pp. 383–388, Apr. 2002.[11] A. Maki, Y. Yamashita, Y. Ito, E. Watanabe, Y. Mayanagi, and H. Koizumi,“Spatial and temporal analysis of human motor activity by using noninvasive NIRtopography,” J. Neurosci., vol. 11, no. 12, pp. 1458–1469, 1995.[12] E. Gratton, V. Toronov, U. Wolf, M. Wolf, and A. Webb, “Measurement ofbrain activity by near infrared light,” J. Biologic. Opt., vol. 10, no. 1, p. 011008-1-13, 2005.[13] M.A. Franceschini and D.A. Boas, “Noninvasive measurement of neuronal activ-ity with near-infrared optical imaging,” Neuroimage, vol. 21, no. 1, pp. 372–386,2004.[14] G. Gratton, P.M. Corballis, E. Cho, M. Fabiani, and D.C. Hood, “Shades ofgray matter: Noninvasive optical images of human brain responses during visualstimulation,” Psychophysiology, vol. 32, no. 5, pp. 505–509, 1995.

[15] H.R. Heekeren, H. Obrig, R. Wenzel, K. Eberle, J. Ruben, K. Villringer, R.Kurth, and A. Villringer, “Cerebral haemoglobin oxygenation during sustainedvisual stimulation - A near infrared spectroscopy study,” Philos. Trans. R. Soc.Lond. B Biol. Sci., vol. 352, no. 1354, pp. 743–750, 1997.[16] H. Sato, T. Takeuchi, and K. Sakai, “Temporal cortex activation duringspeech recognition: An optical topography study,” Cognition, vol. 73, no. 3, pp.B55–B66, 1999.[17] P. Zaramella, F. Freato, A. Amigoni, S. Salvadori, P. Marangoni, and A.Suppjei, “Brain auditory activation measured by near-infrared spectroscopy,”Pediatric Res., vol. 49, no. 2, pp. 213–219, 2001.[18] I.-Y. Son, M. Guhe, and B. Yazici, “Human performance assessment usingfNIR,” in Proc. SPIE, vol. 5797, pp. 158–169, 2005.[19] M. Cope, “The development of a near-infrared spectroscopy system and itsapplication for noninvasive monitoring of cerebral blood and tissue oxygenation inthe newborn infant,” Ph.D. dissertation, Univ. College London, London, UK,1991.[20] S.C. Bunce, A. Devaraj, M. Izzetoglu, B. Onaral, and K. Pourrezaei,“Detecting deception in the brain: A functional near-infrared spectroscopy study ofneural correlates of intentional deception,” in Proc. SPIE, vol. 5769, pp. 24–32,2005.[21] K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei, and B. Chance, “Functionaloptical brain imaging using near-infrared during cognitive tasks,” Int. J. Human-Comp. Interact., vol. 17, no. 2, pp. 211–227. [22] K. Izzetoglu, G. Yurtsever, A. Bozkurt, B. Yazici, S. Bunce, K. Pourrezaei,and B. Onaral, “NIR spectroscopy measurements of cognitive load elicited byGKT and target categorization,” in Proc. 36th Hawaii Int. Conf. System Sciences,Jan. 2003, pp. 129-130. [23] M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A. Devaraj, B. Onaral, and K.Pourrezaei, “Functional near-infrared neuroimaging,” IEEE Trans. Neural Sys.Rehab. Eng., vol. 13, no. 2, pp. 153–159, 2005.[24] S.M. Platek, L.C.M. Fonteyn, M. Izzetoglu, T.E. Myers, H. Ayaz, C. Li, andB. Chance, “Functional near infrared spectroscopy reveals differences in self-other processing as a function of schizotypal personality traits,” SchizophreniaRes., vol. 73, no. 1, pp. 125–127, 2005.[25] R. Cabeza and L. Nyberg, “Imaging cognition II: An empirical review of 275PET and fMRI studies,” J. Cogn. Neurosci., vol. 12, no. 1, pp. 1–47, 2000.[26] E.E. Smith and J. Jonides, “Working memory: A view from neuroimaging,”Cogn. Psychol., vol. 33, no. 1, pp. 5–42, 1997.[27] E.E. Smith and J. Jonides, “Storage and executive processes in the frontallobes,” Science, vol. 283, Mar. 1999. [28] T.S. Braver, J.D. Cohen, L.E. Nystrom, J. Jonides, E.E. Smith, and D.C. Noll,“A parametric study of prefrontal cortex involvement in human working memory,”NeuroImage, vol. 5, no. 1, pp. 49–62, 1997.[29] G. Gratton, J.S. Maier, M. Fabiani, W.W. Mantulinm, and E. Gratton,“Feasibility of intracranial near-infrared optical scanning,” Psychophysiology,vol. 31, no. 2, pp. 211–215, 1994.[30] H. Obrig and A. Villringer, “Beyond the visible-Imaging the human brainwith light,” J. Cereb. Blood Flow Metab., vol. 23, no. 1, pp. 1–18, 2003.[31] D.A. Boas, M.A. Franceschini, A.K. Dunn, and G. Strangman, “Non-invasiveimaging of cerebral activation with diffuse optical tomography,” in In-Vivo OpticalImaging of Brain Function, R. Frostig, Ed. Boca Raton, FL: CRC Press, pp.193–221, 2002.[32] B. Chance, E. Anday, S. Nioka, S. Zhou, L. Hong, K. Worden, C. Li, T.Murray, Y. Ovetsky, D. Pidikiti, and R. Thomas, “A novel method for fast imagingof brain function, non-invasively, with light,” Optics Exp., vol. 2, no. 10, pp. 411–423, 1998.[33] M. St. John, D.A. Kobus, and J.G. Morrison, “A multi-tasking environmentfor manipulating and measuring neural correlates of cognitive workload,” in Proc.IEEE 7th Human Factors and Power Plants Conf., 2002, pp. 7-10–7-14, 2002.[34] H. Ayaz, M. Izzetoglu, S.M. Platek, S. Bunce, K. Izzetoglu, K. Pourrezaei,and B. Onaral, “Registering fNIR data to brain surface image using MRItemplates,” in Proc. EMBS’06, Aug. 2006, pp. 2671-2674.35] B. Chance, S. Nioka, S. Sadi, and C. Li, “Oxygenation and blood concentrationchanges in human subject prefrontal activation by anagram solutions,” Adv. Exp.Med. Biol., vol. 510, pp. 397–401, 2003. [36] M. Izzetoglu, S. Nioka, B. Chance, and B. Onaral, “Single trial hemodynamicresponse estimation in a block anagram solution study using fNIR spectroscopy,”in Proc. ICASSP’05, 2005, pp. 633-636. [37] F.M. Miezin, L. Maccotta, J.M. Ollinger, S.E. Petersen, and R.L. Buckner,“Characterizing the hemodynamic response: Effects of presentation rate, samplingprocedure, and the possibility of ordering brain activity based on relative timing,”NeuroImage, vol. 11, no. 6, pp. 735–759, 2000. [38] S. Bunce, M. Izzetoglu, K. Izzetoglu, B. Onaral, and K. Pourrezaei,“Functional near infrared spectroscopy: An emerging neuroimaging modality,”IEEE Eng. Med. Biol. Mag, vol. 25, no. 4, pp. 54-62, 2006. [39] G. McCarthy, M. Luby, J. Gore, and P. Goldman-Rakic, “Infrequent eventstransiently activate human prefrontal and parietal cortex as measured by functionalMRI,” J. Neurophysiol., vol. 77, no. 3, pp. 1630–1634, 1997.[40] J. Polich and A. Kok, “Cognitive and biological determinants of P300: Anintegrative review,” Biol. Psychol., vol. 41, no. 2, pp. 103–146, 1995.[41] B.A. Ardekani, S.J. Choi, G. Hossein-Zadeh, B. Porjesz, J.L. Tanabe, K.O.Lim, R. Bilder, J.A. Helpern, and H. Begleiter, “Functional magnetic resonanceimaging of brain activity in the visual oddball task,” Cogn. Brain Res., vol. 14,no. 3, pp. 347–356, 2002.

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE JULY/AUGUST 2007