melodic contour identification by cochlear implant listeners

Melodic Contour Identification by Cochlear ImplantListeners

John J. Galvin III, Qian-Jie Fu, and Geraldine Nogaki

Objective: While the cochlear implant providesmany deaf patients with good speech understand-ing in quiet, music perception and appreciationwith the cochlear implant remains a major chal-lenge for most cochlear implant users. The presentstudy investigated whether a closed-set melodiccontour identification (MCI) task could be used toquantify cochlear implant users’ ability to recog-nize musical melodies and whether MCI perfor-mance could be improved with moderate auditorytraining. The present study also compared MCIperformance with familiar melody identification(FMI) performance, with and without MCI training.

Methods: For the MCI task, test stimuli were me-lodic contours composed of 5 notes of equal dura-tion whose frequencies corresponded to musicalintervals. The interval between successive notes ineach contour was varied between 1 and 5 semitones;the “root note” of the contours was also varied (A3, A4,and A5). Nine distinct musical patterns were gener-ated for each interval and root note condition, result-ing in a total of 135 musical contours. The identifica-tion of these melodic contours was measured in 11cochlear implant users. FMI was also evaluated in thesame subjects; recognition of 12 familiar melodieswas tested with and without rhythm cues. MCI wasalso trained in 6 subjects, using custom software andmelodic contours presented in a different frequencyrange from that used for testing.

Results: Results showed that MCI recognition per-formance was highly variable among cochlear im-plant users, ranging from 14% to 91% correct. Formost subjects, MCI performance improved as thenumber of semitones between successive notes wasincreased; performance was slightly lower for theA3 root note condition. Mean FMI performance was58% correct when rhythm cues were preserved and29% correct when rhythm cues were removed. Sta-tistical analyses revealed no significant correlationbetween MCI performance and FMI performance(with or without rhythmic cues). However, MCI per-formance was significantly correlated with vowel rec-ognition performance; FMI performance was not cor-related with cochlear implant subjects’ phonemerecognition performance. Preliminary results alsoshowed that the MCI training improved all subjects’MCI performance; the improved MCI performancealso generalized to improved FMI performance.

Conclusions: Preliminary data indicate that theclosed-set MCI task is a viable approach towardquantifying an important component of cochlearimplant users’ music perception. The improvementin MCI performance and generalization to FMI per-formance with training suggests that MCI trainingmay be useful for improving cochlear implant users’music perception and appreciation; such trainingmay be necessary to properly evaluate patient per-formance, as acute measures may underestimatethe amount of musical information transmitted bythe cochlear implant device and received by co-chlear implant listeners.

(Ear & Hearing 2007;28;302–319)

With advances in cochlear implant technology,cochlear implant users’ speech recognition in quietlistening conditions has generally improved. How-ever, music perception and appreciation remains amajor challenge for cochlear implant users. Mostcurrent cochlear implant devices provide only 16 to22 electrodes with which to represent the entirespectrum of sound. Although this coarse spectral res-olution may be adequate for speech recognition, it isnot sufficient to code musical pitch information (e.g.,Gfeller et al., 2000; Kong et al., 2004; McDermott &McKay, 1997; Oxenham et al., 2004; Shannon et al.,2004; Smith et al., 2002). The limited spectral reso-lution in cochlear implants also precludes resolvingof harmonics, which is important for pitch percep-tion, timbre perception and segregating instru-ments. For complex sounds, resolved low-order har-monics generally yield stronger pitch percepts thanhigher-order harmonic components (e.g., Houtsma& Smurzynski, 1990); given that the cochlear im-plant does not provide enough spectral resolution tosupport resolving of harmonic components, the pitchof complex tones is less salient than for normal-hearing (NH) listeners, and timbre percepts may bequite distorted relative to normal. Besides the rela-tively poor spectral resolution, cochlear implant lis-teners receive limited temporal information, whichreduces the ability to track melodies according to thefundamental frequency (F0) of musical notes. Co-chlear implant listeners are able to detect amplitudemodulation over a limited frequency range (up to�300 Hz), further limiting access to temporal cues(Shannon, 1992).

Department of Auditory Implants and Perception, House EarInstitute, Los Angeles, California.

0196/0202/07/2803-0302/0 • Ear & Hearing • Copyright © 2007 by Lippincott Williams & Wilkins • Printed in the U.S.A.

302

Pitch perception is an important underlying com-ponent of music perception, as melody recognitionstrongly depends on perceiving exact intervals be-tween successive notes. In normal and electric hear-ing, the pitch of a sound is thought be extracted fromthe place of stimulation in the cochlea (place code),by resolving frequency components in the neuralfiring pattern (temporal code), or possibly by analyz-ing phase components across the signal spectrum atdifferent cochlear locations (e.g., Loeb, 2005; McKayet al., 2000; Tong et al., 1982). With NH listeners, itis difficult to disentangle the relative contributionsof the place code and temporal code to pitch percep-tion. Several studies have examined “nonspectral”pitch perception for sinusoidal amplitude modulated(SAM) noise (e.g., Burns & Viemeister, 1976, 1981)and have found that temporal pitch cues are gener-ally weak above 300 Hz. In cochlear implant re-search, it is possible to separate the place and rate ofstimulation and theoretically examine their relativecontributions to pitch perception. Although there isgreat intersubject variability in cochlear implantusers’ electrode pitch range and resolution, temporalpitch becomes weaker above 300 Hz, similar to NHlisteners (e.g., Shannon, 1983). Intersubject vari-ability may be due to the location of the implantedelectrodes and their proximity to healthy neuralpopulations (which may be affected by the durationof deafness and etiology of hearing loss before im-plantation). Single- and multi-electrode place pitchsensitivity has been somewhat correlated with co-chlear implant users’ speech performance (e.g.,Henry et al., 2000; Nelson et al., 1995). It is alsopossible that place and/or rate pitch percepts are notcomparable to pitch percepts by NH listeners; whileinterpreted as pitch cues due to testing constraints(e.g., forced-choice tests, pitch scaling, and so on),cochlear implant users may actually be perceiving“timbre” cues, which, while ordered along a pitchcontinuum, may not support pitch-intensive listen-ing tasks such as melody recognition or musicalinterval identification. Speech perception (e.g., pho-neme, word, or sentence recognition in quiet) mayrely less strongly on pitch cues and more strongly onspectral profile cues. Music perception, on the otherhand, much more strongly depends on pitch cues. Itis unclear how single-channel pitch discrimination,as typically measured in cochlear implant psycho-physical studies, may contribute to multichannelmusic perception.

Several studies have tested cochlear implant lis-teners’ music perception by directly stimulating theimplanted electrodes with varying stimulation rateand electrode place (e.g., McDermott & McKay,1997; Pijl & Schwarz, 1995a; 1995b). Pijl & Schwarz(1995a) measured familiar melody recognition

(without rhythmic cues) in cochlear implant usersby varying the stimulation rate delivered to a singleelectrode. Several frequency ranges were used tocode familiar melodies. Results showed that melodyrecognition was significantly better at the lowerfrequency ranges, reflecting the limits of patients’temporal processing above 300 Hz. Pijl & Schwarz(1995b) also tested the ability of non–musicallytrained cochlear implant subjects to reproduce mu-sical pitch intervals by adjusting the stimulationrate delivered to a single electrode pair. Resultsshowed that subjects were able to reproduce inter-vals within 1.5 semitones, using only temporal cues.McDermott & McKay (1997) similarly measuredmusical pitch perception in a musically trainedcochlear implant user; both the electrode place andthe rate of stimulation were varied. Results showedthat although low-frequency rate cues (�500 Hz)provided some musical pitch information, electrodeplace provided the dominant pitch cue. Given thelimits of temporal and spectral resolution measureddirectly in cochlear implant users, the cochlear im-plant can provide only a limited representation ofmusical melody.

Familiar melody recognition has been used todirectly assess cochlear implant listeners’ musicperception abilities (e.g., Fujita & Ito, 1999; Kong etal., 2004; Pijl & Schwarz, 1995a, 1995b). In general,cochlear implant users’ familiar melody recognitionperformance is much worse than that of NH listen-ers. Recently, Kong et al. (2004) investigated therelative contribution of temporal and spectral cuesto familiar melody recognition in both cochlear im-plant listeners and NH subjects listening to acousticcochlear implant simulations (in which the spectralresolution was varied between 1 and 64 channels).Melody identification was evaluated with two sets of12 familiar melodies. One set contained both rhythmand melody information, whereas the other set con-tained only melody information. For NH subjects,high levels of performance were possible when bothrhythm and melody cues were available, regardlessof the spectral resolution. Interestingly, mean co-chlear implant performance was lower than that ofNH subjects listening to a single-channel acousticcochlear implant simulation, suggesting that co-chlear implant users were not able to access therhythm cues contained in the melodies as well asNH listeners. When the rhythm cues were removed,32 or more frequency channels were required for NHlisteners to obtain a high level of performance.Without rhythm cues, cochlear implant listeners’melody recognition performance was most compara-ble to that of NH subjects listening to 1 to 2 spectralchannels. Given that cochlear implant users typi-cally can access 4 to 8 spectral channels provided by

EAR & HEARING, VOL. 28 NO. 3 303

the cochlear implant device for speech recognition(e.g., Fishman et al., 1997; Friesen et al., 1999; Fu etal., 1998; Fu & Nogaki, 2005), it is unclear whycochlear implant users cannot utilize a similar num-ber of spectral cues for melody recognition. It isprobable that perception of exact pitch intervals ismore important to melody recognition than tospeech recognition. Although music listening expe-rience (before and after implantation) may havecontributed to differences in cochlear implant andNH performance, familiar melody identification per-formance is quite similar between musician andnonmusician NH listeners (e.g., Bigand, 2003;Madsen & Staum, 1983). It is unclear whether mu-sical experience may benefit recognition of spec-trally distorted, isochronously presented familiarmelodies. For example, Lynch et al. (1991) foundthat music experience/education and cultural back-ground greatly influenced melody recognition per-formance when notes within a melody were mis-tuned, similar to the sort of mistuned pitch effectscochlear implant listeners may experience. Gfelleret al. (2000) described the listening habits and musicappreciation of a large number of subjects (65 co-chlear implant users). Music appreciation with thecochlear implant tended to vary according to thelistening environment, prior familiarity with a mu-sical piece, and structural components (type of in-strument, type of rhythm, and so forth). It is alsopossible that the pitch quantization and pitch dis-tortion associated with the implant device affectedcochlear implant users more strongly than NH sub-jects listening to cochlear implant simulations (whichmay have more to do with the veracity of the simula-tions than perceptual differences between the twosubject groups).

One shortcoming of testing familiar melody rec-ognition is that the limits of the cochlear implantdevice to represent melody cannot be accuratelyquantified, and in turn cannot be used to guide anyspeech processor optimization for music perception.Presently, there are no music perception tests thatcan provide this sort of information, whether be-cause of limited number of stimuli, or constraintsimposed by the task. For example, Gfeller et al.(1991, 1997) measured cochlear implant users’ mel-ody and rhythm discrimination using the PrimaryMeasures of Musical Audition (PMMA) test (Gordon,1979). In the PMMA test, cochlear implant subjectswere asked whether a pair of melodies or rhythmswere the same or different; melodies had between 2and 5 notes, and one or more of the notes was varied.Results showed that cochlear implant listeners weremore sensitive to changes in rhythm than in melody.However, because a two-alternative discriminationtask was used to measure performance, the high

chance level (50% correct) as well as the limitednumber of stimuli (40 items) did not allow the limitsof melody recognition performance to be quantifiedprecisely.

In developing an appropriate and meaningful testof cochlear implant users’ music perception, it isimportant to consider the perceptual processes tar-geted by the listening task. For example, familiarmelody identification may require closer attentionto the exact intervals presented in the melody,whereas perception of novel melodies may involveattention to the “pitch contour” of the melody (e.g.,Dowling, 1994). As such, deviations from the ex-pected intervals for a familiar melody may stronglyaffect identification performance. For novel melo-dies, perceiving the structural elements may dependless on the exact intervals, and more on the generalcontour of changes in pitch, rhythm and timbre.Because cochlear implant users’ perception of thesestructures will be limited by the amount of informa-tion transmitted by the device, it is important toquantify these limits in a systematic way. Currently,there is not a repeatable metric with which to comparemusic perception for different speech processor set-tings within individual patients. Such a metric wouldbe very beneficial for evaluating novel speech process-ing strategies in terms of the amount of melodicinformation transmitted and received.

In the present study, we developed the melodiccontour identification (MCI) test as a metric toassess cochlear implant users’ music perception. Inthe MCI test, cochlear implant users were asked toidentify one of nine 5-note melodic contours. Theinterval between successive notes in each contourwas systematically varied between 1 and 5 semi-tones to test the musical pitch resolution providedby the cochlear implant device. The “root note” (i.e.,the lowest note in the contour) was also varied (A3,A4, and A5) to test whether there was optimalsensitivity to musical pitch in different frequencyranges. Three-harmonic complexes were used torepresent the musical notes to test cochlear implantuser performance in a multichannel context. Forcomparison purposes, familiar melody recognitionwas also evaluated with two sets of 12 familiarmelodies, in which the rhythm cues were eitherpreserved or removed.

EXPERIMENT 1: MELODIC CONTOURIDENTIFICATION AND FAMILIAR MELODY

IDENTIFICATION

Methods

Subjects • Eleven cochlear implant users partici-pated in the present experiment. None were musi-

304 EAR & HEARING / JUNE 2007

cians before becoming deaf. All subjects were paidfor their participation, and all provided informedconsent before participating in the experiment (incompliance with the local Institutional ReviewBoard protocol). There was a wide range in speechrecognition performance and in music appreciationamong the cochlear implant subjects. Relevant de-mographic details are shown in Table 1. For com-parison, multitalker vowel and consonant recogni-tion were also measured in these subjects. Detaileddescription of testing materials and procedures forphoneme recognition can be found in Fu et al.(2005). The recognition scores for each individualcochlear implant users are listed in Table 1. NineNH subjects were also tested to compare perfor-mance between cochlear implant and NH listenersin the melodic contour identification task.Stimuli • Melodic contour identification perfor-mance was tested by using nine five-note melodicpatterns. The nine patterns represented simplepitch contours (e.g., “Rising,” “Flat,” “Falling”) andchanges in pitch contour (e.g., “Flat-Rising,” “Fall-ing-Rising,” “Rising-Flat,” “Falling-Flat,” “Rising-Falling,” “Flat-Falling”). Figure 1 shows the melodicpatterns and response screen used in the MCI test.

The melodic contours were generated in relationto a “root note,” (i.e., the lowest note in the melody);the shaded notes in Figure 1 represent the root notein each contour. To test whether contours werebetter represented for different frequency ranges,three root notes were selected: A3 (220 Hz), A4 (440Hz), and A5 (880 Hz). All notes in the contours weregenerated according to: f � 2

x12fref , where f is the

frequency of the target note, x is the number ofsemitones relative to the root note and fref is thefrequency of the root note. The interval betweensuccessive notes in the melodic contour was variedbetween 1 and 5 semitones. For example, for the“Rising” contour, when successive notes were sepa-rated by 5 semitones, the contour spanned almost 2

octaves; when successive notes were separated by 1semitone, the contour spanned less than half anoctave. Thus, for the MCI test, a total of 135 melodiccontours were generated (9 melodic patterns � 3root notes � 5 semitone distances). The fundamen-tal frequency range covered by the entire stimulusset was 220 to 2794 Hz.

Each note was generated as a harmonic complex:fundamental frequency [F0] � first harmonic (2*F0;–3 dB) � second harmonic (3*F0; –6 dB). Harmoniccomplexes (rather than pure tones) were used tobetter approximate some of the spectral componentsfound in natural production of musical notes, andthereby better approximate cochlear implant sub-jects’ “real-world” music perception. Each note was250 msec in duration; the onset was ramped (10msec) and the offset was damped (10 msec) to reduceany transient spectral splatter. The interval be-tween notes was 50 msec.

TABLE 1. Cochlear implant (CI) user demographics for the present experiment

Subject Device Strategy

Inputfrequencyrange (Hz)

Stimulationrate per

channel (Hz) Age SexCause ofdeafness

CI use(yr)

Vowel(%)

Consonant(%)

S1 CII HiRes 250–8000 2578 52 M Unknown 6 82 79S2 N24 ACE 188–7938 900 62 F Genetic 2 83 85S3 N22 SPEAK 120–8658 250 64 M Unknown 15 73 72S4 N22 SPEAK 120–8658 250 75 M Noise-induced 9 86 67S5 N22 SPEAK 150–10853 250 46 M Trauma 13 92 81S6 CII HiRes 250–8000 2578 72 M Unknown 6 74 69S7 N24 SPEAK 188–7938 1200 72 F Unknown 5 63 63S8 MedEI CIS� 300–7000 1488 26 M Prenatal rubella 3 70 86S9 N22 SPEAK 150–10853 250 67 M Hereditary 14 78 75S10 CII HiRes 250–8000 5156 42 F Unknown 2 56 52S11 MedEI CIS� 300–5500 1488 41 F Congenital 5 50 57

Fig. 1. Melodic contours and response screen for the melodiccontour identification (MCI) test. The shaded notes representthe root note (A3, A4, or A5) in each contour.


Figure 2 shows two examples of the “Rising”melodic contour, with root note A5 (880 Hz). The leftcontour has 5 semitones between successive notes;the right contour has 1 semitone between notes. Theright y-axis shows frequency in Hz. The left axisshows the frequency channels assigned by Fre-quency Allocation Table 9, which is used by manyNucleus-22 patients. The horizontal lines show thecutoff frequencies of the analysis bands for Table 9.When there are 5 semitones between successivenotes, each change in F0 is coded by a differentchannel (electrode). Because of the wide range offrequency components, the contour is representedusing 15 of the available 22 electrodes. For thiscontour, cochlear implant listeners may attend tochanges across frequency channels. When there isonly 1 semitone between successive notes, changesin F0 are coded by, at most (depending on the rootnote), only 3 electrodes; the entire contour mayactivate, at most, 9 of the available 22 electrodes.For this contour, cochlear implant listeners mayattend to changes within and across frequency chan-nels. For within-channel pitch percepts, cochlearimplant listeners may attend to temporal periodicitycues found in the modulated envelope, althoughthese pitch cues are generally weaker than electrodeplace cues. Also, cochlear implant listeners maydetect small changes in amplitude across adjacentelectrodes, as the slope of the analysis filters is fairlybroad, allowing some stimulation on nearby elec-trodes even for a tone centered within the analysisband. Small changes in these local spectral enve-

lopes might provide some additional pitch informa-tion for the 1-semitone contours.

Familiar melody identification (FMI) was alsomeasured for 12 familiar melodies used by Kong etal. (2004, 2005), with and without rhythm cues.Three-harmonic complexes were generated for eachnote, using the same methods as for melodic con-tours. The F0 range covered by all melodies was 415to 1047 Hz. Similar to the familiar melody analysesshown in Kong et al. (2004) and Pijl & Schwartz(1995a), we analyzed the familiar melodies used inthe present study. However, we have added morecomponents to the analysis, including: the totalnumber of notes, the number of novel notes (i.e., newnotes within the contour), the number of changes innote, the semitone range, the number of novel beats(i.e., new beats within the contour), the number ofchanges in beat, the range of rhythm (in beats).These components were used to derive a “melodicindex” and “rhythmic index” for each melody,thereby describing, to some extent, differences inmelodic and rhythmic complexities among the testmelodies. The melodic index was defined as themean of the number of novel notes, the number ofchanges in note, and the semitone range, each di-vided by the total number of notes. Similarly, therhythmic index was defined as the mean of thenumber of novel beats, the number of changes inbeat, and the beat range, each divided by the totalnumber of notes. The results of the analysis can beseen in Appendix 1. There are marked differences inthe melodic and rhythmic indices between melodies.

Fig. 2. Spectrogram for “Rising” contour(root note: A5). In the left contour, there arefive semitones between successive notes. Inthe right contour, there is one semitonebetween each note. The x-axis shows time.The right y-axis shows frequency (in Hz).The left y-axis shows the electrode assign-ment for subjects S3, S7, and S9 (Nucleus-22frequency allocation Table 9). The horizon-tal lines show the cutoff frequencies for eachfrequency analysis channel.


For example, “Mary Had a Little Lamb” has amelodic index of 0.36, reflecting the limited numberof novel notes and relatively small semitone range.“The Star Spangled Banner” has a melodic index of0.94, reflecting the comparatively larger semitonerange and number of novel notes. Note that whilethe melodic index may indicate differences in com-plexity between melodies, it does not indicatewhether one melody might be easier to identify thananother. For cochlear implant listeners, melodieswith a large semitone range may be easier to iden-tify than those with a small range. In many previousstudies (e.g., Kong et al., 2004, 2005; Pijl &Schwartz, 1995a, 1995b), rhythm cues were re-moved from familiar songs to test the ability of thecochlear implant device to represent melody. How-ever, the variability in the rhythmic index suggeststhat many melodies are not equally weighted interms of rhythmic cues. For example, “Yankee Doo-dle Dandy” has a rhythmic index of 0.12, reflectingthat the rhythm changes very little during themelody; removing the rhythmic information wouldnot significantly affect the rhythm cues. “HappyBirthday” has a rhythmic index of 0.53, reflecting awider range of beats and more changes in rhythm;thus, removing rhythmic information would morestrongly affect the rhythm cues associated with thismelody. Because both melody and rhythm cues areessential for melody recognition, simply removingthe rhythm cues may not be the best approach tomeasure the ability of the cochlear implant device torepresent melody.

We conducted similar analyses for the MCI stim-uli; the results can be found in Appendix 2. Themean melodic index is 0.98 for the contours, some-what higher than that of the familiar melodies(0.62). This difference is mainly due to the greaternumber of stimuli (45 contours versus 12 melodies)as well as the greater semitone range (1 to 20 forcontours versus 4 to 16 for melodies) and the num-ber of total notes (5 for contours versus 11 to 15 formelodies). Note that the rhythmic index is constantfor all contours (0.07). Given the range of contours,the range of notes in each contour, and the threefrequency ranges tested for each contour (A3, A4,A5), these melodic contours seem well suited formeasuring the ability of the cochlear implant deviceto convey musical melody.Procedures • Cochlear implant users were testedusing their clinically assigned speech processors.Patients were asked to set their sensitivity andvolume settings as they would for comfortably loudspeech; once set, subjects were asked to not changethese settings. Testing was conducted in free field ina sound-treated booth (IAC). Stimuli were delivered

via a single loudspeaker (Tannoy Reveal) at 70 dBA;subjects sat directly facing the loudspeaker.

For the MCI task, a stimulus was randomlyselected (without replacement) from among the 135melodic contours in the stimulus set. Nine responsechoices were shown onscreen (see Fig. 1) and thesubject clicked on the response choice he/shethought was correct. Subjects were allowed to repeateach stimulus up to three times; no trial-by-trialfeedback was provided. Subjects were told to guess ifthey were unsure of the answer, but cautionedagainst choosing the same response for every stim-ulus. Performance with the MCI task was measureda minimum of 2 times, and usually 5 or more timesto obtain a stable measurement. Depending onwhether learning effects were observed from test totest, MCI performance was measured in a singlesession or over two to four sessions. If significantlearning effects were observed between test runs, onlyasymptotic performance was used to calculate meanMCI scores. Each run of the MCI test took 10 to 15minutes to complete. To give some idea of procedurallearning effects during MCI testing, the number oftotal tests is listed for each subject in Table 2.

For the FMI task, melody recognition was firsttested with rhythm cues. The stimulus set contained2 presentations of each of the 12 melodies, with therhythm cues preserved (24 stimuli in total). A stim-ulus was randomly selected from the set (withoutreplacement) and presented to the subject. Thesubject responded by clicking on one of the 12response choices shown onscreen; response choiceswere labeled with the melody titles. Subjects wereallowed to repeat each stimulus up to three times; noimmediate feedback was provided. After testingwith melody recognition with rhythm cues, FMI wasmeasured without rhythm cues. Similarly, the stim-ulus set contained 2 presentations of each of the 12melodies, but with no rhythm cues (24 stimuli intotal). A stimulus was randomly selected from theset (without replacement) and presented to the sub-ject. The subject responded by clicking on one of the12 response choices shown onscreen; responsechoices were labeled with the melody titles. FMIwith and without rhythm cues was tested in a single10- to 15-minute session.

Results

Figure 3 shows MCI performance for individualcochlear implant users. Note that subjects are or-dered according to performance, from best to worst.For each subject, mean performance was calculatedacross all three frequency ranges and all internotesemitone distances. The differently shaded bars rep-resent the different cochlear implant devices. The


limited number of subjects per type of device pre-cludes any statistical analysis; however, no clearadvantage was observed for any particular device.The dashed line shows mean performance from nineNH subjects; mean performance was 94.8% correct,with scores ranging from 88.1% to 100% correct.There was great intersubject variability in cochlearimplant performance, as MCI scores ranged from14.1% to 90.7% correct.

Table 2 shows mean cochlear implant perfor-mance for the MCI task; mean performance was alsocalculated for the 3 root note and 5 semitone-dis-

tance conditions. In terms of cochlear implant users’sensitivity to root note, performance was generallylower for root note A3. Performance was onlyslightly (but significantly) affected by the root note[one-way repeated-measures ANOVA: F(2,20) �3.66, p � 0.044]. Cochlear implant users’ perfor-mance generally declined as the semitone distancebetween successive notes was reduced. A one-wayrepeated-measures ANOVA showed that perfor-mance was significantly affected by the semitonedistance F(4,40) � 27.56, p � 0.001]. Post hocBonferroni t-tests revealed that performance with 1semitone between notes was significantly worsethan that with any of the other semitone distances(p � 0.05), and that performance with 2 semitonesbetween notes was significantly worse than thatwith 5 semitones between notes (p � 0.05); therewas no significant difference between the remainingsemitone distances.

In terms of cochlear implant listeners’ sensitivityto the nine melodic contours, the “Flat” contour wasgenerally easiest to identify. Table 3 shows themean percent correct and percent of total responsesfor each of the nine melodic contours. The columntotal means for a cumulative confusion matrix(across all test runs and subjects) were used toanalyze response bias. Mean recognition perfor-mance was highest for the “Flat” contour and lowestfor the “Falling” contour. Analysis of the distributionof responses revealed that, on average, subjectsresponded with “Flat” most frequently and “Falling”least frequently; note that chance level was 11.1%.

Figure 4 shows FMI performance for individualcochlear implant users, with and without rhythmcues. Note that subjects are ordered according toperformance in the MCI task (Figure 3), from best to

TABLE 2. Mean performance, for each subject, for the MCI and FMI tasks, in percent correct

Melodic contour identification

Base note Semitones between notes Familiar melody identification

Subject % A3 A4 A5 1-semi 2-semi 3-semi 4-semi 5-semiNumber ofMCI tests

Withrhythm cues

Withoutrhythm cues

S1 90.7 90.0 88.9 93.3 63.0 96.3 98.1 98.1 98.1 2 87.5 50.0S2 89.0 83.1 92.0 92.0 62.2 91.8 97.0 97.0 97.0 5 83.3 58.5S3 72.3 64.8 74.6 77.3 47.4 71.1 71.8 83.7 87.4 9 45.0 4.1S4 65.6 55.9 65.4 75.8 37.0 58.1 69.8 77.2 84.3 16 41.7 12.5S5 64.5 64.6 58.3 70.7 27.2 52.8 73.7 81.5 87.5 12S6 62.3 59.6 65.0 62.2 30.6 57.1 72.5 76.7 74.6 7 95.8 45.8S7 40.9 32.5 44.4 45.9 18.5 38.2 39.4 46.9 61.7 3 75.0 25.0S8 31.1 31.1 35.1 27.1 21.4 29.6 35.5 34.0 34.7 5 16.6 4.1S9 28.3 19.4 32.2 33.3 11.1 31.4 29.6 29.6 39.8 4 41.6 25.0S10 27.5 34.6 28.4 19.5 17.7 29.6 29.6 31.8 27.8 5S11 14.0 11.1 15.5 15.5 13.5 14.8 17.2 11.1 13.5 3 54.1 29.1Mean 53.3 49.7 54.5 55.7 31.8 51.9 57.6 60.7 64.2 6.4 60.1 28.2SD 26.2 25.6 25.3 28.6 18.5 26.4 28.3 30.6 30.3 4.3 26.5 19.7

Fig. 3. Melodic contour identification (MCI) performance for11 cochlear implant users. Subjects are ordered according tomean performance. Error bars show 1 standard deviation.


worst. Note also that some subjects were not able tocomplete the FMI task, due to time restrictions. Aswith the MCI data, there was great intersubjectvariability. When rhythm cues were available, FMIscores ranged from 16.7% to 95.8% correct; whenrhythm cues were removed, scores ranged from 4.2%to 58.5% correct. Mean FMI performance (across allsubjects) is shown in Table 2. A one-way repeated-measures ANOVA showed that FMI performancewas significantly better when rhythm cues wereavailable [F(1,8) � 49.30, p � 0.001].

Cochlear implant listeners’ phoneme recognitionperformance was compared with their MCI and FMIperformance. Linear regressions were fit betweenMCI and vowel recognition performance, and be-tween MCI and consonant recognition performance.Linear regressions were also fit between FMI (norhythm cues) and vowel recognition performance,

and between FMI (no rhythm cues) and consonantrecognition performance. Table 4 shows r2 valuesand levels of significance for each of the regressions.Only the regression between cochlear implant users’MCI and vowel recognition performance was statis-tically significant (p � 0.01). There was no signifi-cant correlation between cochlear implant subjects’FMI and phoneme recognition performance, or be-tween MCI and FMI performance.

EXPERIMENT 2: MELODIC CONTOURIDENTIFICATION TRAINING

The results of Experiment 1 demonstrated a largeintersubject variability in music perception. Inter-estingly, cochlear implant subjects’ MCI perfor-mance was correlated with vowel recognition perfor-mance, suggesting that the large intersubjectvariability was common to both music and speechperception. Previous studies have shown that mod-erate auditory training may improve cochlear im-plant users’ speech performance, thereby reducingthe variability in patient performance. The improve-ment in performance may occur over the short term(e.g., Rosen et al., 1999) or long term (e.g., Fu et al.,2002). Daily training that targets phonemic con-trasts has been shown to be especially effective inimproving cochlear implant users’ phoneme recogni-tion (Fu et al., 2005). Structured training usingmusical materials has also been shown to improvecochlear implant listeners’ musical timbre percep-tion (Gfeller et al., 2002). In Experiment 2, wetrained 6 of the 11 subjects who participated inExperiment 1 on the MCI task. Subjects trained forapproximately 1 hour per day, using alternate fre-quency ranges that were not used in the MCI teststimulus set. The purpose of Experiment 2 was totest whether moderate auditory training with sim-ple musical contours would improve cochlear im-plant users’ performance in the MCI and FMI tests,thereby reducing the variability in cochlear implantusers’ music perception performance.

TABLE 3. Mean percent correct (across subjects) for the ninemelodic contours and mean percent (across subjects) of totalresponses for each type of contour.

Melodic contourPercentcorrect

Percent oftotal responses

Rising 48.6 9.7Rising-Flat 49.6 11.5Rising-Falling 53.0 10.0Flat-Rising 54.6 12.1Flat 75.5 15.7Flat-Falling 49.1 10.6Falling-Rising 53.8 11.3Falling-Flat 45.6 10.4Falling 39.4 8.7

Fig. 4. Familiar melody identification (FMI), with and withoutrhythm cues.

TABLE 4. Results of linear regression between MCI, FMI (norhythm cues), and phoneme (vowel and consonant) recognitionperformance for 11 cochlear implant users

Vowel ConsonantFMI

(no rhythm cues)

Task r 2 p r 2 p r 2 p

MCI 0.54 0.01 0.29 0.09 0.22 0.25FMI (no

rhythm cues)0.03 0.64 0.02 0.74

Significant correlations are shown in bold type.


Methods

Subjects • Six of the 11 cochlear implant users whoparticipated in experiment 1 participated in thepresent experiment.Stimuli • MCI training stimuli were similar tothose used in Experiment 1, except that differentroot notes were used for the training contours. Theroot note for the training contours was shifted fromthat of the test contours to avoid any direct trainingof the test contours. For subjects S3, S5, and S9, theroot note for each melodic contour was randomlyselected to be between A3 (220 Hz) and A5 (880 Hz),without including A3, A4, and A5 as root notes; forsubjects S4, S7, and S10, the root note was C4 (262Hz).Procedures • MCI training was conducted at homefor most subjects on a laptop computer, using cus-tom training software (see Fu et al., 2005, for ageneral description of the training software). Sub-jects listened to audio playback during the trainingvia Cochlear’s TV cable connected to the headphonejack of the laptop computer. Subjects S3, S4, S7, S9,and S10 trained for a half-hour per day, every day.The time course for training varied among subjects,ranging from 1 week to nearly 2 months (the timecourse for each subject can be seen in Figure 6).Subject S5 was trained in the lab for 3 hours per day,over a 5-day period; training was conducted in freefield in a sound-treated booth at comfortable listen-ing levels, similar to the test environment.

The training was conducted using trial-by-trialaudio and visual feedback, with repeated playbackcomparison between the subject’s (incorrect) re-sponse and the correct response. Each training ex-ercise consisted of 25 melodic contours; note that the9 melodic contours were distributed randomlywithin the stimulus set. A contour was selected(randomly, without replacement) from the stimulusset. The subject clicked on one of the responsechoices shown onscreen. The number of responsechoices ranged from 2 to 9, depending on the level ofdifficulty. If the subject answered correctly, visualfeedback was provided and a new contour was pre-sented. If the subject answered incorrectly, audioand visual feedback was provided, in which thecorrect response and the subject’s (incorrect) re-sponse were repeatedly played back, allowing thesubject to compare the melodic contours. When thesubject’s performance achieved a criterion level ofperformance (80% correct), the software automati-cally increased the level of difficulty by increasingthe number of response choices and/or reducing thenumber of semitones between successive notes inthe contour. The computer automatically loggedsubjects’ training performance.

Subjects who trained at home returned to the labregularly for retesting. For retesting, MCI perfor-mance was measured for the same root notes used inExperiment 1 (A3, A4, A5). At the end of the trainingperiod, subjects’ FMI performance was also remea-sured. Follow-up measures of MCI performancewere obtained one month after training was stoppedto see if any performance gains with training hadbeen retained.

Results

Figure 5 shows baseline and post-training MCIresults for 6 cochlear implant users. For all subjects,performance significantly improved with training.The amount of improvement ranged from 15.5 to45.4 percentage points. Table 5 shows mean cochlearimplant performance for the MCI task, before andafter training; mean performance was also calcu-lated for the 3 root note and 5 semitone-distanceconditions. Mean results show that not only wasMCI performance improved with training, but theintersubject variability was reduced (as evidencedby the lower standard deviation in mean perfor-mance after training). A one-way repeated-mea-sures ANOVA showed that MCI performance wassignificantly improved with training [F(1,5) � 26.69,p � 0.004]. In terms of sensitivity to root note,although performance improved for all root notes,one-way repeated-measures ANOVAs showed thatthe root note had no significant effect on MCIperformance for baseline [F(2,10) � 2.24, p � 0.157]

Fig. 5. Baseline and post-training MCI performance for indi-vidual cochlear implant users. Baseline performance is thesame as in Figure 3. Error bars show 1 standard deviation.


or post-training measures [F(2,10) � 0.66, p �0.538]; note that for the 11 cochlear implant subjectsin Experiment 1, performance was slightly but sig-nificantly worse with root note A3. In terms ofsensitivity to the semitone distance between succes-sive notes in a contour, a one-way repeated-mea-sures ANOVA showed that both baseline perfor-mance [F(4,20) � 21.65, p � 0.001] and post-trainingperformance [F(4,20) � 37.09, p � 0.001] wassignificantly affected by the semitone distancebetween notes. A two-way repeated-measuresANOVA showed that training significantly im-

proved performance for all semitone distances[F(4,20) � 28.09, p � 0.003].

Although the time course of training was notexplicitly controlled as an experimental parameter(subjects were allowed to continue training as longas they liked), performance was periodically testedduring the training experiment. Figure 6 shows thetime course of training for 6 cochlear implant users.Subjects S3 and S5 only trained for 1 week, whereassubjects S4, S7, S8, and S10 trained for 1 month ormore. For most subjects, performance continued toimprove during the training period. For three sub-jects (S3, S9, and S10), performance was retestedafter training was stopped. For subjects S3 and S10,follow-up measures were only slightly lower thanpost-training levels 1 month after training wasstopped. For subject S9, follow-up measures werereduced from post-training levels, although theyremained much higher than pretraining baselinelevels.

FMI performance was also remeasured at the endof the training period. The same melodies were usedfor testing as in the baseline measures. Unfortu-nately, because baseline FMI performance was notmeasured with subjects S5 and S10, post-trainingFMI results could only be measured for subjects S3,S4, S7, and S9. Figure 7 shows baseline and post-training FMI performance for these four subjects.Note that for subject S9, FMI retesting was con-ducted 1 week after training was begun; for subjectsS3, S4, and S7, FMI retesting was conducted at theend of the training period. Results showed that FMIperformance (with and without rhythm cues) wasimproved with the MCI training. Table 5 showsmean cochlear implant performance for the FMItask (with and without rhythm cues), before andafter training. When rhythm cues were provided,

Fig. 6. MCI test performance over time during the trainingexperiment. Filled symbols show MCI performance that wasretested during the training period. Open symbols showfollow-up testing after training had been stopped.

TABLE 5. Mean performance measures (across subjects) for the MCI and FMI tasks, before (baseline) and after (post-) training, inpercent correct

Melodic contour identification

Base note Semitones between notes Familiar melody identification

Subject % A3 A4 A5 1-semi 2-semi 3-semi 4-semi 5-semi With rhythm cues Without rhythm cues

S3 87.8 83.3 87.8 92.2 66.7 85.1 92.6 98.2 96.3 62.5 29.2S4 85.2 82.2 84.4 88.9 60.5 80.3 95.1 91.4 98.8 62.5 33.3S5 86.3 88.9 86.7 86.7 51.9 100.0 96.3 92.6 96.3S7 69.6 47.0 64.8 65.6 28.7 45.4 63.0 75.9 82.7 58.3 33.0S9 73.7 62.2 73.4 85.6 31.5 72.3 81.5 88.9 94.5 62.5 54.2S10 66.7 74.8 75.6 49.6 39.5 67.9 69.2 80.3 76.6Baseline mean

(from Table 2)49.9 45.4 50.6 53.8 26.5 46.9 52.4 58.5 64.8 52.4 16.7

SD 20.1 19.0 18.6 24.4 13.6 16.5 21.7 25.3 26.1 15.3 10.2Post-train mean 78.2 73.1 78.8 78.1 46.5 75.2 82.9 87.9 90.9 61.5 37.4SD 9.3 15.7 9.1 16.8 15.7 18.4 14.2 8.3 9.0 2.1 11.3


mean performance improved by 9.1 percentagepoints; MCI training improved FMI performancewhile reducing intersubject variability (reducedstandard deviation in mean performance after train-ing). However, a one-way repeated-measures ANOVAshowed that the improvement was not significant[F(1,3) � 1.09, p � 0.373]. Note that for subject S7,FMI performance with rhythm cues declined aftertraining, whereas performance improved for theremaining 3 subjects. When rhythm cues were re-moved, mean performance improved by 20.8 per-centage points. Although MCI training improvedFMI performance, intersubject variability was notreduced (slightly higher standard deviation in meanperformance after training), most likely because ofthe floor effects associated with the baseline FMIperformance. A one-way repeated-measures ANOVAshowed that the improvement in FMI performance(without rhythm cues) was significant [F(1,3) �20.54, p � 0.020].

GENERAL DISCUSSION

The MCI task used in the present study showedthat cochlear implant users were capable of onlylimited melodic contour recognition. Overall, eventhe best cochlear implant performance was substan-tially poorer than mean NH performance. In gen-eral, when there were 2 or fewer semitones between

successive notes in the melodic contour, cochlearimplant users had greater difficulty. However, sub-jects were able to significantly improve MCI perfor-mance with training, suggesting that cochlearimplant users’ music perception may be greatlyimproved with training and music listening experi-ence. Moreover, the MCI training generalized toimproved FMI performance, emphasizing the impor-tance of training to develop better music perceptionskills. We will discuss the results of the presentstudy in greater detail in the sections below.

MCI Performance of Cochlear Implant Users

The most striking result from the present study(and common to much cochlear implant research) isthe large intersubject variability. There was no clearadvantage in cochlear implant device, suggestingthat the frequency allocations, electrode configura-tions and stimulation rates used in cochlear implantusers’ clinically assigned speech processors were notthe limiting factors in MCI performance. Morelikely, differences in subjects’ functional spectralresolution (i.e., the distribution of healthy neuralpopulations and their proximity to the implantedelectrodes) may have contributed to differences inpatient performance. However, these patient differ-ences will also interact with critical speech processorparameters (e.g., frequency allocation, electrode con-figuration), suggesting that for the top-performingpatients, processor parameters may have been bet-ter optimized to the functional spectral resolution.For poorer-performing patients, these parametersettings may be suboptimal and some adjustmentmay be required to provide the maximum functionalspectral resolution that can be supported by theremaining auditory neural populations.

In terms of optimal frequency ranges for MCI,performance with A4 and A5 root notes was slightlybetter than that with the A3 root note. This may bedue to poorer frequency selectivity in the apicalchannels (due to the frequency allocation or elec-trode selectivity) or to the spectral shift associatedwith the limited insertion of the electrode array. Forcochlear implant users, the mismatch would begreatest in the apical end of the array. However, thedifference in MCI performance between the threeroot notes was relatively small (�5 percentagepoints), suggesting that the different input fre-quency ranges and frequency allocations used indifferent cochlear implant devices were able to sup-port similar MCI performance for a wide range ofacoustic frequencies.

In terms of cochlear implant listeners’ musicalfrequency resolution, even the top performers wereunable to identify all melodic contours when there

Fig. 7. Baseline and post-training FMI performance for 4subjects. Filled bars show FMI performance with rhythmcues; hatched bars show performance without rhythm cues.Dark bars show baseline performance; light bars show post-training performance.


was only 1 semitone between successive notes in acontour. With 1 semitone between notes, subjects S1and S2 were able to correctly identify only �63% ofthe contours. If identification of the “Rising,” “Fall-ing,” and “Flat” contours are not considered, MCIperformance was even lower. When there were 2semitones between notes, subjects S1 and S2 wereable to correctly identify more than 90% of thecontours. These results suggest that top performerswere capable of good MCI performance when therewere 1 to 2 semitones between successive notes inmelodic contours (or, �0.13 octave resolution; forreference, 1 semitone between notes corresponds to�0.08 octave resolution). In contrast, poorer-per-forming subjects (subjects S8, S9, S10, and S11)could correctly identify only �34% of the contourswhen there were 5 semitones between successivenotes. For these subjects, musical frequency resolu-tion was greater than 0.40 octaves. Given that thefrequency allocation in cochlear implant speech pro-cessors generally corresponds to relatively coarseresolution (for example, 1.2 octaves, on average, foreach frequency channel in the default Nucleus 24frequency allocation), top performers were able tohear out cross-channel differences in the spectralenvelope (because all notes were three-harmoniccomplexes) or within-channel differences in the tem-poral envelope, whereas poor performers could notattend to these spectral or temporal cues.

There was a significant correlation between MCIand vowel recognition performance, one of few ex-amples of any correlation between speech and musicperception in cochlear implant users. Because lis-teners most likely attended to spectral envelope cuesin both tests, it is not surprising that performancewould be correlated between these tests. Previousstudies that compared FMI to speech recognitionperformance found no correlation between thesemeasures. However, the FMI performance may haverelied more strongly on perception of exact intervals,while the MCI and vowel recognition tests may notrequire such exact pitch perception. The FMI testalso may depend more strongly on perception ofrhythm cues, listeners’ musical experience, and soforth, factors that do not affect MCI performance.

FMI Performance of Cochlear Implant Users

Consistent with previous studies (Kong et al., 2004),FMI performance depended greatly on rhythm cues.Mean performance (across subjects) was more than30 percentage points better when rhythm cues wereprovided than when rhythm cues were removed.While showing the limited capability of the cochlearimplant device to represent melody, these resultsmost strongly point to the importance of rhythm

cues to familiar melody recognition. FMI perfor-mance without rhythm cues shed little light on thelimits of the cochlear implant device to convey mel-ody (e.g., semitone resolution); hence, the impetusfor the MCI test used in the present study.

FMI performance was not significantly correlatedwith MCI performance or with phoneme recognitionperformance. Again, differences in cochlear implantusers’ musical experience (both before and afterimplantation), “distortions” to the familiar melodiesby removing rhythm cues, and differences in theperceptual tasks (i.e., interval perception in FMIand contour perception in MCI) may have contrib-uted to the “unpredictability” of the FMI results. Forexample, subject S11 performed very poorly in theMCI test (14% correct), but moderately well in thevowel recognition test (50.0% correct) and the FMItest without rhythm cues (29.2% correct); she re-ported great familiarity with the melodies used inthe FMI test because she regularly sang them to heryoung daughter. In contrast, subject S3 performedmuch better in the MCI (72.3% correct) and vowelrecognition (73% correct) tests, but much poorer inthe FMI test (4.2% correct). Because prior experi-ence with the test stimuli is not a factor in the MCItest, melodic contours may be preferable for assess-ing cochlear implant users’ music perception.

MCI Training in Cochlear Implant Users

Moderate, regular training with the MCI task sig-nificantly improved all subjects’ MCI performance. Allsubjects were trained with melodic contours that dif-fered from the test contours in terms of the root note.Thus, to some degree, training with alternate fre-quency ranges generalized to the test frequencyranges. This result, along with most subjects’ contin-ued improvement over the training period, suggeststhat some “perceptual learning” rather than simply“procedural learning” or memorization of the test stim-uli occurred during the MCI training (Wright et al.,2001). The targeted, progressive method used in theMCI training was similar to that used in previousspeech training studies (e.g., Fu et al., 2005). In thesetraining studies, the opportunity to carefully listen toauditory-only stimuli (i.e., without lipreading) mayhave strongly contributed to the improved perfor-mance. Top-performing cochlear implant users mayregularly listen to auditory-only stimuli (i.e., talkingon the telephone); for these patients, auditory-onlyspeech tests may be less difficult. Poorer-performingpatients may rely more strongly on lipreading; forthese patients, auditory-only speech tests are usuallyvery difficult. For poor performing patients, acute audi-tory-only tests may underestimate the potential of thecochlear implant device to represent speech patterns, as


targeted speech training has been shown to significantlyimprove performance. Similarly, with the MCI task,training significantly improved performance, and initialbaseline testing underestimated the capability of thecochlear implant device to represent melody.

The MCI training results also showed that mostpatients continued to improve throughout the trainingperiod (see Fig. 6). Follow-up measures in three sub-jects showed that MCI performance remained nearpost-training levels 1 month (or more) after trainingwas stopped. For subjects S9 and S10, follow-up per-formance was slightly worse than at the end of train-ing, yet remained much higher than baseline perfor-mance. These results suggest that subjects were ableto retain the benefits of MCI training.

Interestingly, for 4 of the training subjects, FMIperformance (without rhythm cues) significantly im-proved after MCI training. Note that FMI was notdirectly trained. Thus, besides generalizing to im-proved MCI performance for different frequencyranges, MCI training generalized to improved FMIperformance. Anecdotal reports also suggest thatcochlear implant subjects’ music perception outsidelaboratory test conditions had generally improved.For example, some subjects reported that they werebetter able to separate the singer’s voice from thebackground music while listening to music in thecar. While it is difficult to know exactly how MCItraining may contribute to improved “real-world”music perception and appreciation, careful listeningto melodic contours most likely improves cochlearimplant listeners’ overall attention to the limiteddetails provided by the cochlear implant device.

Comparison of Single-Electrode PitchDiscrimination and Modulation Detection toMCI/FMI Performance

In the present study, both MCI and FMI perfor-mance were measured with multichannel (i.e.,3-harmonic tone complexes) stimuli delivered tocochlear implant users’ speech processors. While thefrequency allocation used in the speech processorsmay be a primary limitation in performance, co-chlear implant users’ electrode pitch range andresolution may also be a limiting factor. Indeed, thefrequency allocation may be manipulated to try toencode more fundamental frequency information(e.g., Laneau et al., 2004), but cochlear implantusers will ultimately be limited by the spectro-temporal resolution of the implanted ear (i.e., limi-tations in electrode discrimination, modulation de-tection, rate discrimination, and so forth). It is notclear that single-electrode pitch range and resolu-tion well reflects multichannel performance; for ex-ample, single-electrode pitch perception has been

only modestly correlated with multichannel speechperception (e.g., Nelson et al., 1995).

Although not directly measured in the presentexperiment, we previously measured single elec-trode pitch discrimination for a variety of stimula-tion modes and rates for five of the subjects (S2, S3,S5, S7, and S9) who participated in the presentexperiment (Galvin & Fu, 2004a, 2004b). Pitch dis-crimination performance was measured for a set ofsix adjacent electrodes located in the middle of theelectrode array; stimuli were presented at loudness-balanced, comfortably loud listening levels. Eachelectrode pair was compared to every other electrodepair in the stimulus set a minimum of 10 times.During testing, subjects were asked to choose whichof two electrode pairs was higher in pitch. Theresults were transformed into units of discriminabil-ity (or d’; from Hacker & Ratcliffe, 1979). As with theMCI and FMI data, there was great variability insubjects’ accumulated d’ values across the stimulusset (ranging from 0.05 to 7.85 for a set of 6 adjacentelectrodes). There was no correlation between accu-mulated d’ and MCI baseline performance (linearregression: r2 � 0.441, p � 0.229), accumulated d’and FMI (no rhythm cues) baseline performance(r2 � 0.816, p � 0.096), accumulated d’ and MCIpost-training performance (r2 � 0.263, p � 0.377), oraccumulated d’ and FMI (no rhythm cues) post-training performance (r2 � 0.262, p � 0.489). Inter-estingly, subject S3 had a relatively low accumu-lated d’ value (0.46), yet was one of the betterperformers in the MCI task (and one of the poorestperformers in the FMI task). Conversely, subject S5had a relatively high accumulated d’ value (6.705),but was only a moderate performer in the MCI task.

Single-electrode pitch discrimination data mightnot provide the ideal comparison to the music percep-tion data in the present study. First, the MCI and FMIstimuli used three-tone complexes, in which the har-monics were increasingly attenuated relative to thefundamental frequency. While the fundamental fre-quency may have contained the most energy, the othercomponents may have contributed inconsistently, dueto interactions with the frequency allocation and theelectrode pitch. This spectral profile would undoubt-edly be quite different from the single-electrode profilemeasured in the pitch discrimination task. Moreover,cochlear implant users lack the spectro-temporal finestructure details to resolve harmonic components,which may give rise to pitch uncertainty for mul-tichannel stimuli. When the energy of harmonic com-ponents is nonlinearly distributed (as can be the casewith many musical instruments), this pitch uncer-tainty may increase. Second, the MCI and FMI tasksmay require different perceptual processes than thesingle-electrode pitch discrimination task. For the


electrode pitch discrimination experiment, perceptionmay be relatively more peripherally based (i.e., simplediscrimination between two channels). The MCI taskmay require attention to pitch only to support a broadprofile analysis of the melodic pattern, whereas theFMI task may require careful attention to the exactintervals supported by the electrode pitch. Both theMCI and FMI tasks involve higher processes, whichwhile supported by the pitch cues at the periphery,may make use of these cues in different ways.

Also, although not directly measured in thepresent study, modulation detection thresholds weremeasured for the same 5 subjects (S2, S3, S5, S7,and S9) as in the electrode pitch discriminationexperiment; some of these data were previouslypresented in Galvin & Fu (2005). Modulation detec-tion thresholds (MDTs) were measured for a varietyof loudness-balanced stimulation levels that spannedthe electrode DR, a range of stimulation rates (250to 2000 pps), and a range of modulation frequencies(5 to 100 Hz). There were no correlations betweenmean MDTs (across each subject’s DR) and MCI orFMI performance, for any of the stimulation rate/modulation frequency conditions. Note that therewas not adequate power for these analyses, due tothe limited number of subjects. However, similar tothe electrode pitch discrimination data, good modu-lation sensitivity did not always produce the bestMCI performance. For example, the mean MDT for a1000-pps carrier (across all stimulation levels andmodulation frequencies) for subject S5 was –34.79dB, whereas MCI performance was 64.6% correct.For subject S2, the mean MDT was –23.52 dB, whileMCI performance was 89.0% correct.

Although it is difficult to know whether a largernumber of subjects may produce better correlationsbetween simple, single-channel psychophysical mea-sures and multichannel MCI performance, it is prob-able that large intersubject variability will persist inall measures. As yet, the relative contributions ofsingle-channel electrode pitch and rate pitch tomultichannel melody perception are unclear.

Viability of the MCI Testing and Training

The present results suggest that the MCI testmay be a viable approach towards measuring thelimits of the cochlear implant to convey melodicinformation. The MCI test may hold some advan-tages over simple electrode pitch discrimination orFMI tests, in terms of assessing and quantifyingcochlear implant users’ melody perception. First, thetest requires no previous familiarity with the stim-uli. FMI testing depends strongly on listeners’ famil-iarity with the test melodies, and may not generalizeto perception of new, unfamiliar melodies. Second,

the MCI test can be repeated as necessary over theshort term, allowing changes in speech processing tobe evaluated for music perception; this test repeat-ability may reduce potential learning effects associ-ated with repeated FMI testing over the short term.Third, the MCI test directly tests melody perception,without the confounding factors of rhythm cues (ortheir removal) associated with FMI testing. Fourth,preliminary data shows high levels of performancefor NH listeners in the MCI test, comparable to NHperformance in FMI tests. Fifth, the MCI test isflexible in terms of the melodic contours, frequencyranges, and instrument stimuli used for evaluation.Multichannel stimuli may also better reflect “real-world” melody perception capabilities than single-channel pitch discrimination (whether pitch is en-coded by electrode place or stimulation rate).

The MCI test may also be appropriate for prelin-gually deafened subjects, who may more easily learnand understand the melodic contours shown on-screen than phoneme/word labels used in a speechtest. For prelingually deafened patients, the MCItest may be more accessible, as these patients mayhave little or no experience with the familiar melo-dies typically used for testing. Unfamiliarity withthese familiar melodies may also extend to cochlearimplant users from other cultures, for whom theMCI test may be more appropriate. Also, for pediat-ric cochlear implant users who have yet to developrobust central speech pattern templates, the MCItest may be useful for quantifying the ability of theimplant device to convey meaningful, complex audi-tory information. Once quantified, adjustmentscould be made to the speech processor to providebetter auditory resolution, which might benefitspeech perception, especially during stages of lan-guage development. However, such an application ofthe MCI test would need to consider issues of childdevelopment, both in terms of cognition and hear-ing. Conceivably, the test could be made appropriatefor young children who may not understand differ-ences between pitch and loudness. Further studiesmay guide the development of an MCI test that isappropriate for pediatric implant users.

Preliminary results also suggest that MCI train-ing may significantly benefit cochlear implant users’overall music perception and appreciation. Becausethe melodic contour stimuli differ in terms of quan-tifiable acoustic features, the training protocol caneasily target difficult contrasts for individual pa-tients, whether in terms of frequency range, intona-tion, or contour. Because the preliminary resultsgeneralized to improved FMI performance, MCImay be useful for developing cochlear implant users’music perception skills that require greater attentionto pitch cues, which may be only weakly represented


by their device. MCI training may also provide somemomentum in developing music appreciation via thecochlear implant. For postlingually deafened co-chlear implant users, speech recognition and speechquality is often initially poor through the implant.As listeners gain experience with their device,speech recognition and speech quality tend to dra-matically improve. Cochlear implant listeners maynot have equivalent meaningful listening opportuni-ties for music as they do for speech, and thus maynever experience similar adaptation for music. Cer-tainly, the listening demands are quite different forspeech and music, and cochlear implant processingdegrades music information more severely thanspeech information. While robust central speechpattern templates may allow for easier adaptation tospectrally degraded speech signals, music percep-tion requires greater attention to comparably weakcues. Regular training may help in the adaptationprocess, allowing melodic pattern recognition togrow stronger with experience.

It should be noted that although “musical,” themelodic contours used in the MCI test are not “music”per se. Music involves a complexity of rhythm, melody,harmony, and dynamics that are not represented bythe melodic contours used in the present study. Assuch, cochlear implant subjects may have only identi-fied “nonspeech” patterns (whose components arbi-trarily correspond to frequencies of the Western musi-cal scale); the same might be said for FMI melodieswithout rhythm cues. However, in the present study,we were most interested in quantifying the ability ofthe cochlear implant device to represent melody. Sim-ple, single-channel psychophysical measures (e.g.,electrode pitch discrimination, stimulation rate dis-crimination, etc.) may overestimate or underestimatecochlear implant users’ music perception performance.Similarly, the simple contours and notes used in theMCI test may not predict other aspects of cochlearimplant users’ music perception. Subjective ratingsmay best reflect the quality of music perception via thecochlear implant. However, the MCI test provides somequantification of the implant’s ability to represent animportant component of music (melodic contour), and tomeasure any benefits with alternative cochlear implantsound processing to music perception.

SUMMARY AND CONCLUSIONS

In the present study, MCI and FMI with three-harmonic complexes were measured in cochlear im-plant users while listening to their clinically as-signed speech processors. MCI training was alsoconducted in some of these cochlear implant users.Although the subject pool was somewhat limited (11cochlear implant users), preliminary data show:

1. There was great intersubject variability inMCI and FMI performance. Despite differ-ences in manufacturers’ devices and speechprocessors, there was no clear advantage forany particular cochlear implant device or pro-cessor. FMI performance depended greatly onthe availability of rhythm cues; without rhythmcues, FMI performance was generally poor.

2. Top performers were able to correctly identifymost melodic contours when there were 2 ormore semitones between successive notes inthe contour. Poor performers identified 40% orless of the melodic contours, even when therewere 5 semitones between successive notes.These limitations in performance are mostlikely due to the limited spectral resolution (inboth transmission and reception) provided bythe cochlear implant device.

3. There was a significant correlation betweencochlear implant subjects’ MCI and vowel rec-ognition performance. There was no significantcorrelation between MCI and FMI, or betweenFMI and phoneme recognition.

4. Moderate MCI training with alternate fre-quency ranges from those used in testing sig-nificantly improved cochlear implant users’MCI test performance. This improvement waslargely retained even a month after traininghad been stopped. MCI training also general-ized to improved FMI performance.

APPENDIX 1Analysis of melodic and rhythm components for

melodies used in the FMI task. Categories of analysisincluded the total number of notes in the melody, thenumber of novel notes (i.e., notes that were not re-peated), the number of times the note changed, therange of notes (in semitones); the frequency range ofF0 (in Hz); the number of novel beats (e.g., quarter-notes, half-notes, and so forth), the number of timesthe beat changed, and the total range of beats. Themelodic index was calculated as the mean of novelnotes, note changes and semitone range, each dividedby the total number of notes in the melody. Therhythm index was calculated as the mean of novelbeats, beat changes and beat range, each divided bythe total number of notes in the melody Table A1.

APPENDIX 2Analysis of melodic and rhythmic components

for melodies used in the MCI task are shown inTable A2. Categories are similar to those describedin Appendix 1.


TABLE A1. Analysis of melodic and rhythm components for melodies used in the FMI task

Familiar melodyTotalnotes

Novelnotes

Changesin note

Semi-tonerange

F0 range(Hz)

Melodicindex

Novelbeats

Changesin beat

Beatrange

Rhythmindex

Twinkle, twinkle little star 14 6 7 9 523–880 0.52 2 3 2.00 0.17Old MacDonald had a farm 12 5 6 9 440–740 0.56 3 3 2.00 0.22Brahm’s lullaby 13 5 10 8 440–698 0.59 4 9 4.00 0.44Star spangled banner 12 7 11 16 415–1047 0.94 4 3 4.00 0.31Happy birthday 12 5 9 7 523–784 0.58 4 7 8.00 0.53London bridge 13 5 12 7 587–880 0.62 2 5 2.00 0.23Mary had a little lamb 13 3 7 4 523–659 0.36 2 5 2.00 0.23This old man 13 5 11 7 494–740 0.59 2 4 2.00 0.21Yankee doodle dandy 14 5 11 9 440–740 0.60 2 1 2.00 0.12She’ll be comin’ round the mountain 11 4 7 8 440–698 0.58 2 1 2.00 0.15Take me out to the ball game 13 6 12 12 440–880 0.77 4 5 5.00 0.36Auld lang syne 15 6 13 13 415–880 0.71 4 10 6.00 0.44Mean 12.92 5.17 9.67 9.08 279 0.62 2.92 4.67 3.42 0.28SD 1.08 1.03 2.39 3.20 178 0.14 1.00 2.84 2.02 0.13

TABLE A2. Melodic and rhythmic components for melodies used in the MCI task

Melodiccontour

Totalnotes

Novelnotes

Changesin note

Semi-tonerange

F0 rangeA3 (Hz)

F0 rangeA4 (Hz)

F0 rangeA5 (Hz)

Melodicindex

Novelbeats

Changesin beat

Beatrange

Rhythmindex

Rising 5 5 4 20 220–698 440–1397 880–2794 2.00 1 0 0 0.07Rising 5 5 4 16 220–554 440–1109 880–2217 1.73 1 0 0 0.07Rising 5 5 4 12 220–440 440–880 880–1760 1.47 1 0 0 0.07Rising 5 5 4 8 220–349 440–698 880–1397 1.20 1 0 0 0.07Rising 5 5 4 4 220–277 440–554 880–1109 0.93 1 0 0 0.07Flat-Rising 5 3 2 10 220–392 440–784 880–1568 1.07 1 0 0 0.07Flat-Rising 5 3 2 8 220–349 440–698 880–1397 0.93 1 0 0 0.07Flat-Rising 5 3 2 6 220–311 440–622 880–1245 0.80 1 0 0 0.07Flat-Rising 5 3 2 4 220–277 440–554 880–1109 0.67 1 0 0 0.07Flat-Rising 5 3 2 2 220–247 440–494 880–988 0.53 1 0 0 0.07Falling-Rising 5 3 4 10 220–392 440–784 880–1568 1.20 1 0 0 0.07Falling-Rising 5 3 4 8 220–349 440–698 880–1397 1.07 1 0 0 0.07Falling-Rising 5 3 4 6 220–311 440–622 880–1245 0.93 1 0 0 0.07Falling-Rising 5 3 4 4 220–277 440–554 880–1109 0.80 1 0 0 0.07Falling-Rising 5 3 4 2 220–247 440–494 880–988 0.67 1 0 0 0.07Rising-Flat 5 3 2 10 220–392 440–784 880–1568 1.07 1 0 0 0.07Rising-Flat 5 3 2 8 220–349 440–698 880–1397 0.93 1 0 0 0.07Rising-Flat 5 3 2 6 220–311 440–622 880–1245 0.80 1 0 0 0.07Rising-Flat 5 3 2 4 220–277 440–554 880–1109 0.67 1 0 0 0.07Rising-Flat 5 3 2 2 220–247 440–494 880–988 0.53 1 0 0 0.07Flat 5 1 0 0 220–220 440–440 880–880 0.13 1 0 0 0.07Falling-Flat 5 3 2 10 220–392 440–784 880–1568 1.07 1 0 0 0.07Falling-Flat 5 3 2 8 220–349 440–698 880–1397 0.93 1 0 0 0.07Falling-Flat 5 3 2 6 220–311 440–622 880–1245 0.80 1 0 0 0.07Falling-Flat 5 3 2 4 220–277 440–554 880–1109 0.67 1 0 0 0.07Falling-Flat 5 3 2 2 220–247 440–494 880–988 0.53 1 0 0 0.07Rising-Falling 5 3 4 10 220–392 440–784 880–1568 1.20 1 0 0 0.07Rising-Falling 5 3 4 8 220–349 440–698 880–1397 1.07 1 0 0 0.07Rising-Falling 5 3 4 6 220–311 440–622 880–1245 0.93 1 0 0 0.07Rising-Falling 5 3 4 4 220–277 440–554 880–1109 0.80 1 0 0 0.07Rising-Falling 5 3 4 2 220–247 440–494 880–988 0.67 1 0 0 0.07Flat-Falling 5 3 2 10 220–392 440–784 880–1568 1.07 1 0 0 0.07Flat-Falling 5 3 2 8 220–349 440–698 880–1397 0.93 1 0 0 0.07Flat-Falling 5 3 2 6 220–311 440–622 880–1245 0.80 1 0 0 0.07Flat-Falling 5 3 2 4 220–277 440–554 880–1109 0.67 1 0 0 0.07Flat-Falling 5 3 2 2 220–247 440–494 880–988 0.53 1 0 0 0.07Falling 5 5 4 20 220–698 440–1397 880–2794 2.00 1 0 0 0.07Falling 5 5 4 16 220–554 440–1109 880–2217 1.73 1 0 0 0.07Falling 5 5 4 12 220–440 440–880 880–1760 1.47 1 0 0 0.07Falling 5 5 4 8 220–349 440–698 880–1397 1.20 1 0 0 0.07Falling 5 5 4 4 220–277 440–554 880–1109 0.93 1 0 0 0.07Mean 5.00 3.44 2.93 7.32 129 258 517 0.98 1.00 0.00 0.00 0.07SD 0.00 0.95 1.10 4.70 110 220 441 0.40 0.00 0.00 0.00 0.00


ACKNOWLEDGMENTS

The authors wish to acknowledge the gracious efforts of all thecochlear implant users who participated in the experiments, aswell as the extremely helpful comments from three anonymousreviewers. This research was supported by NIDCD grant R01-DC004792.

Address for correspondence: John J. Galvin III, Department ofAuditory Implants and Perception, House Ear Institute, 2100West Third Street, Los Angeles, CA 90057. E-mail: [email protected].

Received December 12, 2005; accepted December 7, 2006.

REFERENCES

Bigand, E. (2003). More about the musical expertise of musicallyuntrained listeners. Annals of the New York Academy ofSciences, 999, 304–312.

Burns, E. M., Viemeister, N. F. (1976). Nonspectral pitch. Jour-nal of the Acoustical Society of America, 60, 863–869.

Burns, E. M., Viemeister, N. F. (1981). Played-again SAM:further observations on the pitch of amplitude-modulatednoise. Journal of the Acoustical Society of America, 70, 1655–1660.

Fishman, K., Shannon, R. V., Slattery, W. H. (1997). Speechrecognition as a function of the number of electrodes used inthe SPEAK cochlear implant speech processor. Journal ofSpeech and Hearing Research, 40, 1201–1215.

Friesen, L. M., Shannon, R. V., Slattery, W. H. (1999). Effects offrequency allocation on phoneme recognition with the Nucleus-22 cochlear implant. American Journal of Otology, 20, 729–734.

Fu, Q.-J., Galvin, J. J. III, Wang, X., Nogaki, G. (2005). Moderateauditory training can improve speech performance of adultcochlear implant users. Acoustic Research Letters Online, 6,106–111.

Fu, Q.-J., Nogaki, G. (2005). Noise susceptibility of cochlearimplant users: the role of spectral resolution and smearing.Journal of the Association for Research in Otolaryngology, 6,19–27.

Fu, Q.-J., Shannon, R. V., Galvin, J. J. III. (2002). Perceptuallearning following changes in the frequency-to-electrode as-signment with the Nucleus-22 cochlear implant. Journal of theAcoustical Society of America, 112, 1664–1674.

Fu, Q.-J., Shannon, R. V., Wang, X. (1998). Effects of noise andspectral resolution on vowel and consonant recognition: acous-tic and electric hearing. Journal of the Acoustical Society ofAmerica, 104, 3586–3596.

Fujita, S., Ito, J. (1999). Ability of Nucleus cochlear implantees torecognize music. Annals of Otology, Rhinology and Laryngol-ogy, 108, 634–640.

Galvin, J. J. III, Fu, Q.-J. (2005). Effects of stimulation rate,mode, and levels on modulation detection by cochlear implantusers. Journal of the Association for Research in Otolaryngol-ogy, 6, 269–279.

Galvin, J. J. III, Fu, Q. J. (2004a). Place-pitch and excitationpatterns in cochlear implant listeners. Journal of the Acousti-cal Society of America, 115, 2385.

Galvin, J. J. III, Fu, Q. J. (2004b). Interactions between thefrequency allocation, stimulation mode, and pitch perceptionon speech recognition by cochlear implant listeners. Journal ofthe Acoustical Society of America, 116, 2524.

Gfeller, K., Lansing, C. R. (1991). Melodic, rhythmic, and timbralperception of adult cochlear implant users. Journal of Speechand Hearing Research, 34, 916–920.

Gfeller, K., Woodworth, G., Robin, D. A., Witt, S., Knutson, J. F.(1997). Perception of rhythmic and sequential pitch patterns bynormally hearing adults and adult cochlear implant users. Earand Hearing, 18, 252–260.

Gfeller, K., Christ, A., Knutson, J. F., Witt, S, Murray, K. T.,Tyler, R. S. (2000). Musical backgrounds, listening habits, andaesthetic enjoyment of adult cochlear implant recipients. Jour-nal of American Academy of Audiology, 11, 390–406.

Gfeller, K., Wiit, S., Adamek, M., Mehr, M., Rogers, J, Stordahl,J., Ringgenberg, S. (2002). Effects of training on timbre recog-nition and appraisal by postlingually deafened cochlear im-plant recipients. Journal of the American Academy of Audiol-ogy, 13, 132–145.

Gordon, E. E. (1979). Primary measures of music audiation.Chicago: G.I.A. Publications.

Hacker, M. J., Ratcliffe, R. (1979). A revised table of d’ forM-alternative forced choice. Perception and Psychophysics, 26,168–170.

Henry, B. A., McKay, C. M., McDermott, H. J., Clark, G. M.(2000). The relationship between speech perception and elec-trode discrimination in cochlear implantees. Journal of theAcoustical Society of America, 108, 1269–1280.

HoutsmaA. J., Smurzynski, J. (1990). Pitch identification anddiscrimination for complex tones with many harmonics. Jour-nal of the Acoustical Society of America, 87, 304–310.

Kong, Y. Y., Cruz, R., Jones, J. A., Zeng, F.-G. (2004). Musicperception with temporal cues in acoustic and electric hearing.Ear and Hearing, 25, 173–185.

Kong, Y. Y., Stickney, G. S., Zeng, F.-G. (2005). Speech andmelody recognition in binaurally combined acoustic and elec-tric hearing. Journal of the Acoustical Society of America, 117,1351–1361.

Laneau, J., Moonen, M., Wouters, J. (2004). Relative contribu-tions of temporal and place pitch cues to fundamental fre-quency discrimination in cochlear implantees. Journal of theAcoustical Society of America, 106, 3606–3619.

Loeb, G. E. (2005). Are cochlear implant patients suffering fromperceptual dissonance, Ear and Hearing, 26, 435–450.

Lynch, M. P., Eilers, R. E., Oller, K. D., Urbano, R. C., Wilson, P.(1991). Influences of acculturation and musical sophisticationon perception of musical interval patterns. Journal of Experi-mental Psychology: Human Perception and Performance, 17,467–475.

Madsen, C. K., Staum, M. J. (1983). Discrimination and interfer-ence in the recall of melodic stimuli. Journal of Research inMusic Education, 31, 15–31.

McDermott, H. J., McKay, C. M. (1997). Musical pitch perceptionwith electrical stimulation of the cochlea. Journal of the Acous-tical Society of America, 101, 1622–1631.

McKay, C. M., McDermott, H. J., Carlyon, R. P. (2000). Place andtemporal cues in pitch perception: Are they truly independent?Acoustic Research Letters Online, 1, 25–30.

Nelson, D. A., Van Tasell D. J., Schroder, A. C., Soli, S., Levine,S. (1995). Electrode ranking of “place pitch” and speech recog-nition in electrical hearing. Journal of the Acoustical Society ofAmerica, 98, 1987–1999.

Oxenham, A. J., Bernstein, J. G. W., Penagos, H. (2004). Correcttonotopic representation is necessary for complex pitch percep-tion. Proceedings of the National Academy of Sciences of theUnited States of America, 101, 1421–1425.

Pijl, S., Schwarz, D. W. F. (1995a). Melody recognition andmusical interval perception by deaf subjects stimulated withelectrical pulse trains through single cochlear implant electrodes.Journal of the Acoustical Society of America, 98, 886–895.

Pijl, S., Schwarz, D. W. F. (1995b). Intonation of musical intervalsby deaf subjects stimulated with single bipolar cochlear im-plant electrodes. Hearing Research, 89, 203–211.


Rosen, S., Faulkner, A., Wilkinson, L. (1999). Perceptual adapta-tion by normal listeners to upward shifts of spectral informa-tion in speech and its relevance for users of cochlear implants.Journal of the Acoustical Society of America, 106, 3629–3636.

Shannon, R. V. (1983). Multichannel electrical stimulation of theauditory nerve in man, I: basic psychophysics. Hearing Re-search, 11, 157–189.

Shannon, R. V., Fu Q.-J. Galvin, J. J. (2004). The number ofspectral channels required for speech recognition depends onthe difficulty of the listening situation. Acta-OtolaryngologySupplement, 552, 50–54.

Tong, Y. C., Clark, G. M., Blamey, P. J., Busby, P. A., Dowell, R.

C. (1982). Psychophysical studies for 2 multiple-channel co-chlear implant patients. Journal of the Acoustical Society ofAmerica 71, 153–160.

Shannon, R. V. (1992). Temporal modulation transfer functions inpatients with cochlear implants. Journal of the AcousticalSociety of America, 91, 2156–2164.

Smith, Z. M., Delgutte, B., Oxenham, A. J. (2002). Chimaeric soundsreveal dichotomies in auditory perception. Nature, 416, 87–90.

Wright, B. A., Fitzgerald, M. B. (2001). Different patterns ofhuman discrimination learning for two interaural cues tosound-source location. Proceedings of the National Academy ofSciences of the United States of America, 98, 12307–12312.


melodic contour identification by cochlear implant listeners

Documents