7

Now, as evidence accumulates that synesthesia is a bonafide neurological phenomenon, some areasking how this squares with some of the most

Touching Tastes, Seeing Smells—and Shaking Up

Brain Science

Sometimes the bizarre is just the bizarre, but when

the anomaly is a product of fundamental brain processes

yet seems to confound them, the theory makers have

a problem. For several decades, most neurologists who

heard about synesthetes—people who see colored

letters, feel colored pain, and taste shapes—just

shrugged their shoulders or rolled their eyes.

cherished conceptions of contemporary brainscience. The good news is that when old theoret-ical structures fall, new light may flood in.

by Richard E. Cytowic, M.D.

Some 20 years ago, my dinner host,Michael Watson, delayed our seatingby announcing, “There aren’t

person, object, or concept evokes synes-thetic sensations.

As children, synesthetes are surprisedto discover that others do not share theseexperiences. Often ridiculed and disbelieved,they learn to keep their atypical perceptionsto themselves. Nonetheless, the phenomenonremains involuntary and consistent throughouttheir lives. The trait runs strongly in families,and the genetics of its inheritance are reason-ably well understood. Some type of synestheticexperience occurs in perhaps 1 in 200 individ-uals, and more than 75 percent are women.

Like most anomalies that lie outside theexplanation of conventional theories, synes-thesia was long dismissed as a mere curiosityor, worse, as just subjective imagination. Iwondered how the brains of people likeMichael Watson might differ from the majori-ty, but my colleagues refused to accept thathis experience might be a bona fide neuro-logical phenomenon. Pointing to 200 years of

synesthesia history in the annals of medicineand psychology did not sway them. Today,however, researchers in some 15 countriesare studying synesthesia, and many doctoralcandidates have chosen it for their theses.

If others have gradually come toaccept the reality of synesthesia, they must

As children, synesthetes are

surprised to discover that others do

not share these experiences.

Often ridiculed and disbelieved,

they learn to keep their atypical

perceptions to themselves.

Cerebrum

8

enough points on the chicken.” He meantthat the taste still failed to evoke the pricklysensation he sought. To Michael, tastes andsmells were also felt as a physical touch inhis face and hands. “With an intense flavor,”he tried to explain, “a feeling sweeps downinto my hand, and I feel weight, texture,shape, and whether it’s hot or cold as if I’mactually grasping something.”

“Ah,” I exclaimed, “You have synesthesia.”

Michael looked stunned. “You meanthere’s a name for this thing?”

Sharing a Greek root with anesthesia,which means “no sensation,” synesthesiameans “joined sensation,” wherein two ormore senses are coupled in such a way thata voice, for example, is not only heard butalso felt, seen, or tasted. We call individualswith this coupling synesthetes.

The process of synesthesia usuallytravels in only one direction. For example,a sweet taste made Michael feel a cool,polished, curved surface in his hands, buthandling a billiard ball would not elicitany flavor in him. About 40 percent ofsynesthetes have multiple types of synes-thesia; the couplings that have beenobserved by scientists do not include allpossible pairings. Sensing letters, numbers,or words as colored accounts for two-thirds of the instances of synesthesia. Forthose with “colored hearing,” soundsevoke colored shapes that arise, move,alter, and fade—somewhat like fireworks.For others, merely thinking about a certain

The Dana Forum on Brain Science

9

now relinquish some received wisdom abouthow the brain works. Our concepts of howthings work are but models, after all, reduc-tions of reality that arise from human minds;history has shown repeatedly that reality has a way of making a mess of neat and tidyconcepts. Like most exceptions of nature,synesthesia is forcing a paradigm shift. Onecannot admit a wrecking ball and expect thehouse to remain standing. Paradoxically, thevery thing that destroys simultaneously illu-minates, and what emerges may surprise us.

SUBJECTIVE REALITY OR POPPYCOCK?

In 1989, I reported my initial studies onseveral dozen synesthetes in Synesthesia: A Union of the Senses. Here I proposed thatthe phenomenon pointed to deep cogni-tion, meaning fundamental processes thatunderlie how we perceive and think. It was“a voice in the wilderness,” to quote one of today’s researchers, V. S. Ramachandran.In fact, 11 years before my own effort,Lawrence Marks, a psychophysicist at Yale,suggested in The Unity of the Senses thatunderstanding synesthesia might shed lighton the perceptual basis of metaphor andperhaps even the acquisition of languageitself. He, too, was mostly ignored.

Often throughout its history, synes-thesia had been dismissed simply becausethe condition is revealed only through anindividual’s self-reported mental state.There is no test for it in the usual sense ofthat word. The complaint that introspectionis inherently unreliable and therefore imper-missible as scientific data has a long history.In the 19th century, psychophysicists suchas Gustav Fechner tried to formulate laws

regarding sensation and perception basedon observers’ reports, taking as a given thatmental states exist. Scientists in the 20thcentury, however, consistently strove toeliminate the subjective role of a humanobserver in gathering empirical data. Withinpsychology, the triumph of behaviorist

theory further ensured that inquiry intomental life would remain taboo for decades.

Because a technological focus domi-nated science in general and medicine inparticular, my neurology colleagues unsur-prisingly asked what Michael’s CAT scanshowed. In questioning synesthesia’s reality,they sought a third-person technologicalverification of a first-person experience.Technical corroboration is one thing; butthe sweeping assumption that anyone’s personal experience is invalid is quite another.Even current functional brain imaging,which is supposed to be anatomically objec-tive, starts with what one wants to verifyobjectively: the subject’s state of mind.

Synesthesia refuses to be ignored,affirming loudly that subjective mentalworlds do exist. Among other things, there-fore, synesthesia grants us an opportunityto examine the dichotomy of objective-subjective experience. But its importancegoes deeper than that.

Like most exceptions of nature,

synesthesia is forcing a paradigm

shift. One cannot admit a

wrecking ball and expect the house

to remain standing.

Cerebrum

10

never satisfied, for instance, with saying‘blue,’ but take a great deal of trouble to express or match the particular bluethey mean.”

Indeed, one can take the details of a given synesthete today and findmatching examples in the classical scientificliterature, linking the efforts of scientists a century ago with those of contemporaryones. Ironically, it is precisely synesthetes’subjective claims that now form the basis of today’s experiments that addresspredictions regarding the trait’s perceptualreality.

WHAT DEFINES SYNESTHESIA?

Synesthesia can be acquired via epilepsy orthe ingestion of hallucinogens such asmescaline or LSD, but idiopathic (or devel-opmental) synesthesia arises naturally with-out an external agent or brain abnormality.There is nothing in need of medical treat-ment. The subjective, ineffable, and idio-syncratic nature of this kind of synesthesiadoes make it an easy target for dismissal.Even the term “synesthesia” has been usedimprecisely over time, referring to everythingfrom metaphor (loud tie, sharp cheese,sweet voice) to deliberate contrivances suchas son et lumière theatrical performancesand “smellavision.”

A clear definition avoids a muddle.Idiopathic synesthesia is defined by fiveclinical findings: It is (1) involuntary and automatic, (2) spatially extended, (3) consistent and generic, (4) memorable,and (5) affect-laden. These refer to specificcharacteristics of the synesthetic person’sexperience.

A LINK WITH THE PAST

Compared with the hostility of modernobjectivists, a fair number of earlier scientistsaccepted synesthesia as a genuine phenom-enon after Sir Francis Galton’s 1880 reportin Nature on “visualized numerals”—ifonly because the individual stories soundedso similar, giving it the clinician’s feel of agenuine phenomenon. The earliest medicalreference, a case of sound-induced color,dates to 1710, but the style and details of Galton’s report make his the first recog-nizably modern one. For example, Galton’ssynesthetes express astonishment at discov-ering that they are unusual. Most claim to have had the ability as far back as theycan remember and, far from trying toappear special or call attention to themselves,genuine synesthetes prefer to hide theirtrait because of the ridicule they sufferupon disclosure.

The experience of synesthesia is difficult to express, as witnessed by the

collective struggle to convey exactly what issensed. Even computer animations are saidto be only about 60 percent representativeof “what it is really like.” As Galton notedin his 1883 Inquiries into Human Faculty,those with visual synesthesia are “invariablymost minute in their description of theprecise tint and hue of the color. They are

Compared with the hostility of

modern objectivists, a fair number of

earlier scientists accepted synesthesia

as a genuine phenomenon.

The Dana Forum on Brain Science

11

INVOLUNTARY AND AUTOMATIC

Synesthetes claim to hear a certain sound or to look at a letter, for example, and thento see a color. “It just happens,” some say.How can we demonstrate that they have nocontrol over their experiences? Phenomenacalled “perceptual grouping” and “pop-out”demonstrate that the response is indeedautomatic. For example, imagine that agroup of 2’s arranged to form a triangle isembedded in an array of 5’s drawn in a way

that the figures resemble mirror images.When told to look for a hidden shape, mostof us would take time to hunt down the target triangle buried within the distracting5’s. But a synesthete who sees every numeralas differently colored would immediately seethe target pop out of an alternatively coloredbackground. If the perception is involuntary,synesthetes should perform much faster thannonsynesthetes—and they do.

Because synesthetic associations areidiosyncratic, such tests must be tailored tothe individual. That is, two individuals with the same kind of synesthesia will rarelyagree as to the particulars of what they perceive. The numeral 2 may be green orred or turquoise for different people. Deliberately inducing mismatches—say,

by printing graphemes (a language’s written elements) in ink colors that areeither congruent or incongruent with a given synesthete’s perceptions—and thenmeasuring reaction times to them hasbecome a popular approach in current research.

In another setup, surrounding a targetgrapheme in the visual periphery with otherletters renders it “invisible,” meaning that itis not consciously perceived. Remarkably, itstill evokes the synesthetic color. “It must be‘A’ because I see red,” a subject will say. Thisimplies that synesthesia is evoked at an earlysensory level—a preconscious one, in fact.

As these examples show, many of theprobes designed to reveal whether synesthesiais automatic also turn out to prove thatsynesthesia is perceptual. What are calledrandom dot stereograms do even more,helping us identify the lowest brain level atwhich synesthesia can occur. When the lefteye looks at one pattern of black dots andthe right eye at another, the two imagesfuse in the brain, causing a three-dimen-sional object to pop out from the viewingplane. Synesthetes see the object, as every-one else does, but they see it in color. Thisresult says two things: that synesthetic colorarises after binocular fusion (setting the low-er brain limit above the first synaptic level ofvisual neurons in the cerebral cortex, calledV1), and that color appears to be bound to a form as the form is being recognized.

SPATIALLY EXTENDED

Some synesthetes describe Technicolorreading “on the page,” even as they simul-taneously see the black ink of the printing.Others with colored hearing speak, for

Two individuals with the same

kind of synesthesia will rarely

agree as to the particulars of what

they perceive. The numeral 2

may be green or red or turquoise for

different people.

Cerebrum

12

Tests of visual perception demonstrate the reality and consistency of synesthetes’ responses. These areexamples from synesthetes who see colored numbers—for example 5 is green and 2 is orange. In thetop illustration, the black numeral 5 composed of smaller 2’s is seen as green when this synesthetefocuses on the large figure but as orange when focusing on its components. In the middle series, a ran-dom dot stereogram presents one pattern of dots to the right eye and a different pattern to the left eye.The two images are fused in the brain (“binocular vision”) and, once again, the numeral 2 stands out asorange for the synesthete. At the bottom, a triangle of 2’s is imbedded in a pattern of 5’s; most of uswould have to hunt for the triangle, but it instantly pops out in red for this synesthete.

The Dana Forum on Brain Science

13

days of the week, and the like are not sensesat all; they are categories of knowledge.Because they reckon among the most fre-quent manifestations of synesthesia, weneed to enlarge our definition beyond puresensory-sensory pairings to include thebinding of sensory fragments (qualia) tocategories of mental concepts. I will returnto this later.

CONSISTENT AND GENERIC

Once established in childhood, synestheticassociations remain stable throughout life, asdemonstrated by tests and retests spanningmany years. For example, synesthetes maybe asked to indicate their color responses toa list of words. When tested without warninga year later, they report almost identicalresponses, whereas controls without synes-thesia, even if forewarned of retesting amonth before, perform near chance level.

Synesthetes often remark that somecolors they see are “weird”—ones that theywould never deliberately choose. They may see colors that they do not like or wishthat they saw their favorite ones moreoften. This should not be surprising, giventhat their visual systems are being stimulatedvia nonoptical means over which they haveno control. In one interesting example, acolor-blind synesthete with S-cone deficien-cy—which makes it hard to discriminateblues and purples—speaks of seeing num-bers in “Martian colors,” meaning colors heis unable to see in the real world. Curiously,synesthesia happens to be more common inblind individuals than the general population.

Saying that synesthesia is generic, as wellas consistent, means that what is experienced

example, of watching “a screen about sixinches from my nose.” Michael Watson oftenreached out in front of him to feel shapes atarm’s length. Even those who say the synes-thesia is in their “mind’s eye” remark that itdiffers from ordinary vision and imaginationby its quality of Euclidean locus, meaningthat it has a sense of physical place. That is,synesthetes speak of “going to” or “lookingat” a certain place to examine a sensation.

This quality of spatial extension is particularly dramatic in the perception ofwhat are called “number forms.” (The termis somewhat of a misnomer, given that num-ber forms concern not just integers but anyconcept involving serial order.) The percep-tual qualities of spatial location, shape, and,often, color become synesthetically joined to semantically ordered concepts such as integers, months, the alphabet, shoe sizes, temperature, and so forth. For example,each day of the week or month of the yearmay be associated with a different colored

shape, which is perceived in a location specific to the individual. Number forms areusually colored and create circles, zigzags,loops, and various tortured configurations.

Note that we may speak of synesthesiaas “joined senses”—a sound being associat-ed with a visual perception, for example—but spatial configuration, letters, words,

Synesthetes often remark that

some colors they see are “weird”—

ones that they would never

deliberately choose.

Cerebrum

14

AFFECT–LADEN

Synesthesia carries a sense of certitude, some-times a “Eureka!” feeling. Most find it highlypleasurable. Trivial tasks are laden with emo-tional affect, so that mental calculations are“very pleasurable” and recalling a phone num-ber is “delightful.” Mismatched perceptionscan be “like fingernails on a blackboard.”

In a minority of cases, what is perceived is so wretched—for example vile-tasting words, or nausea when playing a musical instrument—that the conditioninterferes with daily life. Nevertheless,synesthetes say that they would never partwith their perceptions. It is hard to overstate the intensity and pervasiveness ofaffect in synesthesia.

PICTURES, PLEASE

Synesthesia’s reality is demonstrated by itsautomaticity, consistency, and durability; by its induction of perceptual grouping andpop-out; by the evocation of colors by“invisible” graphemes at an unconsciouslevel; by its strong heritability as an X-linked dominant trait; by the fact that havingone type of synesthesia makes one morelikely to have a second or third type; and bythe ability of color-blind and blind personsto see colors.

Despite these kinds of proofs, someskeptics can be satisfied only by machineverifications that produce pictures of thebrain. What is remarkable is how profoundlythe emphasis of those pictures has switchedfrom structure to function. When, around20 years ago, my colleagues asked aboutMichael Watson’s CAT scan, they expectedthat a gross brain abnormality must underlie

is not complex and pictorial, but elementary—blobs, lattices, cold, rough, sour, zigzags,simple geometric shapes, and so forth.

MEMORABLE

When asked what good the trait does,synesthetes immediately answer, “It helpsyou remember.” They do have measurablyhigh memories, sometimes photographicones, “eidetic” in psychological parlance.The extra bits of information help synesthetesremember things like telephone numbersand names. As one synesthetic neuropathol-ogist puts it, “I use it…to help me remem-ber correct sequences of numbers, words,phrases, letters, to help me remember namesand locations of anatomical structures (espe-cially neuroanatomical structures—youshould see the beautiful array of colors inthe brain!) and neuropathological classifica-tions. I could go on and on.”

The memory expert that renownedRussian neuropsychologist Alexander Luriadescribed in The Mind of a Mnemonist

possessed a flawless memory because every-thing he recalled was accompanied bysynesthesiae in each of his senses: “I heardthe bell ringing... A small round objectrolled right before my eyes... My fingerssensed something rough like a rope... thena taste of saltwater... and something white.”

Synesthesia carries a sense of

certitude, sometimes a “Eureka!”

feeling. Most find it highly

pleasurable.

The Dana Forum on Brain Science

15

synesthesia if it were real. In other words,where was “the hole in his head”? But giventhat synesthetes such as Michael are normal,manifesting no evident neurological impair-ment, a structural lesion such as a stroke,tumor, or a bit of missing brain would beunlikely. As expected, his CAT and MRIscans, which assess structure, were normal.What was wanted was a test of function.

In 1980, I performed the first suchfunctional test on a synesthete, using atechnique called “regional cerebral blood

flow.” This showed that Michael’s brainbehaved much differently from nonsynestheticones, being strongly perturbed by ordinarystimuli such as smell. This study also con-firmed that synesthesia was a phenomenonof the brain’s left hemisphere. This left-brain locus disappoints some people, whowant it to be a right-brain function becausethey consider synesthesia artistic and creative.

In 1995, Eraldo Paulesu and col-leagues performed PET scans on six womenwho saw colors in response to spoken

Artists and composers who are synesthetes often seek to express their unique perceptions intheir works. This sculpture by synesthete artistCarol Steen is called Cyto, because it representsfor her the shapes and colors of the name[Richard] Cytowic, from whom she first gainedknowledge, beyond her own experience, of thewidespread phenomenon of synesthesia. “Theforms are constructed one on top of the other in a vertical arrangement,” she says, “because Ioften see flying colored forms appear that way inmy synesthetic visions.”

Cerebrum

16

words. PET offers superior spatial resolutionand other advantages to assessing functioncompared with my earlier technique. In thisstudy, spoken words activated auditory and language areas in both synesthetes andcontrols, but only in the synesthetes didthey also activate some visual areas.

Scientists have labeled only a few ofthe numerous cortical areas involved invision using a numbering scheme. V1, for-merly called the primary visual cortex, is the first level at which retinal projectionssynapse in the cortex. V1 acts like a postoffice, sorting and forwarding different

kinds of signals to different destinationswhere different types of transformations arecarried out, and so it is expected to activatein all visual tasks. At the second synapticlevel, V5 pertains to motion and direction,V4 to color, and V2 and V3 to form perception. At the fourth synaptic level,neither the areas pertaining to facial recog-nition nor spatial-location encoding has yet received a “V” label. Whereas Paulesu’sstudy did not show the hoped-for activationof the unique human color area, V4(probably due to a limitation of the PETtechnique), it did provide a result that was

Areas of the brain’s visual cortex are labeled according to their primary functions, V1 having to do withsorting the signals for various visual tasks, V2 and V3 relating to the perception of form, V4 relating to color, and V5 relating to motion and direction. Imaging studies show that, surprisingly, synesthetescan generate conscious visual experiences without activating V1 or V2.

V3A

V3

V1/V2

Face and object recognition areas

V5(motion)

Front

V4(color)

The Dana Forum on Brain Science

17

startling: a failure to activate V1 or V2 insynesthetes. These two early visual areas doactivate when control subjects view colors.

This result is inconsistent with amajor premise of what is called “blind-sight.” Some brain-damaged patients retaincapacities of which they are not conscious.Oxymoronic terms such as “blindsight” or“numbsense” convey how someone unableto see or feel can nonetheless discriminatevisual or tactile test targets with high accu-racy, despite insisting on not being able to“see” or “feel” anything. Because strickenindividuals are oblivious to their unconsciousknow-how that allows correct discrimination,researchers have postulated that the primarysensory cortex (such as S1, V1, A1), which isdamaged in these individuals, is indispensablefor any conscious awareness. In the words ofLawrence Weiskrantz, the acknowledgedauthority in the field, “striate cortex [V1] isessential…for any ‘seen’ [consciously expe-rienced] perception whatsoever.”

Not any longer. Synesthetes in thePET study proved that the brain can gener-ate conscious visual experiences without contribution from the primary visual cortex(V1). Blindsight’s implications for conscious-ness studies therefore need to be rethought.In the meantime, synesthesia supports theclaim by vision researcher Semir Zeki thatactivity in any given module sustaining a given visual function (V4 for color, V5 formotion, V3 for form) is sufficient, as well as necessary, for one to be conscious of thatcolor, motion, or form. That is, activation ofV4 alone is sufficient to “see” color, withoutthe necessity of recruiting other visualmodules, either upstream or downstream.

BE CAREFUL WHAT YOU WISH FOR

In 2002, a functional MRI (fMRI) study by Julia Nunn and her colleagues at lastconfirmed what was long expected: V4 acti-vation (without V1 or V2 activity) in synes-thetes who see color in response to spokenwords. Whereas both synesthetes and controlsactivated auditory and language areas asexpected, the synesthetes also activated thecolor area (V4), but only on the left—in

agreement with earlier results. Such lateral-ization is tantalizing, given that their colorexperiences were not confined to the rightvisual field. The fMRI technique, which isthe most refined one we have to date, alsodisclosed activation in areas concerned withmemory and emotion, again supportingboth the subjective statements and clinicalobservations of synesthetes.

An unexpected result of this study wasthat when actually viewing colored surfaces,synesthetes do not activate their left V4, the area for color. Right V4 did functionsimilarly for both synesthetes and controls.Ordinarily, viewing colors activates bothright and left V4, as well as the early visualareas V1 and V2. The implication, there-fore, is that the participation of left V4 insynesthetic color experience renders itunavailable for ordinary color perception—

Synesthetes in the PET study

proved that the brain can generate

conscious visual experiences

without contribution from the

primary visual cortex.

Cerebrum

18

projection from auditory speech areas to thevisual color area known as V4.

Those of us who study synesthesiamostly concur now that inheriting a geneticmutation results in a failure in synesthetes’brains to prune the projections betweenbrain structures that normally exist tem-porarily during the development of allbrains. This is what we call the “neonatalhypothesis” for synesthesia: Everyone is

born synesthetic, only to lose the capacityas the brain matures. Because it is not possible to directly map hardwiring in livinghumans, we are at present debating preciselywhere these projections might lie, anddreaming up ways to confirm or disproveour conjectures.

So, the objectivists have finally gottena machine proof of synesthesia, but it hasdisappointed their expectations.

CONVENTION UNDER THREAT

The existence of any physical projection as a basis for synesthesia threatens one ofcontemporary neuroscience’s widely heldconcepts, modularity. As initially proposedby Rutgers University philosopher JerryFodor, the mind is constructed of indepen-dent subsystems that receive inputs onlyfrom a specific category of stimulus andthat operate uninfluenced by activity in other modules or systems. The concept ofmodularity originally referred to cognitive

in other words, synesthesia appears to havehijacked an existing brain function. Thissurprise is consistent with the observationthat nonsynesthetes merely imagining colors(compared to performing a visual controltask not involving color) do not activateV4. Thus, the brain basis of synesthetic color experience is consistent with real colorperception rather than color imagery. Thisrefutes earlier criticisms that synesthetesare just “making it up” or have “overactiveimaginations.”

Lastly, this study has largely over-thrown the only strong alternative explana-tion of synesthesia, namely, that it resultsfrom childhood learning through associa-tion. This claim said that playing with refrigerator magnets or coloring books, forexample, makes some children form enduringassociations such as “ ‘A’ is red.” Rigorousefforts to train controls to imagine colors in response to words demonstrate that this isnot so. Despite training until controlsachieved 100 percent accuracy, they showedno activity whatsoever in V4 on either side. To further show that synesthetes didnot possess extraordinary associative skills,synesthetes who had claimed no spontaneouscolor response to music were trained to asso-ciate colors with a melody, as were controls;neither group had activity in the V4 regionthat had activated when synesthetes heardspoken words. Thus, not only was learningruled out as an explanation, but also the pat-terns of brain activity could easily distinguishthe subjective states that synesthetes claimedto experience (word-color) or denied having(music-color). Taken together, these resultssupport the existence of a direct neural

The objectivists have finally gotten a

machine proof of synesthesia, but it

has disappointed their expectations.

The Dana Forum on Brain Science

19

domains, but over time has extended intothe physical organization of the brain, suchthat relatively self-contained entities such as V1, V4, and the grapheme area are alsoreferred to as modules. The mental andphysical concepts are not wholly comparable,but this is not central to my point. Synes-thesia obviously raises the question ofwhether the concept of modularity per seremains entirely valid.

Another endangered favorite ofphilosophers and cognitive scientists isfunctionalism. This concept relates to whatis called the “hard problem of conscious-ness,” namely, the subjective aspect of perception. Functionalism describes therelations among sensory inputs and theirneural transformations, the resulting behavior,and our conscious experience. The concepthas engendered many varieties of philo-sophical argument. One popular formulationstates that each subjective experience(“quale,” plural “qualia”) is identical to thefunction with which it is associated. That is,functionalism replaces any supposition thatred “feels like” a certain state with, instead,an observable behavior, such as a personsaying “red” or pointing to it. Functionalismsays that qualia are the functions (input-processing-behavioral output) by whichthey are supported and nothing more.

If so, then two conditions incompatiblewith functionalism would be two qualiaproduced by a single function, or two func-tions producing the same quale. In synes-thetes, the quale of “red,” for example, canarise either by optical or nonoptical routes.This is an example of the second condition,since two different neural processes on

opposite sides of the brain, one optical andthe other synesthetic, are both subjectivelyexperienced as the color red.

Another argument put forth for func-tionalism is that functions giving rise toqualia must benefit the organism, becauseevolution selects for traits favoring survival.If this is correct, one should not encounterqualia that interfere with the functions ofwhich they are part. I have already men-tioned the situations where a perceptualmismatch slows performance, however, andI give many examples of sensory interferencein my textbook, to say nothing of theunpleasant and sometimes disruptive affectaccompanying some synesthesiae. Nor isthere any positive evidence that the quale ofcolor helps aural or visual word perception.These observations are incompatible withthe evolutionary claim of functionalism.

In 1997, Jeffrey Gray was the first tonotice the danger that synesthesia posed tothe hard question of consciousness, and he

has studied this problem in depth. Becausefunctionalism purports to be a generalaccount of consciousness, a single negativeinstance that it cannot explain is sufficientto render it invalid, just as the axiom “All swans are white” can be invalidated byobserving a single black swan. If functionalism

Two different neural processes

on opposite sides of the brain, one

optical and the other synesthetic,

are both subjectively experienced as

the color red.

Cerebrum

20

“I watched the black background become pierced by a bright red color that began to form in the middle of the rich velvet blackness. The red began as a small dot of color and grew quite large rather quickly, chasingmuch of the blackness away. I saw green shapes appear in the midst of the red color and move around thered and black fields.” Carol Steen

The Dana Forum on Brain Science

does not work in synesthesia, it does notwork anywhere and thus cannot be a generalaccount of consciousness.

The ready objections that synesthetesare not really seeing red—that they aremerely being artistic or metaphorical, orsaying what they do only because of a vividmemory of some past association such asrefrigerator magnets—have already beenaddressed. Because it is unlikely thatphilosophers will now succeed in eliminatingsynesthesia, they must either eliminate func-tionalism or refine it. I feel confident theywill choose the latter, because philosophersnever tire of arguing.

Lastly, synesthesia deals a blow to thestaunchest objectivists by showing clearlyhow perception is not passive, how it is notan impression in the brain transferred byobjective physics in the world “out there”(philosophers call this direct realism). Whena synesthete responds to the word “butter”by saying “blue circles moving off to theright,” she demonstrates a lack of correspon-dence, let alone an identity, between thephysical world “out there” that produces thepercept and the percept itself. Many otherapproaches have supported this notion thatperception is active and constructive; synes-thesia happily provides a clear example.

So much for the wrecking ball. Whatissues might synesthesia illuminate? Twobig ones are the so-called binding problemand metaphor.

THE BINDING PROBLEM

Diverse perceptual attributes (such as coloror shape) are processed in different areas ofmy brain, yet I perceive an apple as a unitary

21

CREATIVITY AND SYNESTHESIA

Painter and sculptor Carol Steen, whose

work appears on pages 7, 15, 20, and 25,

is one of many artists with synesthesia.

Touch, sound, smell, taste, and pain, as

well as letters and numbers, all give her

perceptions of colors and shapes, most of

which she experiences as internal. Loud

or unexpected sounds or sensations may

produce visions that she sees externally

or feels as compression waves through

her body.

Steen says, “The intensely brilliant,

luminous colors and simple, soft-edged

three-dimensional shapes are also textured

and kinetic, but cast no shadow. In these

rich visions, lustrous, vividly colored

shapes move in layers on equally saturated

colored fields in arbitrary spatial arrange-

ments almost faster than my vision can see

them and my memory can record them.

The shapes move, and the backgrounds

they appear against move as well.”

Many of Steen’s colored touch experi-

ences have arisen during acupuncture

treatments. Vision (1996), on the facing

page, was the first painting in which she

recorded such a vision. Aurora, (2002),

on page 7 was also inspired by Steen’s

perceptions during an acupuncture session.

She says, “What I paint matches my

experience only as closely as the medium

of paint will permit...The colors I see

synesthetically are the colors of light, not

of pigment.”

entity, not something red + round + edible.What is more, attributes are processed notonly in different locations but also at differ-ent times in my brain. For example, color isperceived before motion, which is perceivedbefore form. How all of these sundry, asynchronous attributes get bound into aseamless perception—red apple—endlesslybaffles neuroscientists. Inasmuch as synes-thesia binds perceptual qualia together inanomalous combinations, might it not say something useful about the process ofbinding in general?

There is a further twist. I mentionedearlier that synesthesia’s most commonmanifestation is a coupling of sensory qualiato categories of knowledge: for example,color, flavor, texture, locus, or configura-tion may be bound to letters and integers,members of a serially ordered set (such as

days of the week), words, or even symbolssuch as braille. Consider how many neuro-logical syndromes (the agnosias) as well asimaging studies demonstrate that we thinkin categories. In prosopagnosia, for example,stricken individuals can no longer recognizefaces. They recognize a face as a face, butcannot say whose face it is. Their larger fail-ure is in comprehending examples within a category. Thus, despite all their previous

Cerebrum

knowledge and skill, a stricken bird watchersays that all the birds look alike, a farmercan no longer distinguish his cows, and agardener cannot tell one plant from another.

Might synesthesia relate to the brain’ssearch for constancy and the assignment ofessential features that constitute a category?An enduring puzzle of neuroscience is how,out of a constantly changing and infiniteenergy flux, the brain—whose resources arefinite—assigns objects their constant features.

Color and form, so prominent insynesthesia, are properties constructed bythe brain through what are called constancyoperations. For example, most of us acceptthe explanation that something looks redbecause it reflects red wavelengths morethan others, but color is actually a propertyof brains and not of the physical world. Forsurface colors to be perceived as constantdespite ever-changing illumination, it is precisely the wavelength composition ofreflected light that the brain must ignore.Grass looks green, whether it is in brightsunlight or shade, despite large differencesin wavelength composition of the light.Similarly, all constructed properties requirethat the brain discount certain things. With color, it is wavelength composition ofreflected light that the brain must ignore;with form, it is the viewing angle; and withsize, it is viewing distance.

Synesthesia has led me over time tofavor a model of brain organization calledthe distributed system. The prime featuresof this model are a distribution of function(hence the name) across structures—as inneural networks—and simultaneity of activityon several levels, compared to the older and

22

An enduring puzzle of neuroscience

is how, out of a constantly changing

and infinite energy flux, the brain—

whose resources are finite—assigns

objects their constant features.

The Dana Forum on Brain Science

more familiar hierarchical and sequentialcascade, in which a module is assumed tocomplete its transformation of neural inputsbefore passing the result on to the nextmodule in the sequence. This older ideamay be likened to stations in a factory connected by a conveyor belt by means ofwhich one thing after another is added,

whereas the distributed system is like different authors simultaneously writingseparate chapters of a book without fullyknowing how the other chapters end. Thedistributed system also departs from theolder idea of a strict one-to-one mapping of function to anatomy, depending insteadon topological relations and convergent-divergent connections among brain modules.These two features result in the multiplemapping of a given function, as seen in thenumerous modules pertaining to vision,some of which we understand better thanothers (such as V4 for color, or V5 formotion and direction). Relevant to synes-thesia, what are called transmodal modules(meaning “not pertaining to any singlesense”) do three things: They constructmultisensory representations of the world,they provide memory and affect to experi-ence, and they critically participate inestablishing categories via groups of coarselytuned neurons.

This model organizes brain tissueinto five major networks and many lesserdistributed systems. In any one such system,

a given cerebral module participates inmore than one cognitive function and con-nects with several-to-many other nodes. Agiven function is not so much localized inthe sense of classical neurology, but exists asthe dominant process within its distributedsystem at any given time. Multiple synapticlevels are active simultaneously, each nodeinfluencing the state of adjacent levels (as inthe example of our simultaneous authors).Such organization reminds us that localiza-tion is a function of probability—and not just in this model but in any scheme ofneural organization. (Try drawing theboundaries of Wernicke’s area on a standardbrain atlas—you can’t.) Scans mislead us by emphasizing peak probabilities, whichwe misconstrue as fixedly anatomical. Theanswer to synesthesia will not be a “where”but a “what.”

It would thus be wrong for me toleave the impression that V4 is the seat of synesthesia: Any module found active bya scan (or other means) is really just onenode in the distributed system underlyingexpression. The totality of synesthetic expe-rience involves more than the consciousperception of a single quale, as I hope Ihave conveyed throughout this article. Mycomments regarding the participation oftransmodal modules in synesthesia are notincompatible with the idea, mentioned earlier, that an inherited genetic mutationcauses extraordinary, one-way projectionsbetween cerebral modules that underlievery specific functions. A connectionbetween, say, the grapheme area that allowsone to understand written numbers and theV4 color area does not fully “explain”

23

The answer to synesthesia will not

be a “where” but a “what.”

synesthetically colored numbers, however,because it leaves out the affect of theexperience, its memorability, whether thesynesthetic color moves, has a given spatiallocation, and so forth. As V. S. Ramachan-dran points out, what are called transcription factors can partly solve this shortcomingby causing the gene’s effects to beexpressed either discretely or diffusely—oranywhere between—in the brain. Suchvariability goes a long way toward explain-ing the observed variety of synestheticexperience, and why some people haveonly one kind whereas others have three

or four different kinds of synesthesia. Thus,transcription factors expressed in differentplaces through-out the brain could account,theoretically at least, for subsidiary featuresof synesthesia such as memorability andaffective charge. But it is precisely thisnecessity of widespread expression that makesme point out why synesthetic experienceper se cannot be localized to any one physicalspot in the brain and why scans mislead usin this regard.

METAPHOR AND LANGUAGE

The heterogeneity of the synesthetic expe-rience connotes more than wide variety of perceptual combinations. There is also

Cerebrum

heterogeneity in the depth of subjectiveexperience, from purely sensory-sensory, to categorical-sensory, to verbal-sensory. Inthis last, even a concept—just thinking of the number 5, say, or a person named Marion—is sufficient to trigger synesthesia.Some time ago, both Lawrence Marks andI proposed a cognitive continuum extend-ing from perception to synesthesia tometaphor to language. With time, othershave come to concur.

Systematic correspondences existamong dimensions of a given sense forsynesthetes and nonsynesthetes alike. Forexample, both say that louder tones arebrighter than soft tones, that higher onesare smaller than lower ones, and that lowtones are both larger and darker than highones. The perceptual similarities that yieldsuch orderly relationships among pitch,loudness, brightness, and size, for exam-ple, turn out to be rooted in fundamentalsimilarities of physical experience itself.Perceptual similarities, synesthetic equiva-lences, and metaphoric identities in turnbecome available to the more abstractknowledge that is embodied in language.In other words, the acquisition ofmetaphor relies not on a capacity for ver-bal abstraction, as many mistakenlybelieve, but on our physical interactionwith the world. The subjective-objectivedichotomy of experience should be turnedinto a unity, because we need both pointsof view.

Objectivity fails to see how the humansystem of concepts is metaphoric, involvingan imaginative understanding of one thing in terms of another. We elaborate the

24

I proposed a cognitive continuum

extending from perception to

synesthesia to metaphor to language.

With time, others have come

to concur.

The Dana Forum on Brain Science

metaphor “The mind is an entity” intoanother metaphor, “The mind is a machine,”when we say, “He ran out of steam.”Metaphors emphasize some aspects of anobject but hide others. The machinemetaphor paints the mind as having asource of power, an on-off state, and anexpected level of efficiency, but it hides the vagaries of thought, its ability to make

sense of fragmentary information, and the unexpected suddenness of insight. Byswitching metaphors, we alter how we comprehend a thing.

Subjectivity fails to see that even themost imaginative flights occur in a contextof objective experience gained by living ina physical and cultural world. Increasingly,science is viewing metaphor as an emergent

25

Synesthetes who see colored letters have their own individualalphabets. Carol Steen, who painted this representation of heralphabet, says she saw many of the more brightly colored lettersas a young child, but the iridescent and metallic colored letters didnot appear until she was in her 30s. Many synesthetes say thattheir perceptions become richer and more complex as they age.

property of mind that is rooted in thebody. As semiotics have long known,meaning inheres in affect, which the bodyfeels as physical and the mind apprehends as mental. Because metaphor perceives the similar in the dissimilar, it also pointsto constancy and categorization, features germane to synesthesia. Perhaps a tendencyto map one concept to another unconven-tionally even underlies what appearsto be synesthetes’ distinctive approach tocreativity.

One implication of a continuum fromperception to synesthesia to metaphor tolanguage is that synesthesia resides univer-sally in each of us but, for reasons yetunknown, rises to consciousness in only afew. Heinz Werner suggested as much inthe 1930s but technology takes time tocatch up with ideas. Two bits of recentwork support this conjecture. One study

found that synesthesia is 100 times morefrequent during Zen meditation; the otherconfirmed the ability of both blind andsighted persons to “see” video impulses fed into an electrode array placed on the tongue. We do not see with our eyes, anyway, but with our brains. What this latter demonstration shows is that tactilesensations on the tongue can be unconven-

Cerebrum

tionally bound to discern form, movement,direction, spatial location, and other qualiathat we conventionally ascribe to vision.The capacity for anomalous binding, whichis the essence of synesthesia, is thereforelatent in all brains.

Nature reveals herself through excep-tions. Those objectivists who tried to dis-miss synesthesia throughout its historyseem to have forgotten this maxim. Farfrom being a mere curiosity irrelevant toreal questions, synesthesia turns out to illu-minate a wide swath of mental life andforces us to rethink some fundamentalissues regarding mind and brain. At pre-sent, I can think of nothing more relevantto our quest for self-understanding. �

References

Bogen, JE, Bogen, GM. “Wernicke’s region–Where is it?”Annals of the New York Academy of Science 1975; 280:834-843.

Cytowic, RE. The Man Who Tasted Shapes. Cambridge.MIT Press, 1998.

Cytowic, RE. Synesthesia: A Union of The Senses, 2d ed.Cambridge. MIT Press, 2002.

Gray, JA, et al. “Possible questions of synesthesia forthe hard question of consciousness.” In S Baron-Cohen& JE Harrison, eds., Synaesthesia: Classic and Contem-

porary Readings. Oxford. Blackwell, 1997: 173-181.

Luria, AR. The Mind of a Mnemonist. New York. BasicBooks, 1968.

Marks, LE. The Unity of The Senses: Interrelations

Among the Modalities. New York. Academic Press, 1978.

Nunn, JA, et al. “Functional magnetic resonance imag-ing of synesthesia: activation of V4/V8 by spokenwords.” Nature Neuroscience 2002; 5(4): 571-575.

Paulesu, E, et al. “The physiology of coloured-hearing: a PET activation study of colour-word synaesthesia.”Brain 1995; 118: 661-676.

Ramachandran, VS, Hubbard EM. “Synesthesia: a window into perception, thought, and language.”Journal of Consciousness Studies 2001; 8(12): 3-34.

Zeki, S. A Vision of the Brain. Oxford. Blackwell, 1993.

26

Nature reveals herself through

exceptions. Those objectivists who

tried to dismiss synesthesia

throughout its history seem to have

forgotten this maxim.