gestalt isomorphism and the primacy of subjective conscious experience: a gestalt...

1. Introduction

Contemporary neuroscience finds itself in a state of seriouscrisis, for the deeper we probe into the workings of thebrain, the farther we seem to get from the ultimate goal ofproviding a neurophysiological account of the mechanismof conscious experience. Nowhere is this impasse more ev-ident than in the study of visual perception, where the ap-parently clear and promising trail discovered by Hubel andWiesel (1959) leading up the hierarchy of feature detectionfrom primary to secondary and to higher cortical areasseems to have reached a theoretical dead end. Besides thetroublesome issues of the noisy stochastic nature of theneural signal and the very broad tuning of the single cell asa feature detector, the notion of visual processing as a hier-archy of feature detectors seems to suggest some kind of“grandmother cell” model in which the activation of a sin-gle cell or a group of cells represents the presence of a par-ticular type of object in the visual field. However, it is notat all clear how such a featural description of the visualscene could even be usefully employed in practical interac-tion with the world.

Alternative paradigms of neural representation havebeen proposed, including the suggestion that synchronousoscillations play a role in perceptual representation, al-though these theories are not yet specified sufficiently to

know exactly how they address the issue of perceptual rep-resentation. But the most serious indictment of contempo-rary neurophysiological theories is that they offer no hint ofan explanation for the subjective experience of visual con-sciousness. Visual experience is more than just an abstractrecognition of the features present in the visual field – those

BEHAVIORAL AND BRAIN SCIENCES (2003) 26, 375–444Printed in the United States of America

© 2003 Cambridge University Press 0140-525X/03 $12.50 375

Gestalt isomorphism and the primacyof subjective conscious experience:A Gestalt Bubble model

Steven LeharPeli Lab, The Schepens Eye Research Institute, Boston MA [email protected] http ://cns-alumni.bu.edu/~slehar

Abstract: A serious crisis is identified in theories of neurocomputation, marked by a persistent disparity between the phenomenologi-cal or experiential account of visual perception and the neurophysiological level of description of the visual system. In particular, con-ventional concepts of neural processing offer no explanation for the holistic global aspects of perception identified by Gestalt theory. Theproblem is paradigmatic and can be traced to contemporary concepts of the functional role of the neural cell, known as the Neuron Doc-trine. In the absence of an alternative neurophysiologically plausible model, I propose a perceptual modeling approach, to model thepercept as experienced subjectively, rather than modeling the objective neurophysiological state of the visual system that supposedly sub-serves that experience. A Gestalt Bubble model is presented to demonstrate how the elusive Gestalt principles of emergence, reifica-tion, and invariance can be expressed in a quantitative model of the subjective experience of visual consciousness. That model in turnreveals a unique computational strategy underlying visual processing, which is unlike any algorithm devised by man, and certainly un-like the atomistic feed-forward model of neurocomputation offered by the Neuron Doctrine paradigm. The perceptual modeling ap-proach reveals the primary function of perception as that of generating a fully spatial virtual-reality replica of the external world in an in-ternal representation. The common objections to this “picture-in-the-head” concept of perceptual representation are shown to be illfounded.

Keywords: brain-anchored; Cartesian theatre; consciousness; emergence; extrinsic constraints; filling-in; Gestalt; homunculus; indirectrealism; intrinsic constraints; invariance; isomorphism; multistability; objective phenomenology; perceptual modeling; perspective; phe-nomenology; psychophysical parallelism; psychophysical postulate; qualia; reification; representationalism; structural coherence

Steven Lehar, Ph.D., is an independent researcher atthe Schepens Eye Research Institute in Boston, Mass.,USA. He is the author of twelve different papers on sub-jects ranging from new paradigms and forms of neuro-computation, to philosophical papers on epistemologyand the structure of conscious experience. A principlefocus of Lehar’s work is on the implications of Gestalttheory for the nature of perceptual computation andrepresentation in the brain, including the role of feed-back in visual processing, and harmonic resonance as anexplanation for a number of illusory grouping phenom-ena. Lehar is also author of The World In Your Head: AGestalt View of the Mechanism of Conscious Experience(2003; Erlbaum), a book that covers most of his theoriesacross a wide range of subjects from vision to cognitionto motor control. Lehar is winner of the 1999 WolfgangMetzger award for significant contribution to Gestalttheory, awarded by the Gestalt Theory and Applications(GTA) society.

https://doi.org/10.1017/S0140525X03410091Downloaded from https:/www.cambridge.org/core. Dartmouth College, on 08 May 2017 at 15:51:22, subject to the Cambridge Core terms of use, available at https:/www.cambridge.org/core/terms.

https://doi.org/10.1017/S0140525X03410091https:/www.cambridge.org/corehttps:/www.cambridge.org/core/terms

features are vividly experienced as solid three-dimensionalobjects, bounded by colored surfaces, embedded in a spatialvoid. A number of enigmatic properties of this world of ex-perience were identified decades ago by Gestalt theory, sug-gestive of a holistic emergent computational strategy whoseoperational principles remain a mystery.

The problem in modern neuroscience is a paradigmaticone that can be traced to its central concept of neural pro-cessing. According to the Neuron Doctrine, neurons behaveas quasi-independent processors separated by relativelyslow chemical synapses, with strictly segregated input andoutput functions through the dendrites and axon, respec-tively. It is hard to imagine how such an assembly of inde-pendent processors could account for the holistic emergentproperties of perception identified by Gestalt theory. Infact, the reason these Gestalt aspects of perception havebeen largely ignored in recent decades is exactly becausethey are so difficult to express in terms of the Neuron Doc-trine paradigm. More recent proposals that implicate syn-chronous oscillations as the neurophysiological basis of con-scious experience (Crick 1994; Crick & Koch 1990; Eckhornet al. 1988; Llinas et al. 1994; Singer 1999; Singer & Gray1995) seem to suggest some kind of holistic global processthat appears to be more consistent with Gestalt principles,although it is hard to see how this paradigm, at least as cur-rently conceived, can account for the solid three-dimen-sional nature of subjective experience. The persistent dis-parity between the neurophysiological and phenomenallevels of description suggests that either the subjective ex-perience of visual consciousness is somehow illusory, or thestate of our understanding of neural representation is farmore embryonic than is generally recognized.

Pessoa et al. (1998) made the case for denying the pri-macy of conscious experience. They argued that althoughthe subjective experience of filling-in phenomena is some-times accompanied by a neurophysiological correlate, suchan isomorphism between experience and neurophysiologyis not logically necessary but is merely an empirical issue.For, they claimed, subjective experiences can occur in theabsence of a strictly isomorphic correlate. Their view is thatalthough the subjective experience of visual consciousnessappears as a “picture” or three-dimensional model of a sur-rounding world, this does not mean that the informationmanifest in that experience is necessarily explicitly encodedin the brain. Moreover, that consciousness is an illusionbased on a far more compressed or abbreviated represen-tation, in which percepts such as that of a filled-in coloredsurface can be explained neurophysiologically by “ignoringan absence” rather than by an explicit point-for-point map-ping of the perceived surface in the brain.

In fact, nothing could be further from the truth. For topropose that the subjective experience of perception can bemore enriched and explicit than the corresponding neuro-physiological state, flies in the face of the materialistic basisof modern neuroscience. The modern view is that mind andbrain are different aspects of the same physical mechanism.In other words, every perceptual experience, whether a sim-ple percept such as a filled-in surface or a complex perceptof a whole scene, has two essential aspects, the subjective ex-perience of the percept and the objective neurophysiologi-cal state of the brain that is responsible for that subjectiveexperience. Like the two faces of a coin, these very differententities can be identified as merely different manifestationsof the same underlying structure, viewed from the internal

first-person perspective as opposed to the external third-person perspective. The dual nature of a percept is analo-gous to the representation of data in a digital computer,where a pattern of voltages present in a particular memoryregister can represent some meaningful information, suchas a numerical value, a brightness value in an image, or acharacter of text, when viewed from inside the appropriatesoftware environment, but when viewed in external physicalterms those same data take the form of voltages or currentsin particular parts of the machine. However, whatever formis selected for encoding data in the computer, the informa-tion content of that data cannot possibly be of higher di-mensionality than the information explicitly expressed in thephysical state of the machine.

The same principle must also hold in perceptual experi-ence, as proposed by Müller (1896) in the psychophysicalpostulate. Müller argued that because the subjective expe-rience of perception is encoded in some neurophysiologi-cal state, the information encoded in that conscious experi-ence cannot possibly be any greater than the informationencoded in the corresponding neurophysiological state. Al-though we cannot observe phenomenologically the physi-cal medium by which perceptual information is encoded inthe brain, we can observe the information encoded in thatmedium, expressed in terms of the variables of subjectiveexperience. It follows therefore that it should be possibleby direct phenomenological observation to determine thedimensions of conscious experience, and thereby to inferthe dimensions of the information encoded neurophysio-logically in the brain.

The bottom-up approach that works upward from theproperties of the individual neuron and the top-down ap-proach that works downward from the subjective experi-ence of perception are equally valid and complementaryapproaches to the investigation of the visual mechanism.Eventually, these opposite approaches to the problem mustmeet somewhere in the middle. To date, however, the gapbetween them remains as large as it ever was. Both ap-proaches are essential to the investigation of biological vi-sion because each offers its own unique perspective on theproblem. The disparity between these two views of the vi-sual representation helps to maintain the focus on exactlythose properties that are prominently absent from the con-ventional neural network view of visual processing.

2. The epistemological divide

There is a central philosophical issue that underlies discus-sions of phenomenal experience as seen, for example, in thedistinction between the Gestaltist and the Gibsonian viewsof perception. That is, the epistemological question ofwhether the world we see around us is the real world itselfor merely an internal perceptual copy of that world gener-ated by neural processes in our brain. In other words, thisis the question of direct realism (also known as naïve real-ism) as opposed to indirect realism (or representational-ism). To take a concrete example, consider the vivid spatialexperience of this paper that you hold in your hands. Thequestion is whether the rich spatial structure of this experi-ence before you is the physical paper itself, or an internaldata structure or pattern of activation within your physicalbrain. Although this issue is not much discussed in con-temporary psychology, it is an old debate that has resur-

Lehar: Gestalt isomorphism and the primacy of subjective conscious experience

376 BEHAVIORAL AND BRAIN SCIENCES (2003) 26:4https://doi.org/10.1017/S0140525X03410091Downloaded from https:/www.cambridge.org/core. Dartmouth College, on 08 May 2017 at 15:51:22, subject to the Cambridge Core terms of use, available at https:/www.cambridge.org/core/terms.


faced several times in psychology, and the failure to reachconsensus on this issue continues to bedevil the debate onthe functional role of sensory processing. The reason for thecontinued confusion is that both direct and indirect realismare frankly incredible, although each is incredible for dif-ferent reasons.

2.1. Problems with direct realism

The direct realist view is incredible because it suggests thatwe can have experience of objects out in the world directly,beyond the sensory surface, as if bypassing the chain of sen-sory processing. For example, if light from this paper istransduced by your retina into a neural signal that is trans-mitted from your eye to your brain, then the very first as-pect of the paper that you can possibly experience is the information at the retinal surface, or the perceptual repre-sentation that is downstream of it in your brain. The physi-cal paper itself lies beyond the sensory surface and there-fore must be beyond your direct experience. But theperceptual experience of the page stubbornly appears outin the world itself instead of in your brain, in apparent vio-lation of everything we know about the causal chain of vi-sion. Gibson explicitly defended the notion of direct per-ception and spoke as if perceptual processing occurssomehow out in the world itself rather than as a computa-tion in the brain based on sensory input (Gibson 1972,pp. 217, 239).

Significantly, Gibson refused to discuss sensory process-ing at all and even denied that the retina records anythinglike a visual image that is sent to the brain. This leaves thestatus of the sensory organs in a peculiar kind of limbo, forif the brain does not process sensory input to produce an in-ternal image of the world, what is the purpose of all thatcomputational wetware? Another embarrassment for directperception is the phenomenon of visual illusions, which areobserved out in the world itself; and yet they cannot possi-bly be in the world for they are the result of perceptual pro-cessing that must occur within the brain. With characteris-tic aplomb, Gibson simply denied that illusions are illusoryat all, although it is not clear exactly what he could possiblyhave meant by that. Modern proponents of Gibson’s theo-ries usually take care to disclaim his most radical views(Bruce & Green 1987, pp. 190, 203–204; O’Regan 1992,p. 473; Pessoa et al. 1998), but they present no viable alter-native explanation to account for our experience of theworld beyond the sensory surface.

The difficulty with the concept of direct perception ismost clearly seen when we consider how an artificial visionsystem could be endowed with such external perception.Although a sensor may record an external quantity in an in-ternal register or variable in a computer, from the internalperspective of the software running on that computer, onlythe internal value of that variable can be “seen” or can pos-sibly influence the operation of that software. In an exactlyanalogous manner the pattern of electrochemical activitythat corresponds to our conscious experience can take aform that reflects the properties of external objects, but ourconsciousness is necessarily confined to the experience ofthose internal effigies of external objects, rather than of theexternal objects themselves. Unless the principle of directperception can be demonstrated in a simple artificial sen-sory system, this explanation remains as mysterious as theproperty of consciousness it is supposed to explain.

2.2. Problems with indirect realism

The indirect realist view is also incredible, for it suggeststhat the solid stable structure of the world we perceive tosurround us is merely a pattern of energy in the physicalbrain; that is, the world that appears to be external to ourhead is actually inside our head. This could only mean thatthe head we have come to know as our own is not our truephysical head but is merely a miniature perceptual copy ofour head inside a perceptual copy of the world, all of whichis completely contained within our true physical skull.Stated from the internal phenomenal perspective, out be-yond the farthest things you can perceive in all directions(i.e., above the dome of the sky and below the earth underyour feet, or beyond the walls, floor, and ceiling of the roomyou perceive around you), beyond those perceived surfacesis the inner surface of your true physical skull encompass-ing all that you perceive, and beyond that skull is anunimaginably immense external world, of which the worldyou see around you is merely a miniature virtual-realityreplica. The external world and its phenomenal replica can-not be spatially superimposed, for one is inside your physi-cal head and the other is outside. Therefore, the vivid spa-tial structure of this page that you perceive here in yourhands is itself a pattern of activation within your physicalbrain, and the real paper of which it is a copy is out beyondyour direct experience.

I have found a curious dichotomy in the responses of col-leagues in discussions on this issue. Many people agree withthe statement that everything you perceive is in some senseinside your head, and in fact they often complain that thisis so obvious it need hardly be stated. However, when thatstatement is turned around to say that out beyond every-thing you perceive is your physical skull, they object mostvehemently that that is absurd. And yet the two statementsare logically identical, so how can one appear trivially obvi-ous while the other seems patently absurd? The value ofthis particular mental image is that it helps to smoke out anyresidual naive realism that may remain hidden in our phi-losophy. For although this statement can only be true in atopological, rather than a strict topographical, sense, this in-sight emphasizes the indisputable fact that no aspect of theexternal world can possibly appear in consciousness exceptby being represented explicitly in the brain. The existentialvertigo occasioned by this mental image is so disorientingthat only a handful of researchers have seriously enter-tained this notion or pursued its implications to its logicalconclusion (Harrison 1989; Hoffman 1998; Kant 1781/1991; Koffka 1935; Köhler 1971, p. 125; Lehar 2003b; Rus-sell 1927, pp. 137–143; Smythies 1989; 1994).

Another reason the indirect realist view is incredible isthat the observed properties of the world of experiencewhen viewed from the indirect realist perspective are diffi-cult to resolve with contemporary concepts of neurocom-putation. For the world we perceive around us appears as asolid spatial structure that maintains its structural integrityas we turn around and move about in the world. Perceivedobjects within that world maintain their structural integrityand recognized identity as they rotate, translate, and scaleby perspective in their motions through the world. Theseproperties of the conscious experience fly in the face ofeverything we know about neurophysiology, for they sug-gest some kind of three-dimensional imaging mechanism inthe brain, capable of generating three-dimensional volu-


BEHAVIORAL AND BRAIN SCIENCES (2003) 26:4 377https://doi.org/10.1017/S0140525X03410091Downloaded from https:/www.cambridge.org/core. Dartmouth College, on 08 May 2017 at 15:51:22, subject to the Cambridge Core terms of use, available at https:/www.cambridge.org/core/terms.


metric percepts of the degree of detail and complexity ob-served in the world around us. No plausible mechanism hasever been identified neurophysiologically which exhibitsthis incredible property. The properties of the phenomenalworld are therefore inconsistent with contemporary con-cepts of neural processing, which is exactly why these prop-erties have been so long ignored.

2.3. Spirituality , supervenience, and other nomologicaldanglers

The perceived incredibility of both direct and indirect re-alism has led many over the centuries to propose that con-scious experience is located neither in the physical brainnor in the external world, but in some separate space thatbears no spatial relation to the physical space known to sci-ence. These theories fall somewhere between direct and in-direct perception because they claim that phenomenal ex-perience is neither in the head, nor out in the world. Theoriginal formulation of this thesis was Cartesian dualism,the traditional religious or spiritual view that mind exists ina separate realm that is inaccessible to science. Our inabil-ity to detect spiritual entities is not due to any limitations ofour detector technology but to the fact that spiritual enti-ties are impossible in principle to detect by physical means.Cartesian dualism is a minority position in contemporaryphilosophy, at least as a scientific theory of mind, and forvery good reason. The chief objection to this kind of dual-ism is Occam’s razor: It is more parsimonious to posit a sin-gle universe with one set of physical laws rather than tworadically dissimilar parallel universes composed of dissimi-lar substances and following dissimilar laws, making tenu-ous contact with each other nowhere else but within a liv-ing conscious brain. But if mind and matter come intocausal contact, as they clearly do in both sensory and motorfunction, then surely they must be different parts of oneand the same physical universe. There is another, still moreserious objection to Cartesian dualism than the issue of par-simony. Since the experiential, or spiritual component ofthe theory is in principle inaccessible to science, that por-tion of the theory can be neither confirmed nor refuted.This places the spiritual component of Cartesian dualismbeyond the bounds of science and firmly in the realm of re-ligious belief.

A more sophisticated halfway epistemology is seen in thephilosophy of critical realism (Broad 1925; Drake et al.1920; Russell 1921; Sellars 1916). Critical realists avoid re-ligious explanations involving God or spirits, but their con-cept of conscious experience nevertheless preserves someof the mystery of Cartesian dualism. Critical realists ac-knowledge that perception is not direct, but instead, is me-diated by an intermediate representational entity calledsense-data. However, critical realists insist that sense-dataare

particular existents of a peculiar kind; they are not physical, . . .and there is no reason to suppose that they are either states ofmind or existentially mind-dependent. In having spatial char-acteristics . . . they resemble physical objects . . . but in theirprivacy and their dependence on the body . . . of the observerthey are more like mental states. (Broad 1925, p. 181)

As with the spirit world of the Cartesian view, sense dataand the space in which they are observed are not just diffi-cult to detect, but they are in principle beyond scientificscrutiny.

There is some debate among critical realists over the on-tology of conscious experience. In a book on critical realismby a consortium of authors (Drake et al. 1920), Lovejoy,Pratt, and Sellars claimed that the sensa are completely “thecharacter of the mental existent . . . although its existenceis not given” (pp. 20–21), while Drake, Rogers, Santayana,and Strong agreed that the data are characteristic of the ap-prehended object, although “the datum is, qua datum, amere essence, an inputed but not necessarily actual exis-tent. It may or may not have existence” (Drake 1920 inDrake et al. 1920, pp. 20–21, footnote). So the critical re-alists solved the epistemological problem by defining aunique kind of existent that is experienced, but that doesnot or may not actually exist. This is a peculiar inversion ofthe true epistemological situation because, in fact, sensedata, or the raw material of conscious experience, are theonly thing we can know with any real certainty to actuallyexist. All else, including the entire physical world known toscience, is informed conjecture based on that experience.

A more modern reformulation of this muddled episte-mology is seen in Davidson’s (1970) anomalous monist the-sis. Davidson suggested that the mental domain, on the ba-sis of its essential anomalousness and normativity, cannot bethe object of serious scientific investigation because themental is on a wholly different plane from the physical. Thisargument sounds like the metaphysical dualism ofDescartes which disconnects mind from brain entirely, ex-cept that Davidson qualified his theory with the monisticproviso that every mental event is connected with specificphysical events (in the brain), although there are no lawsconnecting mental kinds with physical kinds, and this pre-sumably rescues the thesis from metaphysical dualism. Kim(1998) pointed out, however, that this is a negative thesis,for it tells us only how the mental is not related to the phys-ical, it says nothing about how they are related. As such, thisis more an article of faith rather than a real theory of anysort, and in the context of the history of the epistemologi-cal debate this can be seen as a last desperate attempt torescue naïve realism from its own logical contradictions.This kind of physicalism has been appropriately dubbed“token physicalism,” for it is indeed a token admission of theundeniable link between mind and the physical brain, with-out admitting to any of its very significant implications.

To rationalize this view of the mind-brain relation,Davidson (1970) introduced the peculiar notion of super-venience, a one-way asymmetrical relation between mindand brain which makes the mind dependent on the brainbut forever closes the possibility of phenomenological ob-servation of brain states. As in the case of Cartesian dual-ism, there are two key objections to this argument. In thefirst place, the disconnection between the experientialmind and the physical brain is itself merely a hypothesiswhose truth remains to be demonstrated. It is at leastequally likely prima facie that the mind does not superveneon the brain, but rather that the mind is identically equal tothe functioning of the physical brain. In fact, this is by farthe more parsimonious explanation because it invokes a sin-gle explanans, the physical brain, to account for the prop-erties of both mind and brain. After all, physical damage tothe brain can result in profound changes in the mind, notjust in the information content of the mind or in observedbehavior but in the experiential or “what it is like” aspect ofconscious experience. The simplest explanation therefore isthat consciousness is a physical process taking place in the




physical brain, which is why it is altered by physical changesto the physical brain.

But the problem of supervenience is more serious thanjust the argument of parsimony. If the properties of mindare indeed disconnected from the properties of the physi-cal brain, this would leave the mental domain completelydisconnected from the world of reality known to science, aswhat Feigl (1958) has called a “nomological dangler.” If theproperties of mind are not determined by the properties ofthe physical brain, what is it that determines the propertiesof the mind? For example, phenomenal color experiencehas been shown to be reducible to the three dimensions ofhue, intensity, and saturation. Physical light is not restrictedto these three dimensions; the spectrum of a typical sampleof colored light contains a separate and distinct magnitudefor every spectral frequency of the light, an essentially infi-nite-dimensional space that is immeasurably greater in in-formation content than the three dimensions of phenome-nal color experience. In answer to Koffka’s (1935) classicalquestion “Why do things look as they do?”, the answer isclearly not “Because they are what they are.” That answeris clearly false in the case of color perception, as well as inthe cases of visual illusions, dreams, and hallucinations. Wenow know that the dimensionality of color experience re-lates directly to the physiology of color vision; it relates tothe fact that there are three different cone types in the hu-man retina and it relates to the opponent color process rep-resentation in the visual cortex. The dimensions of color ex-perience therefore are not totally disconnected from theproperties of the physical brain, as suggested by Davidson(1970), but in fact phenomenal color experience tells ussomething very specific about the properties of the repre-sentation of color in the physical brain. And the same argu-ment holds for spatial vision, for there are a number ofprominent distortions of phenomenal space which clearlyindicate that phenomenal space is ontologically distinctfrom the physical space known to science, as will be dis-cussed in section 6.3.

Daniel Dennett (1991) promoted a similar halfway epis-temology by drawing a distinction between the neural ve-hicles of mental representation and the phenomenal con-tents of those vehicles. Dennett opened the epistemologicalcrack by claiming that the phenomenal contents do not nec-essarily bear any similarity whatsoever to the neural vehi-cles by which they are encoded in the brain. This actuallygoes beyond Davidson’s supervenience because, accordingto Davidson (1970), mental events that are distinct phe-nomenally must also be distinct neurophysiologically. Thisis tantamount to saying that the dimensions of conscious ex-perience cannot be any less than the dimensions of the cor-responding neurophysiological state. Dennett effectivelyremoved this limitation by suggesting that even the dimen-sionality of the phenomenal contents need not match thatof the neural vehicles. And into that epistemological crack,Dennett slipped the entire world of conscious experiencelike a magical disappearing act, where it is experienced butdoes not actually exist. By the very fact that conscious ex-perience, as conceived by Dennett, is in principle unde-tectable by scientific means, this concept of consciousnessbecomes a religious rather than a scientific hypothesis,whose existence can be neither confirmed nor refuted byscientific means. In fact, Dennett even suggested that thereis actually no such thing as consciousness per se, and thatbelief in consciousness is akin to belief in some kind of

mythical nonexistent deity (Dennett 1981). This argumentof course is only intelligible from a naïve realist perspective,by which the sense-data of conscious experience, so plainlymanifest to one and all, are misidentified as the externalworld itself rather than as something going on in the phys-ical brain.

Another modern theorist, Max Velmans (1990), revivedan ancient notion of perception as something projecting outof the head into the world, as proposed by Empedocles andpromoted by Malebranche. But Velmans refined this an-cient notion with the critical realist proviso that nothingphysical actually gets projected from the head; the onlything that is projected is conscious experience, a subjectivequality that is undetectable externally by scientific means.But again, as with critical realism, the problem with this no-tion is that the sense-data that are experienced to exist donot exist in any true physical sense, and therefore the pro-jected entity in Velman’s theory is a spiritual entity to be be-lieved in (for those who are so inclined), rather than any-thing knowable by, or demonstrable to, science. Velmansdrew the analogy of a videotape recording that carries theinformation of a dynamic pictorial scene, expressed in ahighly compressed and nonspatial representation, as pat-terns of magnetic fields on the tape. There is no resem-blance or isomorphism between the magnetic tape and theimages that it encodes, except for its information content.However, the only reason the videotape even represents avisual scene is because of the existence of a video technol-ogy that is capable of reading the magnetic informationfrom the tape and sweeping it out as a spatial image on avideo monitor or television screen, where each pixel ap-pears in its proper place in the image. If that equipment didnot exist, there would be no images as such on the video-tape. But if video technology is to serve as an analogy forspatial representation in the brain, the key question iswhether the brain encodes that pictorial information exclu-sively in abstract compressed form like the magnetic pat-terns on the tape, or whether the brain reads those com-pressed signals and projects them as an actual spatial imagesomewhere in the brain like a television monitor, wheneverwe have a visuospatial experience. If it is the former, thensense-data are experienced but do not actually exist as a sci-entific entity, so the spatial image we see is a complete illu-sion, which, again, is an inversion of the true epistemology.If it is the latter, then there are actual “pictures in the head,”a notion that Velmans emphatically rejected.

In fact, the only epistemology that is consistent with themodern materialistic world view is an identity theory (Feigl1958; Russell 1927) whereby mind is identically equal tophysical patterns of energy in the physical brain. To claimotherwise is to relegate the elaborate structure of consciousexperience to a mystical state beyond the bounds of science.The dimensions of conscious experience, such as phenom-enal color and phenomenal space, are a direct manifesta-tion of certain physical states of our physical brain. The onlyright answer to Koffka’s question (Koffka 1935) is thatthings appear as they do because that is the way the worldis represented in the neurophysiological mechanism of ourphysical brain. In principle, therefore, the world of con-scious experience is accessible to scientific scrutiny after all,both internally through introspection and externallythrough neurophysiological recording. And introspection isas valid a method of investigation as is neurophysiology, justas in the case of color experience. Of course, the mind can




be expected to appear quite different from these two per-spectives, just as the data in a computer memory chip ap-pear quite different when examined internally by data ac-cess as opposed to externally by electrical probes. But theone quantity that is preserved across the mind/brain barrieris information content, and therefore that quantity can helpto identify the neurophysiological mechanism or principlein the brain whose dimensionality, or information content,matches the observed dimensions of conscious experience.

2.4. Selection from incredible alternatives

We are left therefore with three alternatives, each of whichappears to be absolutely incredible. Contemporary neuro-science seems to take something of an equivocal position onthis issue, recognizing the epistemological limitations of thedirect realist view and of the projection hypothesis, yet be-ing unable to account for the incredible properties sug-gested by the indirect realist view. However, one of thesethree alternatives simply must be true, to the exclusion ofthe other two. And the issue is by no means inconsequen-tial, for these opposing views suggest very different ideas ofthe function of visual processing, or what all that neuralwetware is supposed to actually do. Therefore, it is of cen-tral importance for psychology to address this issue head-on, and to determine which of these competing hypothesesreflects the truth of visual processing. Until this most cen-tral issue is resolved definitively, psychology is condemnedto remain in what Kuhn (1970) calls a pre-paradigmaticstate, with different camps arguing at cross-purposes due toa lack of consensus on the foundational assumptions andmethodologies of the science. Psychology is, after all, thescience of the psyche, the subjective side of the mind/brainbarrier, and neurophysiology only enters the picture to pro-vide a physical substrate for mind. Therefore, it is of vitalimportance to reach a consensus on the nature of the ex-planandum of psychology before we can attempt an ex-planans. In particular, we must decide whether the vividspatial structure of the surrounding world of visual experi-ence is an integral part of the psyche and thus within the ex-planandum of psychology, or whether it is the externalworld itself, as it appears to be naively, and thus in theprovince of physics rather than of psychology.

The problem with the direct realist view is of an episte-mological nature, and is therefore a more fundamental ob-jection; for direct realism, as defended by Gibson (1979), isnothing short of magical – that we can see the world out be-yond the sensory surface. The projection theory has a sim-ilar epistemological problem and is equally magical andmysterious, suggesting as it does that neural processes inour brain are somehow also out in the world. Both of theseparadigms have difficulty with the phenomena of dreamsand hallucinations (Revonsuo 1995), which present thesame kind of phenomenal experience as spatial vision, ex-cept independent of the external world in which that per-ception is supposed to occur in normal vision. It is the im-plicit or explicit acceptance of this naive concept ofperception which has led many to conclude that conscious-ness is deeply mysterious and forever beyond human com-prehension. For example, Searle (1992, p. 96) contendedthat consciousness is impossible to observe, for when we at-tempt to observe consciousness we see nothing but what-ever it is that we are conscious of; there is no distinction be-tween the observation and the thing observed.

On the other hand, the problem with the indirect realistview is more of a technological or computational limitation,for we cannot imagine how contemporary concepts of neu-rocomputation, or even of artificial computation for thatmatter, can account for the properties of perception as ob-served in visual consciousness. It is clear, however, that themost fundamental principles of neural computation andrepresentation remain to be discovered, and therefore wecannot allow our currently limited notions of neurocompu-tation to constrain our observations of the nature of visualconsciousness. The phenomena of dreams and hallucina-tions clearly demonstrate that the brain is capable of gen-erating vivid spatial percepts of a surrounding world inde-pendent of that external world, and that capacity must be aproperty of the physical mechanism of the brain. Normalconscious perception can therefore be characterized as aguided hallucination (Revonsuo 1995), which is as much amatter of active construction as it is of passive detection. Ifwe accept the truth of indirect realism, this immediatelydisposes of at least one mysterious or miraculous compo-nent of consciousness, which is its unobservability. Con-sciousness is indeed observable, contrary to Searle’s con-tention, because the objects of experience are first andforemost the product or “output” of consciousness, andonly in secondary fashion are they also representative of ob-jects in the external world. Searle’s (1992) difficulty in ob-serving consciousness is analogous to saying that you can-not see the moving patterns of glowing phosphor on yourtelevision screen, all you see is the ball game that is show-ing on that screen. The indirect realist view of television isthat what you are seeing is first and foremost glowing phos-phor patterns on a glass screen, and only in secondary fash-ion are those moving images also representative of the re-mote ball game.

The choice therefore is between accepting a magicalmysterious account of perception and consciousness thatseems impossible in principle to implement in any artificialvision system, or facing the seemingly incredible truth thatthe world we perceive around us is indeed an internal datastructure within our physical brain (Lehar 2003b). Theprincipal focus of neurophysiology should now be to iden-tify the operational principles behind the three-dimen-sional volumetric imaging mechanism in the brain, themechanism responsible for generating the solid stableworld of visual experience that we observe to surround usin conscious experience.

3. Problems in modeling perception

The computational modeling of perceptual processes is aformidable undertaking. But the problem is exacerbated bythe fact that a neural network model of perception attemptsto model two entities simultaneously: the subjective expe-rience of perception and the neurophysiological mecha-nism by which that experience is generated in the brain.The chief problem with this approach is that our knowledgeof neurophysiological principles is known to be incomplete.We do not understand the computational functionality ofeven the simplest neural systems. For example, the lowlyhouse fly, with its tiny pinpoint of a brain, seems to thumbits nose at our lofty algorithms and complex computationalmodels as it dodges effortlessly between the tangledbranches of a shrub in dappled sunlight, compensating for




gusty cross-winds to avoid colliding with the branches. Thisremarkable performance by this lowly creature far exceedsthe performance of our most powerful computer algo-rithms and our most sophisticated neural network modelsof human perception. In fact, the “dirty little secret” of neu-roscience, as Searle (1997, p. 198) called it, is that we haveno idea what the right level of analysis of the brain shouldbe because there is no universally accepted theory of howthe brain actually codes perceptual or experiential infor-mation. The epistemological question highlights this un-certainty, for it shows that there is not much consensus onwhether the world of conscious experience is even explic-itly represented in the brain at all, the majority view being,apparently, that it is not. Palmer (1999) went even further,saying that “to this writer’s knowledge, no one has ever sug-gested any theory that the scientific community regards asgiving even a remotely plausible causal account of how ex-perience arises from neural events.” Without this key pieceof knowledge, how can we even begin to model the com-putational processes of perception in neurophysiologicalterms?

One approach is to begin with the neurophysiology of thebrain and attempt to discover what it is computing at the lo-cal level of the individual neuron, the elemental buildingblock of the nervous system. The fruit of this branch of in-vestigation is neural network theory. But it is unclearwhether neural network theory offers an adequate charac-terization of the actual processing going on in the brain, orwhether it is asking too much of simple integrate-and-fireelements, no matter how cleverly connected in patterns ofsynaptic connections, to provide anything like an adequateaccount of the observed properties of conscious experience.Churchland (1984) argued in the affirmative, that we dohave enough knowledge of the principles of neurocompu-tation to begin to propose realistic models of perceptualprocessing. Palmer (1992) and Opie (1999) presented dy-namic neural network models of Gestalt phenomena, suchas the perceptual grouping of triangles, showing how thedynamics of perceptual phenomena can be modeled by adynamic neural network model. But those models are pro-posed in the abstract, presenting general principles ratherthan complete and detailed models of specific perceptualphenomena expressed as sense-data. For example, Palmer(1992) discussed the perceptual experience of an equilat-eral triangle, perceived as an arrow pointing in one of threedirections. Palmer modeled this perceptual phenomenonas a competition between three dynamic neural networknodes in a mutually inhibitory relationship, resulting in a“winner-take-all” behavior. Although this model is com-pelling as a demonstration of Gestalt principles in a neuralnetwork model, Palmer left out the most difficult part of theproblem, which is not just the competition between threealternative percepts but the perceptual representation ofthe percept itself. The perceptual experience of a trianglecannot be reduced to just three phenomenal values but isobserved as a fully reified triangular structure that spans aspecific portion of perceived space. This sense-data com-ponent of the phenomenal experience is very much moredifficult to account for in neural network terms.

In recent decades a number of attempts have been madeto quantify the sense-data of visual consciousness in com-putational models (see Lesher 1995, for a review). Zuckeret al. (1988) presented a model of curve completion that ac-counts for the emergent nature of perceptual processing by

incorporating a feedback loop in which local feature detec-tors tuned to detect oriented edges feed up to global cur-vature detector cells, and those cells in turn feed back downto the local edge level to fill in missing pieces of the globalcurve. A similar bottom-up/top-down feedback is given inGrossberg and Mingolla’s (1985) visual model to accountfor boundary completion in illusory figures like the Kanizsasquare by generating an explicit line of neural activationalong the illusory contour. An extension of that model(Grossberg & Todoroviçz 1988) accounted for the filling-inof the surface brightness percept in the Kanizsa figure, withan explicit diffusion of neural activation within the regionof the illusory surface. These models have had a significantimpact on the discussion of the nature of visual illusions be-cause they highlight the fact that illusory features, like theillusory surface of a Kanizsa figure, are observed as ex-tended image-like data structures, and therefore a com-plete model of the phenomenon must also produce a fullyreified image-like spatial structure as its output. In fact,Grossberg’s concept of visual reification in his BoundaryContour System (Grossberg & Mingolla 1985) and FeatureContour System (Grossberg & Todoroviç 1988) were theoriginal inspiration behind the perceptual modeling pro-posed in the present hypothesis.

Although these models finally offer a reasonable accountof perceptual experience (in two dimensions), they alsodemonstrate the profound limitations of a neural networkarchitecture for perceptual representation because neuralnetwork theory is no different in principle than a templatetheory (Lehar 2003a), a concept whose limitations are wellknown. Grossberg and Mingolla (1985) account for col-linear illusory contour completion by way of specializedelongated receptive fields, tuned to detect and enhancecollinearity. This concept works well enough for simplecollinear boundary completion (as long as it remains re-stricted to two dimensions), but any attempt to extend thismodel to higher order perceptual processing runs headlonginto a combinatorial explosion in required receptive fields(Lehar 2003a). For example, perceptual completion is ob-served not only for collinear alignments but it can also de-fine illusory vertices composed of two, three, or more edgesthat meet at a vertex (Lehar 2003a). Grossberg himself pro-posed an extension to his model equipped with “corner de-tector” receptive fields (Grossberg & Mingolla 1985), al-though this line of thought was subsequently quietlyabandoned because, just as with the cells that performcollinear completion, the corner detectors would have to beprovided at every location and every orientation across thevisual field. To extend the model to account for T, V, Y, andX intersections, specialized receptive fields would have tobe provided for each of those features at every location andat every orientation across the visual field. This combinato-rial explosion in the required number of specialized recep-tive fields does not bode well for neural network theory asa general principle of neurocomputation.

The most serious limitation of Grossberg’s approach toperception is that, curiously, Grossberg and his colleaguesdid not extend their logic to the issue of three-dimensionalspatial perception. In going from two dimensions to three,Grossberg no longer advocated explicit spatial filling-in, butinstead represented the depth dimension by binocular dis-parity, using left and right eye image pairs (Grossberg 1987;1990; 1994). Although a stereo pair does encode depth in-formation, it does not do so in a volumetric manner because




it can only encode one depth or disparity value for every(x,y) point on the image. This makes it impossible for Gross-berg’s model to represent transparency with multiple depthvalues at a single (x,y) location, or to represent the experi-ence of empty space between the observer and a visible object. Moreover, it precludes the kind of volumetric fill-ing-in required to account, for example, for the three-di-mensional version of the Ehrenstein illusion constructed ofa set of rods arranged radially around a circular void (Ware& Kennedy 1978). The filling-in processes in this illusiontake place through the depth dimension, which producesan illusory percept of a glowing disk, hanging in space, as avolumetric spatial structure. If Grossberg’s argument forexplicit filling-in of the two-dimensional illusions is at allvalid, then that argument should apply equally to volumet-ric filling-in also.

The reason Grossberg declined to extend his model intothe third dimension is neurophysiologically motivated. Foralthough Grossberg’s model is a de facto perceptual model,it is actually presented as a neural network model; that is,the computational units of the model represent actual neu-rons in the brain rather than perceptual entities. And thishighlights the problem of perceptual modeling in neuralnetwork terms, for whenever there is a conflict between theperceptual phenomenon and our current understanding ofneurophysiological principles, there is then a conflict be-tween the neural and the perceptual models of the phe-nomenon. In this case the percept is clearly volumetric, butthe corresponding cortical neurophysiology is assumed tobe two-dimensional. Another reason Grossberg was reluc-tant to extend his model into the third dimension is that,even for simple collinear completion, such an approachwould require a volumetric block of neural elements eachequipped with elongated receptive fields; and those fieldsmust be replicated at every orientation in three dimensionsand at every volumetric location across the entire volume ofthe perceptual representation – a notion that seems too im-plausible to contemplate, let alone the idea of T, V, Y, and Xintersections defined in three dimensions. But until a map-ping has been established between the conscious experi-ence and the corresponding neurophysiological state, thereis no way to verify whether the model has correctly repli-cated the psychophysical data. Because these models strad-dle the mind/brain barrier, they run headlong into the issuethat Chalmers (1995) dubbed the “hard problem” of con-sciousness. Simply stated, even if we were to discover theexact neurophysiological correlates of conscious experi-ence, there would always remain a final explanatory gap be-tween the physiological and the phenomenal levels of de-scription. For example, if the activation of a particular cellin the brain were found to correlate with the experience ofred at some point in the visual field, there would remain avivid subjective quality, or quale, to the experience of redthat is not in any way identical to any externally observablephysical variable such as the electrical activity of a cell. Inother words, there is a subjective experiential componentof perception that can never be captured in a model ex-pressed in objective neurophysiological terms.

Even more problematic for neural models of perceptionis the question of whether perceptual information is ex-pressed neurophysiologically in explicit or implicit form.For example, Dennett (1992) argued that the perceptualexperience of a filled-in colored surface is encoded in moreabstracted form in the brain, in the manner of an edge im-

age that records only the transitions along image edges.Support for this concept is seen in the retinal ganglion cellsthat respond only along spatial or temporal discontinuitiesin the retinal image and produce no response within regionsof uniform color or brightness. This concept also appears tomake sense from an information-theoretic standpoint, foruniform regions of color represent redundant informationthat can be compressed to a single value, as is the practicein image compression algorithms. These kinds of theoreti-cal difficulties have led many neuroscientists to simply ig-nore the conscious experience and to focus instead on thehard evidence of the neurophysiological properties of thebrain.

4. A perceptual modeling approach

The quantification of conscious experience is not quite ashopeless as it might seem. Nagel (1974) suggested that weset aside temporarily the relation between mind and brainand devise a new method of objective phenomenology – inother words, quantify the structural features of the subjec-tive experience in objective terms without committing toany particular neurophysiological theory of perceptual rep-resentation. For example, if we quantify the experience ofvision as a three-dimensional data structure, like a model ofvolumes and surfaces in a surrounding space to a certainperceptual resolution, this description could be meaningfuleven to a congenitally blind person or to an alien creaturewho had never personally experienced the phenomenon ofhuman vision. Although this description could never cap-ture everything of that experience, such as the qualia ofcolor experience, it would at least capture the structuralcharacteristics of that subjective experience in an objectiveform that would be comprehensible to beings incapable ofhaving those experiences.

Chalmers (1995) extended this line of reasoning with theobservation that the subjective experience and its corre-sponding neurophysiological state carry the same informa-tion content. On that ground, Chalmers proposed a princi-ple of structural coherence between the structure ofphenomenal experience and the structure of objectively re-portable awareness, to reflect the central fact that con-sciousness and physiology do not float free of one anotherbut cohere in an intimate way. In essence this is a restate-ment of the Gestalt principle of isomorphism, of whichmore in section 5. The connecting link between mind andbrain therefore is information in information-theoreticterms (Shannon 1948) because the concept of informationis defined at a sufficiently high level of abstraction to be in-dependent of any particular physical realization, and yet itis specified sufficiently to be measurable in any physical sys-tem given that the coding scheme is known. A similar ar-gument was made by Clark (1993, p. 50). Chalmers mod-erated his claim of the principle of structural coherence bystating that it is a hypothesis that is “extremely speculative.”However, the principle is actually solidly grounded episte-mologically because the alternative is untenable. If we ac-cept the fact that the physical states of the brain correlatedirectly with conscious experience, then the claim that con-scious experience contains more explicit information thandoes the physiological state on which it was based amountsto a kind of dualism that would necessarily involve somekind of nonphysical “mind stuff” to encode the excess in-




formation observed in experience that is not encoded by thephysical state. Some theorists have even proposed a kind ofhidden dimension of physical reality to house the unac-counted information in conscious experience (Harrison1989; Smythies 1994).

The philosophical problems inherent in neural networkmodels of perceptual experience can be avoided by propos-ing a perceptual modeling approach (Lehar 2003b), whichmodels the conscious experience directly in the subjectivevariables of perceived color, shape, and motion, as opposedto neural modeling, where the conscious experience is mod-eled in the neurophysiological variables of neural activa-tions or spiking frequencies, or the like. The variables en-coded in the perceptual model therefore correspond towhat philosophers call the sense-data or primitives of rawconscious experience, except that these variables are notsupposed to be the sense-data themselves, they merely rep-resent the value or magnitude of the sense-data they are de-fined to represent. In essence this amounts to modeling theinformation content of subjective experience, which is thequantity that is common between mind and brain, thus al-lowing an objectively quantified description of a subjectiveexperience. In fact, this approach is exactly the concept be-hind the description of phenomenal color space in the di-mensions of hue, intensity, and saturation, as seen in theCIE (Commission Internationale L’Eclairage) chromaticityspace. The geometrical dimensions of that space have beentailored to match the properties of the subjective experi-ence of color as measured psychophysically, expressed interms that are agnostic to any particular neurophysiologicaltheory of color representation.

Clark (1993) presented a systematic description of othersensory qualities in quantitative terms, based on this sameconcept of “objective phenomenology.” The thorny issue ofthe hard problem of consciousness is thus neatly side-stepped because the perceptual model remains safely onthe subjective side of the mind/brain barrier, and thereforethe variables expressed in the model refer explicitly to sub-jective qualia rather than to neurophysiological states of thebrain. The problems of explicit versus implicit representa-tion are also neatly circumvented because those issues per-tain to the relation between mind and brain and so do notapply to a model that does not straddle the mind/brain bar-rier. For example, the subjective experience of a Neckercube is of a solid three-dimensional structure, and for thatreason the perceptual model of that experience should alsobe an explicit three-dimensional structure. The sponta-neous reversals of the Necker cube, on the other hand, areexperienced as a dynamic process, and on that groundshould be represented in the perceptual model as a dy-namic process – that is, as a literal reversal of the solidthree-dimensional structure. The issues of whether a per-ceived structure can be encoded neurophysiologically as aprocess or whether a perceived process can be encoded asa structure are therefore irrelevant to the perceptualmodel, which by definition models a perceived structure asa structure, and a perceived process as a process.

This is of course only an interim solution, for eventuallythe neurophysiological basis of conscious experience mustalso be identified; nevertheless, the perceptual model doesoffer objective information about the informational contentencoded in the physical mechanism of the brain. This is anecessary prerequisite to a search for the neurophysiologi-cal basis of conscious experience, for we must clearly cir-

cumscribe that which we are to explain before we can at-tempt an explanation of it. This approach has served psy-chology well in the past, particularly in the field of colorperception where the quantification of the dimensions ofcolor experience led directly to great advances in our un-derstanding of the neurophysiology of color vision. The fail-ure to quantify the dimensions of spatial experience hasbeen responsible for decades of futile debate about its neu-rophysiological correlates. I will show that application ofthis perceptual modeling approach to the realm of spatialvision opens a wide chasm between phenomenology andcontemporary concepts of neurocomputation and therebyoffers a valuable check on theories of perception basedprincipally on neurophysiological concepts.

5. The Gestalt principle of isomorphism

The Gestalt principle of isomorphism represents a subtlebut significant extension to Müller’s psychophysical postu-late and to Chalmers’s principle of structural coherence. Inthe case of structured experience, equal dimensionality be-tween the subjective experience and its neurophysiologicalcorrelate implies similarity of structure or form. For exam-ple, the percept of a filled-in colored surface, whether realor illusory, encodes a separate and distinct experience ofcolor at every distinct spatial location within that surface toa particular resolution. Each point of that surface is not ex-perienced in isolation but in its proper spatial relation toevery other point in the perceived surface. In other words,the experience is extended in at least two dimensions, andtherefore the neurophysiological correlate of that experi-ence must also encode at least two dimensions of percep-tual information. The mapping of phenomenal color spacewas established by the method of multidimensional scaling(Coren et al. 1994, p. 57) in which color values are orderedin psychophysical studies on the basis of their perceivedsimilarity, to determine which colors are judged to be near-est to each other or which colors are judged to be betweenwhich other colors in phenomenal color space. A similarprocedure could just as well be applied to spatial percep-tion to determine the mapping of phenomenal space. If twopoints in a perceived surface are judged psychophysically tobe nearer to each other when they are actually nearer andfarther when they are actually farther, and if other spatialrelations such as betweenness are also preserved phenom-enally, then direct evidence is thereby provided that phe-nomenal space is mapped in a spatial representation thatpreserves those spatial relations in the stimulus. The out-come of this proposed experiment is so obvious it needhardly be performed. And yet its implication – that our phe-nomenal representation of space is spatially mapped – isnot often considered in contemporary theories of spatialrepresentation.

5.1. Structural versus functional isomorphism

The isomorphism required by Gestalt theory is not a strictstructural isomorphism, a literal isomorphism in the phys-ical structure of the representation, but rather, it is merelya functional isomorphism, a behavior of the system as if itwere physically isomorphic (Köhler 1969, p. 92). This is be-cause the exact geometrical configuration of perceptualstorage in the brain cannot be observed phenomenologi-




cally any more than the configuration of silicon chips on amemory card can be determined by software examinationof the data stored within those chips. Nevertheless, themapping between the stored perceptual image and the cor-responding spatial percept must be preserved, as in the caseof the digital image, so that every stored color value ismeaningfully related to its rightful place in the spatial per-cept.

The distinction between structural and functional iso-morphism can be clarified with a specific example. Con-sider the spatial percept of a block resting on a surface, de-picted schematically in Figure 1A. The information contentof this perceptual experience can be captured in a paintedcardboard model built explicitly like Figure 1A, with ex-plicit volumes, bounded by colored surfaces, embedded ina spatial void. Because perceptual resolution is finite, themodel should also be considered only to a finite resolution;that is, the infinite subdivision of the continuous space ofthe actual model world is not considered to be part of themodel, which can only validly represent subdivision ofspace to the resolution limit of perception. The same per-ceptual information can also be captured in quantized ordigital form in a volumetric or voxel (volume-pixel) imagein which each voxel represents a finite volume of the cor-responding perceptual experience, as long as the resolutionof this representation matches the spatial resolution of thepercept itself; in other words, the size of the voxels shouldmatch the smallest perceivable feature in the correspond-ing spatial percept. Both the painted cardboard model andits quantized voxel equivalent are structurally or topo-graphically isomorphic with the corresponding percept;they have the same information content as the spatial per-cept that they represent.

Consider now the flattened representation depicted inFigure 1B, which is identical to the model in Figure 1A ex-cept that the depth dimension is compressed relative to theother two dimensions, like a bas-relief. If the defined scaleof the model (the length in the representation relative tothe length that it represents) is also correspondingly com-pressed, as suggested by the compressed gridlines in thefigure, then this model is also isomorphic with the percep-tual experience of Figure 1A. In other words the flatteningof the depth dimension is not really registered in the modelbecause the perceived cube spans the same number of grid-lines in Figure 1B (in all three dimensions) as it does in Fig-ure 1A, and therefore this flattened model encodes a non-flattened perceptual experience. Though this model is nowno longer structurally isomorphic with the original percep-tual experience, it does remain topologically isomorphic,preserving neighborhood relations, as well as betweenness,and so forth. In a mathematical system with infinite resolu-tion, this model would encode the same information as theone in Figure 1A. However in a real physical representationthere is always some limit to the resolution of the system,or how much information can be stored in each unit dis-tance in the model itself. In a representational system withfinite resolution, therefore, the depth information in Fig-ure 1B would necessarily be encoded at a lower resolutionthan that in the other two dimensions. If our own percep-tual apparatus employed this kind of representation, thisflattening would not be experienced directly; the only man-ifestation of the flattening of the representation would be areduction in the resolution of perceived depth relative tothe other two dimensions, making it more difficult to dis-

tinguish differences of perceived depth than differences ofperceived height and width.

Consider now the warped model depicted in Figure 1C,which is like the flattened model of Figure 1B with a wavydistortion applied, as if warped like the gyri and sulci of thecortical surface. This warped representation is also isomor-phic with the perceptual experience it represents for it en-codes the same information content as the flattened spacein Figure 1B, although again this is a topological rather thana topographical isomorphism. The warping of this spacewould not be apparent to the percipient because the verydefinition of straightness is warped along with the space it-self, as suggested by the warped gridlines in the figure. Incontrast, consider the flattened representation depicted inFigure 1D, where the perceptual representation has beensegmented into discrete depth planes that distinguish onlyforeground from background objects. This model is nolonger isomorphic with the perceptual experience it sup-posedly represents because, unlike this model, the percep-tual experience manifests a specific and distinct depth valuefor every point in each of the surfaces of the percept. Fur-thermore, the perceptual experience manifests an experi-ence of empty space surrounding the perceived objects,every point of which is experienced simultaneously and inparallel as a volumetric continuum of a certain spatial res-olution, whereas the model depicted in Figure 1D encodesonly a small number of discrete depth planes. This kind ofmodel therefore is inadequate as a perceptual model of theinformation content of conscious experience because the


384 BEHAVIORAL AND BRAIN SCIENCES (2003) 26:4

Figure 1. A. A volumetric spatial model, for example built ofpainted cardboard surfaces, is structurally isomorphic with a per-ceptual experience of a block resting on a surface if it has the sameinformation content. B. If the model is compressed in one di-mension relative to the other two, the model can still be isomor-phic with the original percept if the representational scale of themodel (indicated by the shaded gridlines) is also correspondinglycompressed, although this is no longer a structural isomorphismbut merely a topological isomorphism. C. The model can even bewarped like the gyri and sulci of the cortical surface and remainisomorphic with the original percept. D. But a model composedof a small number of discrete depth planes is not isomorphic withthe original percept because it no longer encodes the same infor-mation content.

https://doi.org/10.1017/S0140525X03410091Downloaded from https:/www.cambridge.org/core. Dartmouth College, on 08 May 2017 at 15:51:22, subject to the Cambridge Core terms of use, available at https:/www.cambridge.org/core/terms.


dimensions of its representation are less than the dimen-sions of the experience it attempts to model.

A functional isomorphism must also preserve the func-tional transformations observed in perception, and the ex-act requirements for a functional isomorphism depend onthe functionality in question. For example, when a coloredsurface is perceived to translate coherently across per-ceived space, the corresponding color values in the per-ceptual representation of that surface must also translatecoherently through the perceptual map. If that memory isdiscontinuous, like a digital image distributed across sepa-rate memory chips on a printed circuit board, then the per-ceptual representation of that moving surface must jumpseamlessly across those discontinuities in order to accountfor the subjective experience of a continuous translationacross the visual field. In other words, a functional isomor-phism requires a functional connectivity in the representa-tion, as if a structurally isomorphic memory were warped,distorted, or fragmented, but at the same time, the func-tional connectivity between its component parts were pre-served. Consider a representational mechanism, such asthat shown in Figure 1A, equipped with additional compu-tational hardware capable of performing spatial transfor-mations on the volumetric image in the representation. Therepresentational mechanism might be equipped with func-tions that could rotate, translate, and scale the spatial pat-tern in the representation on demand. This representationwould thereby be invariant to rotation, translation, andscale, because the spatial pattern of the block itself wouldbe encoded independent of its rotation, translation, andscale. The fact that an object in perception maintains itsstructural integrity and recognized identity despite rota-tion, translation, and scaling by perspective is clear evi-dence for this kind of invariance in human perception andrecognition. If the warped model shown in Figure 1C wereequipped with these same transformational functions, thewarped representation would also be functionally isomor-phic with the non-warped representation as long as thosetransformations were performed correctly with respect tothe warped geometry of that space.

A functional isomorphism is even possible for a repre-sentation that is fragmented into separate pieces, if thosepieces are wired together in such a way that they continueto perform the spatial transformations exactly as in the cor-responding undistorted mechanism. A functional isomor-phism can even survive in a volumetric representationwhose individual elements or voxels are scrambled ran-domly across space, if the functional connections betweenthose elements are preserved through the scrambling. Theresult is a representation that is neither topographically nortopologically isomorphic with the perceptual experience itrepresents. However, it does remains a volumetric repre-sentation, with an explicit encoding of each point in the rep-resented space to a particular spatial resolution, and it re-mains functionally isomorphic with the spatial experiencethat it represents, capable of performing coherent rotation,translation, and scaling transformations of the perceptualstructures expressed in the representation.

An explicit volumetric spatial representation capable ofspatial transformation functions, as described above, ismore efficiently implemented in either a topographicallyisomorphic form or a topologically isomorphic form, whichrequire shorter and more orderly connections between ad-jacent elements in the representation. However, the argu-

ment for structural or topological isomorphism is an argu-ment of representational efficiency and simplicity, ratherthan of logical necessity. On the other hand, a functionalisomorphism is strictly required in order to account for theproperties of the perceptual world as observed subjec-tively. The volumetric structure of visual consciousnessand perceptual invariance to rotation, translation, andscale offer direct and concrete evidence for an explicit vol-umetric spatial representation in the brain, which is at leastfunctionally isomorphic with the corresponding spatial ex-perience.

A neurophysiological model of perceptual processingand representation should concern itself with the actualmechanism in the brain. In the case of a distorted repre-sentation (as in Fig. 1C), the warping of that perceptualmap would be a significant feature of the model. A percep-tual model, on the other hand, is concerned with the struc-ture of the percept itself, independent of any warping of therepresentational manifold. Even for a representation that isfunctionally but not structurally isomorphic, a descriptionof the functional transformations performed in that repre-sentation is most simply expressed in a structurally isomor-phic form, just as a panning or scrolling function in imagedata is most simply expressed as a spatial shifting of imagedata even when that shifting is actually performed in hard-ware in a non-isomorphic memory array. For that reason,the functional operation of a warped mechanism like Fig-ure 1C is most simply described as the operation of thefunctionally equivalent undistorted mechanism in Figure1A. In the present discussion, therefore, our concern willbe chiefly with the functional architecture of perception, adescription of the spatial transformations observed in per-ception, whatever form those transformations might take inthe physical brain. And those transformations are most sim-ply described as if taking place in a physically isomorphicspace.

In the discussion that follows, the terminology “spatialrepresentation,” “data expressed in spatial form,” “literalvolumetric replica of the world inside your head,” “three-dimensional pattern of opaque-state units,” “explicit three-dimensional replica of the surface,” and “volumetric spatialmedium,” will refer not to a topographically isomorphicmodel of space, as suggested in Figure 1A, but to a func-tionally isomorphic model of space like the warped modelin Figure 1C, in which the explicit volumetric representa-tion is possibly warped and distorted but still encodes an ex-plicit value for every volumetric point in perceived space aswell as the neighborhood relations between those values.This is in contrast to the more commonly assumed flattenedor abstracted cortical representation depicted in Figure1D, where the volumetric mapping is no longer preserved.

5.2. Second-order , complementary , and otherparamorphisms

The issue of isomorphism is so profoundly problematic fortheories of perceptual representation that theorists havegone to no end of trouble in an effort to dispel the issue andto argue that isomorphism is not actually necessary. A care-ful examination of these proposals, however, reveals thenaïve realist assumptions on which they are founded.

Shepard and Chipman (1970) argued that when we per-ceive a square, for example, there is no need for an internalperceptual replica of that square in the brain of the percip-




ient. They argued that we learn the appropriate use ofwords such as “square” from a verbal community that hasaccess only to the public object and not to any such privateimage. If there is some internal event that corresponds toour experience of a square, whether it is the activation of acell or cell assembly in the brain, our ability to form an as-sociation between this event and the word “square” re-quires only that this event have a regular relation to the ex-ternal object of causality, not of structural isomorphism. Toinsist, additionally, that these neurons must be spatiallyarranged in precisely the form of a square does not in theleast help to explain how they come to trigger the namingresponse “square,” at least according to Shepard and Chip-man.

As can be discerned from their brief introductory para-graph summarized above, Shepard and Chipman neatlyturned the tables on the debate by characterizing the per-ception of a square as the issue of learning the naming re-sponse “square,” which is an issue of recognition ratherthan of perception. To be sure, recognition is an importantaspect of perception, and the problem of learning a namingresponse is a formidable one that deserves further investi-gation. But the recognition response is by no means thesame thing as the perceptual experience of the square as acontinuous filled-in, square-shaped region of sense-data ex-perienced in the visual field. How can so intelligent and ed-ucated researchers come to make such a profound error inidentification of the issue at hand? The answer is clear fromtheir assertion that a verbal community has access only tothe public object and not to any private image. This naïverealist assumption is passed off casually as a statement offact, but in fact it reveals an implicit commitment to the no-tion that the three-dimensional volumetric objects that weobserve to occupy the space of our perceptual field are theactual objects themselves, and that therefore they need notbe replicated or re-represented again in the brain. The factthat this assumption has gone unchallenged, and evenlargely unnoticed by the community at large, demonstrateshow deeply the assumptions of naïve realism have becomeentrenched in contemporary thought.

Shepard (1981) made another attempt to dispel the issueof isomorphism by arguing for psychophysical complemen-tarity rather than isomorphism. Appropriately enough,Shepard cited that grand master of naïve realism, B. F.Skinner, who argued that even if we were to discover a partof the brain in which the physical pattern of neural activityhad the very same shape as the corresponding external ob-ject – say, a square – we would not in this way have madeany progress toward explaining how the subject is able torecognize that object as a square, or to learn to associate toit a unique verbal response “square.” So again the issue ofperception is confounded with the issue of recognition re-sponse. Skinner’s statement is true enough, as far as it goes.But what Shepard and Skinner failed to acknowledge is thatit would be very much harder to learn to recognize a squareif you could not “see” it, that is, if you did not have directaccess to an internal representation of the square as asquare-shaped sense-datum to associate with the appropri-ate recognition response. To claim that we can experiencethe square without such an internal replica is just plainmagic. Furthermore, until we do discover a part of the brainin which the physical pattern of neural activity (or someother physically measurable quantity) has the very sameshape as the corresponding external object, the phenome-

nal aspect of that volumetric spatial structure remains as anomological dangler, something that is experienced as aspatial picture, something that is clearly distinct from theactual square in the real world (especially when that squareis illusory), but something that does not actually exist in anyspace known to science. Like the Behaviorists before him,Shepard attempted to discount the entire edifice of con-scious experience as if it simply did not exist as a scientificentity.

There is a further difficulty with the notion of psy-chophysical complementarity. Shepard (1981) argued thatthe relation of the mental representation to the external ob-ject it represents might be one of complementarity, ratherthan one of similarity or resemblance. Just as a lock has ahidden structure that is to some extent complementary tothe visible contour of the key that fits it, the internal struc-ture uniquely activated by a given object must have a struc-ture that somehow meshes with the pattern manifested byits object; in other words, the “shape” of the representationis complementary to, rather than isomorphic with, the ob-ject that it represents. But, again, this notion of perceptualrepresentation is only coherent from a naïve realist per-spective. If we interpret this argument from an indirect per-ceptual view, it would have to be that the square shape weexperience in immediate consciousness is complementaryto the external square, which is beyond our direct experi-ence. In other words, the real “square” in the external worldis not actually square as we observe it to be, but rather itwould have to be somehow complementary to the squareshape we observe in conscious experience, an idea that isobviously absurd.

In yet another, somewhat different, defense of naïve re-alism, Shepard (1981, p. 292) argued that the relation be-tween the external object and its internal representationmight be a kind of paramorphism rather than isomorphism,as seen for example in the Fourier transform of an image,which encodes all of the information in a spatial image butin a very abstract nonspatial form. Again, this argument is founded on the naïve assumption that the world we see around us is the world itself, and that therefore theparamorphic representation of that world is not identifiedas the image of the world we see around us but as our ver-bal or conceptual recognition of that world. If the percep-tual brain did indeed employ a Fourier representation in-stead of a spatial one, then the world we see around uswould necessarily appear in the form of a Fourier transformrather than as a spatial structure, which, again, is obviouslyabsurd. The fact that the world around us appears as a vol-umetric spatial structure is direct and concrete evidence fora spatial representation in the brain. What is most interest-ing about this issue is that Shepard clearly did not fully com-prehend the position that he challenged, and therefore hiscriticisms of isomorphism inevitably missed the mark.

Steven Palmer (1999) on the other hand struck at thevery heart of the issue of isomorphism. Palmer drew a dis-tinction between two different aspects of conscious experi-ence, the intrinsic qualities of experiences themselves ver-sus the relational structure that holds among thoseexperiences. The intrinsic qualities, such as the color qualiain the experience of color, are in principle impossible tocommunicate from one mind to another, and therefore theyare inaccessible to science (except through phenomenol-ogy), a restriction that Palmer calls the subjectivity barrier.All that can be communicated about conscious experience




is the relational structure that holds among those experi-ences. In the case of color experience, for example, subjectssay that orange is more similar to red than it is to green orblue, and that aqua is experienced as intermediate betweengreen and blue, and so forth. It was exactly these relationalfacts of color experience that were used to define the colorsolid in the CIE chromaticity diagram. A relational struc-ture like the color solid encodes a great deal of informationimplicitly about the relations between its variables in a man-ner that is practically impossible to express as explicit rela-tions because the number of binary, trinary, and other rela-tions between colors implicitly expressed in the color solidis so astronomical as to defy any kind of exhaustive listing or discrete associative links. And yet all of those relationsare evidently available to the psychophysical subject whenmaking phenomenal color judgments. This

gestalt isomorphism and the primacy of subjective conscious experience: a gestalt...

Documents