The gaze selects informative details withinpictures 1

NORMAN H. MACKWORTH2 AND ANTHONY J. MORAND13HARVARD UNIVERSITY

The visual fixations of 20 Ss viewing each of two pictureswere measured. Each picture was later divided into 64squares, and 20 other Ss judged their recognizability on alO-point scale. Both measures gave high readings for unusualdetails and for unpredictable contours. Although they werejudged to be highly recognizable, all the redundant (or pre­dictable) contours received few fixations. Areas of meretexture scored low on both measures. The relations betweenfixation densities and estimated recognizability suggest thata scene may be divided into informative features and redun­dant regions. Not only do the eyes have to be aimed, theyare usually aimed intelligently, even during the casualinspection of pictures.

The main experimental question is: Do adults lookat dominant details in the stimulus with foveal vision?If, as an alternative, adults learned to perceivewhole patterns, it would presumably not matterwhere they looked on the picture. There would thennot be much correlation between the particularareas selected by different individuals.

Perception is no passive sampling from externalevents. It is so closely interlocked with memorythat Hake (1957) has claimed that there is no per­ception without recognition. Bartlett's work (1932)has also indicated that remembering is based onreconstructing from mental schemata rather thanfrom playing back unmodified recordings. Such aselective process implies that the organism activelycontrols the stimuli to which it is sensitive byusing the mismatch between input and plan (Miller,Galanter, & Pribram, 1960). These control functionsdepend upon the establishing of a program by priorlearning, because without such a plan there can beno basis for selectivity toward stimuli (Bruner, 1965).

The empirical question is, therefore, whetherthese cortical effects alter the eye tracks acrosspictures. Ten years ago, Hake (1957) stated that" •.• Vision does systematically select parts of stim­ulation in perception" and that the parts selectedare those which lead to the greatest coherence.Those aspects are chosen which produce the greatestconsistency of stimulation-those which produce thegreatest resemblance between present and past stim­ulation and which lead to the most efficient predictionabout future stimulations." More recently, this ap­proach has been taken further by considering visualperception as a selective process distinguishingsignal from noise in the visual environment (Hakeet aI, 1966).

If adults have learned to pick out the importantdetails for analysis, they might be expected to agreeto some extent on which were the informative areas.Group records of fixation density show that thesame specific areas are usually selected by the lineof sight (Mackworth & Bruner, 1966). We mightexpect regions that are rated highly informative toreceive more fixations than others, in line withsenders' (1966) finding that rapidly changing instru­ment dials are fixated more frequently. Perhapscomparisons between different pictures would alsoreveal the general characteristics of these importantregions by a content analysis of those stimuli thatdominate the others in the scene.

ObjectivesThe main purpose of our paper is to determine

whether key regions exist within pictorial displays.Preliminary attempts are also made to explain Whysome stimuli are more important than others withinthe displays.

ProcedureThe approach was to make a direct comparison

between (I) the visual choices, shown by the positionof the gaze while the -sa were looking at an entirepicture, and (2) the verbal estimates of the relativeimportance of the different regions within the samepicture. The studies included the two pictures givenin Fig. 1. The first, called the "eyes" picture,shows a pair of eyes behind a crimson mask, witha bright background of yellow and orange. Thesecond, called the "map" picture, is an astronaut'sview of Baja California seen against a dark bluesea. (The pictures shown to the Ss did not containthe superimposed squares or Ds) ,

The visual scanning choices were determined byrecording the density of visual fixations or pausesmade by the line of sight while the picture was beingviewed by 20 Harvard or Radcliffe students. Theeye track of each S was photographed by a standeye-camera on a copy of the stimulus display (Mack­worth, 1967). By superposing the individual eyetracks from all Ss, we could calculate what percentageof the total number of fixations made by the groupof sa fell within each separate square inch of the8 x 8 in. picture that subtended a visual angle of160 at the selected viewing distance. Every S wasshown each of the two color photographs separatelyfor 10 sec. The task then assigned to them while

Perception & Psychophysics, 1967. Vol. 2 (11) Copyright 1967. Psyehonomie Press. Goleta. Cali], 547

5r---r------r------r-----,II>C~0

; s>< 0 4

U. l-

e -0~II>

i: :3s Q)

'0 ~Q)

Cl.Q)

2<JC eQ)

~ C<J ~

E C"(/)

C G;CQ) Cl.~

0Lower0-2.9

ing the density of eye fixations. These separatesquares were given to each S, who was asked torate the squares individually for "informativeness."Each square was to be placed upon one of the 10rungs of a scale which was drawn on an answersheet and ranged from "completely informative"to "completely uninformative"; these estimates werenumbered 0-9 on the scale. Informativeness wasdefined in terms of "recognizability"-in the senseof how easy a square would be to recognize onanother occasion. The Ss did not see the picture asa whole beforehand.

ResultsThe main result was that a few regions in each

picture dominated the data. These outstanding areas,which have been emphasized by dotted lines in Fig. I,gave high readings on both measures. Each dottedsquare received more then 2 percent of the visualfixation density, and it also fell above the midpointon the scale of readings for estimated informative­ness. (Completely random scanning would have allo­cated 1.6 percent of all fixations to each of the 64squares in the 8 by 8 matrtx.)

Figure 2 summarizes the general relationshipbetween the mean estimated informativeness for agiven square and the percentage of the total visualfixations that fell in that square, as calculated fromthe group scatterplots. Squares that were rated highin informativeness averaged many times the numberof fixations per square as those rated lower in in­formativeness.

An extreme instance of the tendency for informative

Medium Upper:3-5~ 6-8~

Mean Estimated Level of Informativeness

Per Square

Fig. 2. Relation between the infonnativeness ratings givensquares and their average incidence of visual fixations. Thesquares from each picture were grouped according to tbeir ratedinformativeness. Then the incidence of visual fixations for thesquares in each group were averaged to obtain the mean fixationincidence for that group of squares,

in the eye-camera apparatus was to decide whichpicture they preferred.

The verbal estimates of the relative importance ofthe different regions within each picture were ex­pressed as the average ratings or subjective scalevalues for each of the 64 squares comprising thesame picture. These ratings were obtained fromanother group of 20 Harvard or Radcliffe students;none of these Ss had taken part in the previous eye­camera research. Each picture was dissected intothe 64 1 in. squares used as a matrix for determin-

Fig. 1· Stimulus displays with group responses superimposedfor all areas verbally rated above average. Note that'all high densi­ties of fixations (enclosed by dotted lines) were on unique edgesor areas. Conversely, all low densities of fixations (marked bysolid D's) were on redundant contours. Disregar:led by directvision, these redundant contours were being processed by pert­pheral vision.

548 Perception & Psychophysics, 1967, VoL 2 (11)

j Table 1. Density of Fixations (Fixations per sq in.) Table 2. Informativeness Ratings (Mean Scale Readings)

On TexturesRedundant Non-redundant

On ContoursRedundant Non-redundant

On TexturesRedundant Non-redundant

On ContoursRedundant Non-redundant

Sea O.S

Mask 0.6

Land 1.4

Nil

Predictable UnpredictableCoast 0.9 Coast 2.8

Clouds 4.7Mask Edge 1.0 Eyes 6.9

Sea 0.8

Mask 1.3

Land 2.3

Nil

PredictableCoast 6.0

Mask Edge 5.0

UnpredictableCoast 5.9Clouds 7.2

!Oyes 8.4

areas to receive clusters of visual fixations wasnoted in a content analysis of the eyes picture inFig. 1. The squares actually presenting the eyeswere the only areas in the picture rated above 8 onthe informativeness scale. The same four squareseach had fixation readings between 5 and 8 percent.Mackworth and Bruner (1966) have noted that indi­vidual visual fixations show higher percentages thanare obtained from group scatterplot data, which areusually easier to obtain. But highly informativeareas can cluster the fixations on particular regionswithin a display so markedly that even the groupvisual concentrations may be as high as five timesthe 1.6 percent that represents a completely randomdistribution of fixations. As many as 60 percent ofall fixations fell on the 10 squares across the eyesor just above them; that is, two-thirds of all fixa­tions were crowded into one-tenth of the total area.This clustering had the other effect, that three­quarters of all the squares in the picture were soseldom fixated that their fixation densities fell ator below 2 percent per square inch. They are leftunmarked in Fig. 1. (A similar analysis follows forthe map display.)

Each fixation was scored for informativeness bybeing given a number according to the estimatedimportance of the square in which it fell (Mack­worth & Bruner, 1966). The sum of these readingswas determined for all the fixations within eachsuccessive 2 sec period. It was then found that thevalue for this sum did not increase in successiveperiods. Therefore, ·It must be assumed that theSa placed their fixations just as effectively duringthe first 2 sec as they did during the last 2 sec ofthe 10 sec exposure. There was rapid rejection ofunwanted areas at the very beginning of the periodof inspection. Attention started with inhibition, inthe sense of the processing and rejection of unwantedstimuli (Berlyne, 1965; Spinelli & Pribram, 1966).

This processing and rejection must have beenmediated by peripheral vision, because three-quartersof the area of the eyes picture and nearly two-thirdsof the map picture were scarcely fixated at all, andreceived only 2 percent or fewer fixations persquare inch (see unmarked regions of Fig. 1). Thisearly sifting of the input within the first 2 sec al­lowed no time for each different region within thedisplay to be examined by the fovea, which is only2 degrees wide. The display arrangements were

Perception & Psychophysics, 1967, Vol. 2 (11)

such that this would have entailed machinelike scan­ning of each of the 64 squares in turn. This im­probable process would have called for at least 20sec rather than the 2 sec actually required; thereason the hypothetical 64 fixations would need somuch time is that pictures are usually inspectedat a speed of three fixations per second (Mackworth& Bruner, 1966).

Peripheral vision edited out the redundant stimuliin the pictures. The perceptual process is constantlytrying to find simplifying regularities and consis­tencies to detect and discard unwanted redundantstimuli which may overload input channels. Suchlawfulness, or constraint, between adjacent detailsallows people to predict a portion of the field froma knowledge of SOme other region (Attneave, 1954;Bevan, 1958). Redundancy can arise from overem­phasis by an extra cue along a further dimensionas well as from unnecessary details (staniland, 1966).

A content analysis of the details within each picturewas therefore essential. The square inches of eachpicture were grouped into those that contain texturesand those that contain contours, and Table 1 makesthis distinction. The mean incidence of fixations persquare was noticeably lower for most textures thanfor contours. Table 2 shows that textures were alsojudged less informative than contours. Between vari­ous textures, Table 1 shows how the rougher, lesspredictable textures of the land mass attractedslightly more fixations than did the smoother tex­tures of the sea. The smooth textures in the eyepicture were also seldom fixated.

The most marked effect of predictability wasseen when various contours were compared, espe­cially in the map picture. As with the eyes picture,a few squares were outstanding on bOth fixationdensity and judged recognizability. Figure 1 showsthat these squares contained unpredictable contoursof coastline. Their high fixation densities averaged2.8 percent, which was four times the incidence forthe smooth coastline at the right. There was nooverlap in the distributions between the averagefixation readings for each of the outstanding squaresshowing unpredictable coastline patterns and thosefor other squares displaying simpler coastline shapes.On the rating estimates, the unpredictable coastlineswere all above the midpoint of the scale, rangingbetween 5.3 and 7.4.

Predictable coastline contours were rated higher

5019

than their fixation density readings suggested theydeSOl"Ved. SUbjective rat1Dgs, therefore. give too higha score for the predictable lines in the pictures,at least in terms of fixation ltensity. All contours,whether predictable or not, were rated equally highon the scale. On the map, for example, the right­hand coastline, which was smooth and predictable,aver-aged 6.3-0 whereas the less predictable lowerleft-band coastline averaged 5.8, and the quite pre­dictable upper left-hand coastline ave~ 549. Allsquares which the ratings scored too high have beenmarked D (for 1!1sregarded) in Fig. 1. These weresquares which scored above the midpoint of 5.1 onthe .recop1zabiltty scale but received only 2 percentor less of the-fixations.

This three-way comparison between the subjective~ the fiDt10n density, and the pictorial contenthas shown lhat the discrepancies between fixationdensity and estimated values were a blessing indisguise. Comparisons between fixation readings andscale values for each square inch have provided anew experimental method of ident1{y1ng the areasIn stimulus displays that do not attract the gaze,even though they may contain features that areeasily recognized. The soUd D markings in Fig. 1,for example, have virtually reoutlined the predictableparts of the ~astline. This approach may thereforeprove to be a way of estimating and locating redun­dancy in contours. For example, the redundantregions of the predictable contours under the eyesand along the stra.1ght border of the mask also showsome D marking.

DllcusiH'Attneave (1954, 1959) was correct when he as­

s~rted that most of the informational content of ad!rawing is found where the contours change theirdirection. But his exposition of the informationalaspects of visual perception gave too much emphasisto the role of stra.1ght Unes or redundant contoursjoining such points, mainly because subjective mea­sures tend to give greater importance to such cuesthan is shown by visual fixation on reaUstic picturesin color. It may be that Attneave overemphasizedcontours because (1) he extrapolated to texturedmaterial from Une drawings, and (2) in his exposi­tion of shape perception he also compared televisionand human scanning procedures. (Although he dis­claimed that human perception depended upon anyprocess of this kind, Attneave did use machinescanning as an illustration.) Comparisons betweentelevision and human scanning can be dangerous,because the television fiy1ng-spot would representfor human Ss only a marked case of tunnel vision.Simple, predictable contours providing boundariesbetween colors or textures do not play a large partin direct human foveal scanning. These predictableoutlines and regular textures are processed by the

550

peripheral retina, and they therefore restrict thegeneral area of attention that the fovea exploresin its search for the unusual. S1m1larly, artists oftenuse blank or recurring items to keep the line ofsight <m saUent features.

Redundant and predictable contours in texturedpictures are processed and discarded by the periph­eral retina in less than 2 sec. Research on thevisual processes of cats has demonstrated a possiblephysiological mecbanism for this rapid process,because certain cortical cells are directly activatedby the peripheral retina. These cells in the cortexrespond electrically only to straight lines or neg­ligibly curved boundaries with a particular orien­tation. just as other cells respond only to linesthat show changes in direction. again with theappropriate orientation (Hubel 81 Wiesel, 1965). Atfirst one might believe that the reason for dis-

, carding smooth, predictable. and redundant contourswas merely that such outlines would not only beredundant but would also be, easier to perceive byperipheral vision. Against this possibility one hasto place the rejection of redundant textures; thesetextures are also viewed by peripheral vision, andyet the visual' acuity demands here may be veryhlgh. But with textures. it can easUy be demon­strated that one f1xa.t1on is sufficient to identify a.whole area comprising very fine but regular details.Quite different visual fixation results are obtainedwhen the task is changed from a judgment aboutthe relative coarseness of two adjacent textures toan inspection of each texture for fiaws. Many morefixations then occur per unit area (Mackworth 81Hiebert, 1967).

The distinction between unpredictable and simpleoutlines resembles findings by Berlyne (1966) thatSs spend longer times looking at the more complexthan at the less complex member of a pair of pat­terns. Zusne and Michels (1964) and Michels andZusne (1965) have found that angular complexitiesin polygon shapes attracted more fixations than didstraight lines. A check test with the map displayreversed from left to right ruled out possible spa­tial effects in the map piCture., Theunlque coastlinepatterns still attracted more fixations. This con­centration of fixations was therefore not due to theleft side of the picture dominating the right. Notealso that in the eyes picture there was the oppositetrend of the right side dominating the left.

In a comparison of Attneave's views on perceptionwith those of the Gibsons. Bevan (1958) noted thatboth approaches imply that the faithfulness withwhich the perception matches the object dependsupon the effIciency with which the perceptual mech­anism scans and encodes the input. Therefore, boththeories stress the correspondence between theexternal environment and our perception of it. Theindividual becomes more sensitive to the variables

Perception & Psychophysics. 1967. Vol. 2 (11)

of the stimulus array as he leams differential re­sponses to relevant cues.

OUr visual fixation data clearly support the Gibsons'(1953, 1955, 1963&. b) approach to perception. OUrrecords have demonstrated that details in the pic­tures determine to a considerable extent the adjust­ments made by the adult visual sensory mechanism,as Hake (1957) also predicted. But this confirmationof the Gibsonian views on the importance of per­ceptual differentiation does not in any way weakenBl'Wler's case for more accurate classification andbetter internal models of reality as features ofperceptual learning (Bl'Wler, 1957; Bl'Wler, Goodnow,& Austin, 1961; Bruner; Olver. & Greenfield, 1966).The use of more precise mental models impliesno retreat from the visual world (Zinchenko, Chzhi­Tsin, &I Tarakanov. 1963). On the contrary, thepresence of more accurate schemata suggests aneven more detailed cross-check between such in­ternal models and external stimulus patterns. Inadults. this matching process inevitably guides thefovea toward the unusual. specific features in thedisplay. Simultaneously. redundant and more pre­dictable features are relegated to the periphery ofthe retina; the peripheral retina therefore quicklyscreens of! the predictable features and leaves thefovea free to process the unpl'ed1ctable and unusualstimuli.

CoacllsitasA few outstanding areas within pictures received

high concentrations of the gaze and these regionsof the pictures were also judged to be very recog­nizable. These dominant regions always containedunpredictable contours or- unusual details.

Simpler, predictable contours were estimated tobe equally recognizable, but these outlines wereseldom fixated.

Half to two-thirds of each picture received veryfew fixations; in particular. areas of texture seldomreached the fovea, especially when the textures weresmooth and therefore predictable.

Peripheral vision edited out predictable contours,as well as regions of smooth texture, so rapidlythat the unusual and informative areas were fixatedin 2 sec or less from the start of the presentation.

Adults fixated details in pictures even during arelatively casual inspection of the scenes. This re­sult supports the Gibsonian view that perceptuallearning improves ability to discern distinctive fea­tures in the stimulus array.

A general approach has been devised which pin­points the salient features of pictures most likelyto be fixated. The combination of (1) objectivemeasurement by visual fixation patterns with (2)subjective estimates of recognizability or informa­tiveness can also indicate the predictable areas oftexture and find redundant contours. Both these

Perception & Psychophysics, 1967, Vol. 2 (11)

latter types of stimuli are less likely to be fixatedbecause they are so uninformative.

RefereacesAttneave. F. Applications of information theoT1l to pS1lcholog1l.

New York: Henry Holt and Co., 1959.Attneave. F. Some informational aspects of visual perception.

PS1lchol. Rev•• 1954,61, 183-193.Bartlett, F. C. Remembering. London: Cambridge, University

Press, 1932.Bedyne, D. E. Curiosity and exploration. Science. 1966. 153,25-33.Bevllll. W. Perception: Evolution of a concept. Psycho!. Rev .•

1958, 65. 34-55.Bruner. J. S. On perceptual readiness. Psuchol: Rev•• 1957. 64.

123-152.Bruner. J. S. The cognitive consequences of. early sensory depriva­

tion. In P. Solomon, et al. (Eds.), Sensory deprivation. Cam­bridge, Mass: Harvard University Press, 1965.

Bruner. J. S.• Goodnow. J. J .• & Austin. 6. A. A study of thinking.New York: John Wiley & Sons, lnc., 1961.

Bruner. J. S•• Olver. R. R.• & Greenfield. P. lit. Studies i7r cogni­tive growth. New York: John Wiley & Sons. Inc., 1966.

Gibson, E. J. Improvement in perceptual judgment as a function ofcontrolled practice or training. Psychol. Bull•• 1953, 50. 401-431.

Gibson. E. J. Perceptual learning. Annu. Rev. PS1lchol•• 1963a.14. 2~56.

Gibson, J. J. The useful dimensions of sensitivity. Amer.Psucho!••1963b. 18. 1-15.

Gibson. J. J .• & Gibson. E. J. Perceptual learning: Differentiationor enrichment? Psucnol, Rev•• 1955. 62, 32-41.

Hake. H. W. Contribution of psychology to tile study of patternvision. USAW WADe Tech.. Rept; 1957.57-621.

Hake. H. W.• Rodwan. A.• & Weintraub. D. NoIse reduction inperception. In K. R. Hammond (Ed,), The pSllcholog1l of EgonBrunswik. New York: Holt, Rinehart and Winston, Inc .. 1966.

aJbel. D. G.• & Wiesel. T. N. Receptive fields and functionalarchitecture in two non-striate areas (18 and 19) of the cat. J.Neuroph,lsiol.• 1965, 29, 229-289. especially pp. 285-286.

Mackworth, N. H. A stand camera for line-of-sight recording. Per­cept. & PS1/chophys•• 1967, 2, 11~127.

Mackworth. N. H.• & Bnmer, J. S. Selecting visual informationduring recognition by adults and children. Mimeographed mono­graph, Harvard University Center for Cognitive studies, 1966.

Mackwortll. N. H•• & Hiebert. Joyce. Measuring the useful field ofview during visual search. (In preparation) 1967.

Micbels. K. M••& Zusne. L. Metrics of visual form. Psychol. Bull••1965. 63. 74-86.

Miller, G. A., Galanter, E. H.• & Pribram, K. H. Plans and thestructure of behavior. New York: Henry Holt & Co., 1960.

Senders. J. W. A re-analysis of the' pilot eye-movement data.I.E.E.E. Trans. on Human Factors in Electronics. Vol. HFE-7.No.2, June, 1966, 103-106.

Spinelli, D. H., & Pribram, K. H. Changes in visual recoveryfunctions and unit activity produced by frontal and temporalcortex stimulation. EEG clin. Neuroph1lsiol•• 1966. 20. 44-49.

Staniland. A. C. Patterns of redundancy: A psychological stud,l.New York: Cambridge University Press, 1966.

Zincbenko, V. P .• Chzhi-Tsin. V.• & Tarakanov. V. V. The forma­tion and development of perceptual activity. Soviet Psychol. &PS1/chiat .• 1963, 2, 3-12.

Zusne, L .• & Michels, K. M. Non-representational shapes and eyemovements. Percept. mot. Skills. 1964, 18, 11-20.

NOleS1. The research reported herein was supported by the NationalScience Foundation, Contract No. G8-192. to Harvard University.Center for Cognitive Studies and by the Cooperative ResearchProgram of the Office of Education, U. S. Department of Health,

551

Education. and Welfare Contract No. OE-4-10-136. Project No. E­020, to Harvard University, Center for Cognitive Studies, and alsoby the National Aeronautic and Space Administration ResearchGrant NsG718 to Harvard University, Guggenheim Center for Aero­space Health and Safety. Thanks are due to Dr. Jerome S. Bruner,Dr. Jane F. Mackworth, and Dr. Herbert E. Krugman for advice,as well as to Joyce M. Hiebert for technical assistance. Acknow­ledgements are also owing to Mr. Ken Heymans for his courtesy

552

in allowing us to use his mask photograph. Similarly, we thankthe National Aeronautics and Space Administration for providingthe picture of Baja California.2. Now at Department of Psychiatry (Neuropsychology), StanfordUniversity, Palo Alto, California.3. Now at the Department of Psychology, Northeastern University,Boston, Massachusetts.

(Accepted fOT publication July 25, 1967.)

Perception & Psychophystcs , 1967. Vol. 2 (11)