Why do things happen the way they do in your imagination?

● Is it because of the format of your image or your cognitive architecture? Or because of what you know? Did it reveal a capacity of mind? Or was it because you made it do what it did?

● Can you make your image have any properties you choose? Or behave in any way you want? Why not? How about imagining an object from all directions at once, or

from no particular direction? How about imagining a 4-dimensional object? Can you imagine a printed letter which is neither upper nor

lower case? A triangle that is not a particular type?

More on Representation of space and arguments from neuroscience

● The intuition that images are spatial (or, as some put it, that they “have space”) is an interesting one and deeper than most other questions being studied in laboratories. I have discussed this in excruciating detail in the last chapter of my “Things and Places”.

● According to Kosslyn, the argument has now entered the third and final stage, where neuroscience evidence has put an end to the debate. I will look at some of this alleged debate-ending evidence and will show that it is characterized by desperation and lack of critical analysis.

A brief look at whether we may be storing information in the form of an icon

● For present purposes I want to look at the issue of explanation and I will do this by considering the sense in which spatial information may (must?) itself be spatial

● This matters to the question of what forms of representation we have in our minds and how these refer to things in the world.

Our studies of mental scanning

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

1 2 3 4

Relative distance on imageL

aten

cy (

secs

)

scan image

imagine lights

show direction

(Pylyshyn & Bannon. See Pylyshyn, 1981)

There is even reason to doubt that one can imagine scanning continuously (Pylyshyn & Cohen, 1998)

Does visual imagery use the visual system?• It depends on what you mean by use and visual system

To use vision can mean actually to perceive the image Only early vision is relevant to this use of “perceive”, since general vision

can involve all of cognition This question is of interest to picture theorists because if vision is involved it

may suggest that images are uninterpreted picture-like (or depictive) spatial displays. The visual module cannot be applied to already-interpreted data structures

• Let’s assume that “the visual system” is “active” during mental imagery as some have suggested

What follows from that?

• Why should we believe that vision may be involved? In the last 15 years the main support for the assumption that vision is

involved has come from neuroscience.

Major Question 2:

Reasons for thinking that images are interpreted by the visual

system• Similar phenomenology of imagining &

seeing– This reason overshadows all others

• Superposition & interference studies

• Visual illusions with projected images– The ubiquitous role of attention

• Re-perceiving and novel construals– A large but very problematic literature

More demonstrations of the relation between vision and imagery

• Images constructed from descriptions– The D-J example(s)– The two-parallelogram example

• Amodal completion

• Reconstruals: Slezak

Can images be visually reinterpreted?• There have been many claims that people can visually

reinterpret images These have all been cases where one could easily figure out

what the combined image would look like without actually seeing it (e.g., the J – D superposition).

• Pederson’s careful examination of visual “reconstruals” showed (contrary to her own conclusion) that images are never ambiguous (no Necker cube or figure-ground reversals) and when new construals were achieved from images they were quite different from the ones achieved in vision (more variable, more guessing from cues, etc).

• The best evidence comes from a philosopher (Slezak, 1992, 1995)

Slezak figures

Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

Slezak figures rotated 90o

Do this imagery exercise:Imagine a parallelogram like this one

Now imagine an identical parallelogram directly below this one

Connect each corner of the top parallelogram with the corresponding corner of the bottom parallelogram

What do you see when you imagine the connections?Did the imagined shape look (and change) like the one you see now?

Amodal completion by images?

Is this what you saw?

Off-retinal info different from foveal info

Off-retinal info different from foveal info

Labels propagate over picture

Anorthoscope: Scan view How many contours are there?

Anorthoscope: Scan view What is the shape?

Vision is involved when images are superimposed onto vision

Many experiments show that when you project an image onto a display the image acts very much like a superimposed display• Shepard & Podgorny, Hayes, …• Interference effects (Brooks)• Mental scanning• Interaction with the motor system (Finke)

Shepard & Podgorny experiment

Both when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

Brooks’ spatial interference study

Respond by pointing to symbols in a table or by saying the words left or right

Visual-Motor adaptation and motor adaptation to images

● The basic prism adaptation setup: arm movement towards a target while wearing prism glasses

● Now repeat with arm unseen but subject told where it is (actually where it would have been in the prism case)

● Get adaptation {Finke, R. A. (1979). The Functional Equivalence of Mental Images and Errors of Movement. Cognitive Psychology, 11, 235-264.}

● But in the original experiment you don’t need to see a hand, any indicator of where the hand is will do as long as the subject believes it is where indicated

Standard view of saccadic integration by superposition

Is it plausible? Is it true?

Superposition does not work (O’regan 1983)

This is what our conscious experience suggests goes on in vision…

Kliban

This is what the demands of explanation suggests must be going on in vision…

Are there pictures in the brain?

• There is no evidence for cortical displays of the right kind to explain visual or imaginal phenomena

Is there very short iconic storage?

● Although the idea of pictorial long-term memory is not supported, there is some provisional evidence that sensory information outlasts the duration of the stimulus. Many people have studied these “sensory buffers” including George Sperling and Michael Posner.

Sperling’s partial report method for showing an iconic memory

Posner’s demonstration of short lasting shape information

Fast

Fast

Slower

Slower

So what does the visual system produce?

• The best hypothesis so far (i.e., the only one that has not been shown to be clearly on the wrong track) is that the brain is a species of computer in which representations of the world are encoded in the form of symbol structures, and actions are determined by calculations (i.e., inferences) based on these symbolic encodings.

• Therefore vision (and other sensory systems) compute symbolic expression for the rest of the (Turing) computing machinery to use.

So why does it not feel like we are doing computations?

The content of our conscious experience is a very poor guide to what is actually going on that causes both our experiences and our behavior. Science is concerned with causes, not just correlations.

We have learned (since Galileo) that we can’t assume that the way things seem has much to do with how it works (consider the example of language understanding) As in most sciences, the essential causes are far from obvious

(e.g., How can the moon exert a pull on the earth without contacting it? What is this table made of? etc.).

In the case of cognition, what is going on is a delicate mixture of the obvious (what Granny or Shakespeare knew about people and why they do what they do) and the incredible.

We can’t even be sure that we have the right methods or instruments

Mental Scanning

● Some hundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance.

● These have been reviewed and described in: Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental

images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.

Rarely cited are experiments by Pylyshyn & Bannon which I will summarize for you.

Studies of mental scanningDoes it show that images have metrical space?

(Pylyshyn & Bannon. See Pylyshyn, 1981)

Conclusion: The image scanning effect is Cognitively Penetrable i.e., it depends on goals and beliefs, or on Tacit Knowledge.

What is assumed in imagist explanations of mental scanning?

● In actual vision, it takes longer to scan a longer distance because real distance, real motion, and real time is involved, therefore this equation holds due to natural law:

Time = distance

speed

But what ensures that a corresponding relation holds in an image?

The obvious answer is: Because the image is laid out in space!

But what if that option is closed for empirical reasons?● Imagists appeal to a “Functional Space” which they liken to a matrix

data structure in which some cells are adjacent to other cells, some are closer and others further away, and to move from one to another it is natural that you pass through intermediate cells

● Question: What makes these sorts of properties “natural” in a matrix data structure?

Thou shalt not cheat

● There is no natural law that requires the representations of time, distance and speed to be related according to the motion equation. You could equally easily imagine an object moving instantly or according to any motion relation you like, since it is your image!

● There are two possible answers why the relation

Time = Representation of distance

Representation of speed

typically holds in an image-scanning task:

1. Because subjects have tacit knowledge that this is what would happen if they viewed a real display, or

2. Because the matrix is taken to be a simulation of a real-world display, as it often is in computer science

Mental Rotation has been one of the most cited demonstrations of all

● Look at the following 3D figures and judge which pairs are the same except for orientation. The other pair are enantiomorphs – 3D mirror images so they can’t be put into correspondence by 3D rotation only.

Mental rotation

Time to judge whether (a)-(b) or (b)-(c) are the same except for orientation increases linearly with the angle between them (Shepard & Metzler, 1971)

What do you do to judge whether these two figures are the same shape?

When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?

Is this how the process looked to you?

Thou shalt not cheat● What happens in ALL imagist accounts of

phenomena from mental scanning to mental rotation is that they assume the properties of real space in order to provide a principled explanation, then retreat to something not-quite-real space when it is pointed out that they are assuming that images are laid out in real space-in-the-head.

● This happens with mental rotation as well, even though the tacit knowledge account is not plausible there (it is an involuntary and universal way of solving the rotated-figure task so long as the task involves tokens of enantiomorphs).

The missing bit of logic:

● According to Prinz (2002) p 118,“If visual-image rotation uses a spatial medium of the kind Kosslyn envisions, then images must traverse intermediate positions when they rotate from one position to another. The propositional system can be designed represent intermediate positions during rotation, but that is not obligatory.”

● Given that this happens in 3D so that it can’t be a literal brain space, the question arises: What makes this obligatory in “functional Space”?

Sources of obligatory constraint● It could be that there is a medium or a set of analog properties

that together happen to simulate a virtual space, and the physical properties of this medium enforce the continuity of motion through it. This is very unlikely.

● It could be that the constraint comes from the fact that it holds in the world. Then its transfer from the real to the mental world occurs either ; Voluntarily, because we know how it would happen and we can use

that fact to solve the problem Naturally, because the constraint got built in over time through

evolutionary pressures Naturally, but not because the constraint is built in, but because of

other properties of the architecture that make it more efficient to compute the rotated shape incrementally until there is a match

Do images have low-level psychophysical properties?

● Bring the bars closer and closer together. In which can you see the bars when the spacing is closest?

● In experiments, it was shown that the oblique effect occurs in mental images just as it does in vision Kosslyn, Thompson & Ganis (2006) argued that this is

because there are more vertical and horizontally tuned cells than obliquely tuned cells in visual cortex. Does this explain the finding?

Do images have low-level visual properties?● Imagine a grating in which the bars are:

1. Horizontal2. Vertical3. Oblique (45°)

A final point…● In Kosslyn, Thompson & Ganis (2007) the authors cite

Ned Block to the effect that one does not need an actual 2D surface, so long as the connections upstream from the cortical surface can decode certain pairs of neurons in terms of their imagined distance. Imagine long stretchy axons going from a 2D surface to subsequent processes. Now imagine that the neurons are randomly moved around so they are no long on a 2D layout. As long as the connections remain fixed it will still behave as though there was a 2D surface.

● Call this the “encrypted 2D layout” version of literal space.

The encrypted-spatial layout alternative● By itself the encrypted-layout alternative will not do because

without referring to the original 2D locations, the relation between pairs of neurons and scan time is not principled. In the end the only principle we have is Time=distance/speed so unless the upstream system decrypts the neuron locations into their original 2D surface locations the explanation for the increase in time with increased imagined distance remains a mere stipulation. It stipulates, but does not explain why, when two points are further away in the imagined layout it takes longer to scan between them or why scanning between them requires that one visit intermediate locations along the way.

● But this is what we needed to explain! One can apply such a mere stipulation to any form of representation. What was a principled explanation with the literal 2D display has now been given up for a mere statement of how it shall be.

The ‘Imagery Debate’ Redux

● According to Kosslyn there have been 3 stages in the debate over the nature of mental images:

1. The role of images in learning and memory (Paivio’s Dual Code theory). Influential at the time but now abandoned except for a few recidivists like Barsalou.

2. Spatial properties of images and their role in dynamic processes, as assessed by reaction time measures (Kosslyn’s research on ‘metric properties of images’)

3. Discovery of brain mechanisms underlying visual imagining, which led to ‘the resolution of the imagery debate’.

Explaining mental scanning, mental rotation and image size effects in terms of “functional space”

● When people are faced with the natural conclusion that the “iconic” position entails space (as in scanning and size effects) they appeal to “functional space”

● A Matrix in a computer are often cited as an example● Consider a functional space account of scanning or of

mental rotation: Why does it take longer to scan a greater distance in a

functional space? Why does it take longer to rotate a mental image a greater

angle?

But there are examples of solving geometry problems easily with imagery

• There are many problems that you can solve much more easily when you imagine a layout than when you do not.

• In fact many instances of solving problems by imagining a layout that seem very similar to how would solve them if one had pencil-and-paper.

• The question of how pictures, graphs, diagrams, etc help in reasoning is very closely related to the question of how imagined layouts function in reasoning. That is not in question. What is in question is what happens in either the visual or imagined cases and how images can benefit from this processes even though there is no real diagram.

How do real visual displays help thinking?

Why do diagrams, graphs, charts, maps, icons and other visual objects help us to reason and to solve problems?

The question why visual aids help is nontrivial and Seeing & Visualizing, chapter 8 contains some speculative discussion, e.g., they allow the visual system to:• make certain kinds of visual inferences

• make use of visual demonstratives to offload some of the memory load

• Capitalize on the fact that the displays embody the axioms of measure theory and of geometry (which are then inherited by thought)

The big question is whether any of these advantages carry over to imaginal thinking! Do mental images have some (or any) of the critical properties that make diagrams helpful in reasoning?

Visual inferences?● If we recall a visual display it is because we have encoded

enough information about its visual-geometrical properties that we can meet some criteria, e.g., we can draw it. But there are innumerably many ways to encode this information that are sufficient for the task (e.g. by encoding pairwise spatial relations, global spatial relations, and so on). For many properties the task of translating from one form to another is much more difficult than the task of visually encoding it – the translation constitutes visual inference.

● The visual system generalizes from particular instances as part of its object-recognition skill (all recognition is recognition-as and therefore assumes generalization from tokens to types). It is also very good at noticing certain properties (e.g., relative sizes, deviations from square or circle, collinearity, inside, and so on). These capabilities can be exploited in graphical layouts.

Memorize this map so you can draw it accurately

From your memory:• Which groups of 3 or more locations are collinear?• Which locations are midway between two others?• Which locations are closest to the center of the island?• Which pairs of locations are at the same latitude?• Which is the top-most (bottom-most) location?

If you could draw the map from memory using whatever properties you noticed and encoded, you could easily answer the questions by looking at your drawing – even if you had not encoded the relations in the queries.

Draw a rectangle. Draw a line from the bottom corners to a point on the opposite vertical side. Do these two lines intersect? Is the point of intersection of the two lines below or above the midpoint? Does it depend on the particular rectangle you drew?

A B

CD

m m’

y

x

Which properties of a real diagram also hold for a mental diagram?

● A mental “diagram” does not have any of the properties that a real diagram gets from being on a rigid 2D surface.

● When you imagine 3 points on a line, labeled A, B, and C, must B be between A and C? What makes that so? Is the distance AC greater than the distance AB or BC?

● When you imagine drawing point C after having drawn points A and B, must the relation between A and B remain unchanged (e.g., the distance between them, their qualitative relation such as above or below). Why?

● These questions raise what is known as the frame problem in Artificial Intelligence. If you plan a sequence of actions, how do you know which properties of the world a particular action will change and which it will not, given that there are an unlimited number of properties and connections in the world?