Sunday, April 25, 2010

Consciousness (12): What the brain thinks it knows

Number twelve in my series of posts on consciousness. Table of Contents is here.

In this continuation of the previous two posts, let’s finish examining the parallels between linguistic interpretation and visual perception. Recall the first three similarities were: 1) applicability of the content/vehicle distinction, 2) the possibility of ambiguity, and 3) sensitivity to context. In this post we'll add one final point to the list.

4. Use of background knowledge/assumptions
As we discussed previously (post eight), interpreting a sentence draws on your background knowledge about the world. A sentence whose meaning is transparent to you (e.g., ‘George Washington was the first President of the US’) will be perplexing to someone who doesn’t share your knowledge of American history.

For another example, if someone were to utter ‘John was told what to do by the river,’ we (often implicitly) use our knowledge that rivers cannot talk to home in on the correct interpretation. If we were to replace 'river' with 'teacher' we would not be able to use our knowledge to determine the correct interpretation.

Even fallible assumptions about the structure of the world can bias us toward interpreting a sentence a certain way. For instance, we can modify a semantically "bistable" sentence from post eleven ('I saw the man with the telescope') in ways that will make folks much more likely to settle on one of the interpretations:
[1] The astronomer stared at the girls with his telescope.
While [1] is technically still ambiguous, the modifications make us much more likely to interpret the sentence as describing someone using his telescope to spy on some girls.

Our brain also seems to use background knowledge and assumptions about the world when generating its perceptual response to a stimulus. As an example, let’s consider the brain’s apparent knowledge of perspective.

Perspective refers to the depth cues that result when a scene is projected onto a flat sheet located at a particular location (such as a retina, as described in post nine). One such cue is linear perspective, which refers to the fact that a projection of two parallel lines that recede into the distance will converge to a common point rather than remain parallel. This can be seen in any stock image of railroad tracks, as in Figure 1. While we know the tracks are actually parallel to one another, when they project to a flat surface (such as a retina or the plate of sensors in your digital camera), they appear to converge.

Figure 1: Railroad tracks illustrating linear perspective.

The brain seems to "understand" linear perspective, as evidenced by the powerful experience of depth that results when artists recreate its effects on a two-dimensional sheet of paper or canvas. There are artists that, armed with nothing more than wits and a few pieces of chalk, are able to create a vivid impression of a rich three-dimensional scene on a sidewalk (Figure 2) or even an entire street (Figure 3). You can see the convergence of lines in these works, the use of linear perspective that helps create a striking impression of depth.

Figure 2: Taking the Plunge, a sidewalk chalk drawing by Julian Beever.

Figure 3:Lava Burst, as painted on the streets of Gelden, Germany.

Because our brain evolved partly to help us navigate three-dimensional terrain, it isn’t too surprising that it is so keyed in to such cues. Even when we know at a cognitive level that we are looking at a two-dimensional surface, our brains reflexively construct an experience of depth. The great psychologist Gordon Shepard, in his inimitable style, explains things as follows:
[W]e cannot choose to see a drawing merely as what it is—a pattern of lines on a flat, two dimensional surface. To the extent that that pattern of lines conforms with the rules of linear perspective, for example, that pattern automatically triggers the circuits in the brain that make the three-dimensional interpretation appropriate to such a perspective display. Any consciously adopted intentions to ignore such an interpretation are largely powerless against the swift deliverances of this underlying machinery. This should not surprise us. We have inherited this machinery from individuals who, long before the advent of picture making, interpreted—by virtue of this machinery--what was going on in the three-dimensional world around them with sufficient efficiency to survive and to continue our ancestral line.
In the normal world of perception in a three-dimensional world, such perspective cues are quite reliable indicators of the spatial structure of the world. Clever artists (including 3D cinematographers) and psychologists merely exploit the neuronal circuits which unquestionably believe such cues even in ethologically peculiar contexts.

Let’s finish by looking at a couple of rather stunning visual illusions psychologists have created to exploit the brain’s knowledge of perspective. First, consider the two tables in Figure 4. The tops of the two tables are actually the exact same size! If you were to rotate the left table top by 90 degrees and move it to the right, the two tabletops would overlap perfectly. Especially surprising is that the long edge of the table on the left (the edge that appears to be receding away from the viewer) is the same length as the front edge of the table on the right (the edge roughly parallel to the viewer).

Figure 4: Turning the tables.

It’s as if an understanding of the physics of projection is hard-wired into our visual system, and our brain automatically makes adjustments when it "thinks" an object’s size has been contracted because of viewing angle. Shepard says, "The fact that the retinal images of the two quadrilaterals interpreted as table tops are identical in length then implies that the real length of the table going back in depth must be greater than the real length of the crosswise table." In other words, because the table on the left is (apparently) receding into the distance, it must actually be longer than the table on the right, which runs parallel to the viewer.

Such compensation by our brain for such perspective-dependent distortions can also explain why an object such as a coin appears circular even when presented at an angle so it actually projects an oval to the retina.

A related illusion is shown in Figure 5, which appears to be a sort of display case for four pieces of plumbing. Each piece consists of two tubes connected in the middle by a ball. While the angles between the tubes looks quite different, they are actually all identical, as demonstrated in the bottom panel of the figure.

Figure 5: Purves’ plumbing.

Figure 5 contains strong cues about the spatial arrangement of the pieces relative to the viewer. For instance, the tubes in the red structure span from the back of the scene to the front, which would only be possible if the two tubes subtended an obtuse angle. While the structure happens to be projecting a right angle to the retina, this is because of its contingent orientation with respect to the viewer. Similarly, the angle between the tubes in the green piece clearly seems to be acute, even though it projects an identical 90 degree angle to our retina.

Again we find that the brain seems to compensate for the distorting filter of perspective, generating an experience to conform more closely with the actual angle than the stimulus (i.e., the angle projected to the retina).

Where next?
That's the last parallel I'll explore between language and perception. With that, we are ready to more closely evaluate the hypothesis that perception is a form of stimulus interpretation. We'll do that in the next post.

Sources of examples
‘John was told what to do by the river’ is from Norvig (1988), an article which also inspired sentence [1]. 'Taking the Plunge' was created by Julian Beever ( The street carnage 'Lava Burst' was drawn by Edgar Mueller ( 'Turning the Tables' is from Shepard (1991). Purves' Plumbing is from Purves and Lotto (2003).

Norvig, P (1988) Multiple Simultaneous Interpretations of Ambiguous Sentences. Proceedings of the 10th Annual Conference of the Cognitive Science Society.

Purves, DP, and Lotto, RB (2003) Why we see what we do: An empirical theory of vision Sinauer Associates.

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

Table of Contents of posts on consciousness.

Thursday, April 01, 2010

Consciousness (11): Ambiguity and Context in Perception and Language

Eleventh in my series of posts on consciousness. Table of Contents is here.

In this sequel to the previous post, we continue to examine the similarities between linguistic interpretation and perception, and explore the hypothesis that perception is stimulus interpretation.

2. Bistability
Just as we can experience bistable visual percepts, sentences can exhibit a “semantic bistability” in which there exist two equally reasonable interpretations. For example:
[1] Adam asked him to use the bathroom.
What is the meaning of [1]? Did Adam request that someone else go to the bathroom, or did Adam wish to use the bathroom?
[2] I saw the man with the telescope.
Who has the telescope, me or the man? There isn't enough information in sentences [1] or [2] to determine which interpretation is correct.

Similarly, ambiguous visual stimuli (discussed here and here) such as the Necker Cube don’t provide enough disambiguating information for our visual system to settle on one perceptual interpretation. Koch (2004) says:
Consider the twelve lines making up the Necker Cube. Due to the inherent ambiguity of inferring its three-dimensional shape from a two-dimensional drawing, the lines of the cube can be interpreted two ways, differing only in their orientation in space. Without perspective and shading cues, you are as likely to see one as the other. The physical stimulus--the line drawing--doesn't change, yet conscious perception flips back and forth between these two interpretations, in what is a paradigmatic example of a bistable percept.
Koch is claiming that because the visual stimulus does not adequately disambiguate the source object (i.e., the line drawing is ambiguous), our conscious experience flips back and forth between two plausible interpretations of the image.

Both types of bistability (i.e., linguistic and stimulus) can easily lead to errors. For instance, someone may understandably interpret sentence [2] to mean you were spying on someone, when in fact you really meant to say you innocently observed a guy that was carrying a telescope around. In the case of vision, if you see an actual wire frame of a cube (a real-life Necker Cube) your visual system will lock in on one or the other percept with about equal probability, and half the time it will be wrong.

3. Context dependence
The context in which a sentence is uttered can have strong effects on how we will interpret it. For example, in a conversation about money, you will likely interpret ‘I went to the bank’ differently than in a conversation about a canoe trip down a river.

Likewise, the visual experience produced by a localized stimulus depends strongly on the context of concurrent surrounding stimuli. For example, the top panels of Figure 1 show two cubes that are tiled with colored squares (a Rubik’s Cube sort of situation). The cubes seem to be placed in different sources of illumination: the cube on the left seems to be in yellow light, and the right cube in blue light. Remarkably, the blue tiles on top of the left cube are identical to the yellow tiles on top of the right cube. If you look at them in isolation, these tiles actually appear gray, as shown in the bottom half of Figure 1. Changing the visual context of these gray squares radically changes how we experience them.

Figure 1: Context dramatically affects color perception.

Another example of contextual influences on visual perception is shown in Figure 2, which depicts one monster chasing another through a sewer. While the monster chasing the other looks appreciably bigger, the two monsters are actually identical copies of one another, and project the same image to the retina. The context of the receding tunnel makes the monster on top seem much further away, and our brain somehow magnifies its apparent size based on such depth cues.

Figure 2: Big meanie chasing terrified rascal.

These examples demonstrate that our experience of something at a localized region of space is based partly on what is happening at other locations in our visual field. Our brain is quite sensitive to contextual factors in its construction of a percept.

In the next post we will finish this foray comparing perception and interpretation.

Sources of examples
The analogy between perceptual bistability and sentence ambiguity has been noted before. Norvig (1988) points to Hockett (1954) as the first. Hockett wrote, "The hearer, confronted with The old men and women stayed at home, is in much the same position as the observer who sees a picture of a hollow cube and can, almost at will, see first one corner and then another as closer to him." The sentence mentioned by Hockett can either mean that the old men and old women stayed, or that the women and the old men stayed.

Sentence [1] is from John Limber (personal communication), whose article Syntax and sentence interpretation (1976) has clear influences on this post. Sentence [2] seems to be ubiquitous in the linguistics literature, and its origins are opaque to me. I saw a reference to it in a 1961 memorandum from the RAND corporation, and I am trying to track it down. Figure 1 is from Shepard (1991). Figure 2 is from Purves and Lotto (2003) (there are many exceptional illusions at Purves' web site).

Hockett, CF (1954) Two models of grammatical description, Word, 386-399.

Koch, C (2004) The Quest for Consciousness: A neurobiological approach Roberts & Company Publishers.

Limber, J. (1976) Syntax and sentence interpretation, In R. Wales & E. C. T. Walker (Eds.), New
approaches to language mechanisms
(pp. 151-181). Amsterdam: North Holland.

Norvig, P (1988) Multiple Simultaneous Interpretations of Ambiguous Sentences. Proceedings of the 10th Annual Conference of the Cognitive Science Society.

Purves, DP, and Lotto, RB (2003) Why we see what we do: An empirical theory of vision Sinauer Associates.

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

Table of Contents of posts on consciousness.