Consciousness (15): Opening the time capsule

Table of Contents of posts on consciousness.
-------------------------------------

Below you'll find a series of quotations that highlight the topics we have been discussing in the last nine posts. I chose them for their exceptional eloquence, clarity, and influence. They are not in chronological order, but are roughly in ascending order of specificity of the claims. This will be the final post in this narrative arc.

The quotes

[V]ision is the process of discovering from images what is present in the world, and where it is.
     -David Marr (1982)

We may define visual perception as attributing objects to images.
     -Richard Gregory (2009)

Visual perception involves coordination between sensory sampling of the world and active interpretation of the sensory data. Human perception of objects and scenes is normally stable and robust, but it falters when one is presented with patterns that are inherently ambiguous or contradictory. Under such conditions, vision lapses into a chain of continually alternating percepts, whereby a viable visual interpretation dominates for a few seconds and is then replaced by a rival interpretation. This multistable vision, or ‘multistability’, is thought to result from destabilization of fundamental visual mechanisms, and has offered valuable insights into how sensory patterns are actively organized and interpreted in the brain…
     -Nikos Logothetis (2002)

[The Necker cube] has an interesting property. Look at it fairly steadily for a while, and the cube will invert, as if it were being viewed from another angle. After a time the percept switches back to the original one, and so on. In this case there are two equally plausible 3D interpretations of the image, and the brain is uncertain which it prefers. Notice that it only chooses one at a time, not some odd mixture of both of them…

The reason you normally see without ambiguity is that the brain combines the information provided by the many distinct features of the visual scene (aspects of shape, color, movement, etc) and settles on the most plausible interpretation of all these various visual clues taken together…[W]hat the brain has to build up is a many-leveled interpretation of the visual scene, usually in terms of objects and events and their meaning to us.
     -Francis Crick (1995)

We don’t directly experience what happens on our retinas, in our ears, on the surface of our skin. What we actually experience is a product of many processes of interpretation—editorial processes, in effect. They take in relatively raw and one-sided representations, and yield collated, revised, enhanced representations, and they take place in the stream of activity occurring in various parts of the brain. This much is recognized by virtually all theories of perception…
     -Dan Dennett (1991)

The mental activities that lead us to infer that in front of us at a certain place there is a certain object of a certain character, are generally not conscious activities, but unconscious ones. In their result they are equivalent to a conclusion, to the extent that the observed action on our senses enables us to form an idea as to the possible cause of this action; although, as a matter of fact, it is invariably simply the nervous stimulations that are perceived directly, that is, the actions, but never the external objects themselves. But what seems to differentiate them from a conclusion, in the ordinary sense of that word, is that a conclusion is an act of conscious thought… Still it may be permissible to speak of the mental acts of ordinary perception as unconscious conclusions, thereby making a distinction of some sort between them and the common so-called conscious conclusions.
     -Hermann von Helmholtz (1866)

Perception consists of interpreting two-dimensional retinal images of a three-dimensional world. The process of projecting a three-dimensional scene onto a two-dimensional retina necessarily discards information about the three-dimensional structure of the scene. This makes it impossible, in principle, to deduce all of the three-dimensional structure of a scene…However, even though such problems cannot be solved by deduction, acceptable solutions can be found using statistical inference.
     -JV Stone (2009)

Our visual experience evidently is the product of highly sophisticated and deeply entrenched inferential principles that operate at a level of our visual system that is quite inaccessible to conscious introspection or voluntary control. We do not first experience a two-dimensional image and then consciously calculate or infer the external three-dimensional scene that is most likely, given that image. The first thing we experience is the three-dimensional world—as our visual system has already inferred it for us on the basis of the two-dimensional input. Hermann von Helmholtz, the great nineteenth century scientist who more than any other single individual laid the foundations for our present understanding of visual and auditory perception, expressed this by characterizing perception as “unconscious inference.”
     -Roger Shepard (1991)

[T]he brains’ representations are hypotheses, predictive like the hypotheses of science. Like science, perception bets from available evidence on what is likely to be true…For perception, there is always guessing and going beyond available evidence. On this view, the closest we ever come to the object world is by somewhat uncertain hypotheses, selected from present evidence and enriched by knowledge from the past. Some of this knowledge is inherited—learned by the statistical processes of natural selection and stored by the genetic code. The rest is brain-learning from individual experience, especially important for humans.
     -Richard Gregory (2009)

It is the business of the brain to represent the outside world. Perceiving is not just sensing but rather an effect of sensory input on the representational system. An ambiguous figure provides the viewer with an input for which there are two or more possible representations that are quite different and about equally good, by whatever criteria the perceptual system employs. When alternative representations or descriptions of the input are equally good, the perceptual system will sometimes adopt one and sometimes another. In other words, the perception is multistable.
     -Fred Attneave (1971)

We have suggested that the biological usefulness of visual consciousness in humans is to produce the best current interpretation of the visual scene in the light of past experience, either of ourselves or of our ancestors (embodied in our genes), and to make this interpretation directly available, for a sufficient time, to the parts of the brain that contemplate and plan voluntary motor output, of one sort or another, including speech.
     -Francis Crick and Christof Koch (1998)

References
Attneave, F (1971) Multistability in perception. Sci Am. 6: 63-71.

Crick, Francis (1995) The Astonishing Hypothesis: The Scientific Search for the Soul. Scribner.

Crick, F, and Koch, K (1998) Cerebral Cortex, 8:97-107.

Dennett, D (1991) Consciousness Explained. Back Bay Books.

Gregory, RL (2009) Seeing Through Illusions. Oxford University Press.

Helmholtz, H. von 1866 Concerning the perceptions in general. In Treatise on physiological optics, vol. III, 3rd edn (translated by J. P. C. Southall 1925 Opt. Soc. Am. Section 26, reprinted New York: Dover, 1962).

Leopold, Wilke, Maier, and Logothetis (2002) Stable perception of visually ambiguous patterns. Nature Neuroscience 5: 605-609.

Marr, D (1982) Vision. WH Freeman, NY.

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

Stone, JV, Kerrigan, IS, and Porrill, J (2009) Where is the light? Bayesian perceptual priors for lighting direction. Proc R Soc B 276: 1797-1804.

-------------------------------------
Table of Contents of posts on consciousness.

Consciousness (14): Interpretation Mechanics

Number fourteen in my series of posts on consciousness. Table of Contents is here.
-------------------------------------

The perception-as-interpretation view, summarized in the previous post, is useful as an informal ordinary-language hypothesis about consciousness. However, it is obviously a long shot from a final literal scientific theory. While it seems to be a useful way of speaking, I think we should look at it as an interesting suggestion, or perhaps even an inspiration that will lead us toward a more specific and fleshed-out theory.

There is a diverse range of theories that turn out to be special cases of the perception-as-interpretation hypothesis. These theories describe the interpretation-building mechanism alternatively as neuronal ‘model building’, ‘emulation,’ ‘virtual reality construction,’ ‘simulation of a world’, or ‘unconscious inference.’ Each specific theory carries slightly different assumptions about how the brain constructs experience, but most of them share two or more of the general features of the interpretation view delineated in previous posts.

Among psychologists, the most influential of these views is that the brain uses an unconscious inference procedure to construct a hypothesis about the source of a retinal projection. Because this theory of perception is so interesting, influential, and useful, I’ll describe it in a little bit of detail before stepping back to speak more generally about all of these theories.

Marty the Brain Scientist
To help us understand this theory, let’s imagine a tiny scientist, Marty, who lives and works in your brain (Figure 1). His sole occupation, every moment, is to monitor the movies playing on your retinae, and to build a hypothesis about their source in the world. Our conscious experience is identical to the specific hypotheses that Marty generates. For instance, if Marty’s best hypothesis about the source of the stimuli is that there is a red ball three feet to your left, that is precisely what you will see. Marty is fairly motivated to generate hypotheses accurately and quickly: after all, if you die, he dies. The more accurate his hypotheses, the better you will be able to interact with the world.

Figure 1: Marty the brain scientist.


Retinal movies are Marty’s primary source of evidence. He uses such evidence, along with various assumptions and background knowledge about how the world works, to generate hypotheses about the source of the observed projections (that is, he makes an inference about the source of the stimuli). We are only conscious of the outputs of Marty’s vocation, not any details of his inference-generating procedures. Hence the hypothesis that perception is unconscious inference.

Hypothesis formation is a special case of inference. It is a type of inference that doesn't enjoy the level of certainty granted to deductive inferences (as you’d find in mathematical proofs). Rather, when we form a hypothesis we are often throwing out our best hunch, an educated guess based on limited evidence and previous assumptions about the way the world works. Philosophers sometimes call this type of inference an ‘inference to the best explanation’ or ‘abductive inference.’

For instance, say the evidence we wish to explain includes late-night scratching sounds in the cupboard and small fecal nuggets deposited in the pantry. We could use such evidence, and our general understanding of how the world works (mice are nocturnal, etc), to construct a hypothesis that would best explain the evidence. In this case, we would likely hypothesize that there are mice living in our kitchen. Perhaps Marty settles on his hypotheses about visual stimuli using a similar process of abductive reasoning.

Obviously, Marty is merely a useful fiction. Nobody thinks there is literally a little man in your head viewing your retinal movies. Advocates of this theory believe that we will ultimately be able to give a more literal story that describes how brains construct hypotheses based on information coming in from the retinae. In the meantime, I should spell out why the unconscious inference theory appeals to psychologists.

The appeal of the theory
There are three main reasons for the theory’s appeal (aside from its impressive intellectual pedigree since Helmholtz (1866)). For one, the theory would explain how certain illusions are generated. For instance, recall Shepard’s Monsters (Figure 2) from post eleven. Shepard explains the illusion as follows:

[T]he linear perspective of the subterranean tunnel (along with other depth cues, such as the relative heights of the projections of the two monsters on our retinas) supports the automatic perceptual inference that one of the two monsters is farther back in depth. The two monsters, nevertheless being exactly the same size in the drawing, subtend the same visual angle at the eye [i.e., their projections occupy the same surface area on the retinae]. The visual system therefore makes the additional inference that in order to subtend the same visual angle, the monster that is farther back in depth must also be larger.

Notice how the idea of inference-making is built into multiple layers of Shepard’s explanation of the illusion. The brain makes inferences about which monster is further away, and then uses this information to make further inferences about which monster is larger, which explains why one monster looks bigger than the other. Note the claim isn’t that the brain only uses inferences in cases of illusions (how would the brain know if it were seeing an illusion or not?), but that illusions help reveal the underlying inferential machinery of normal perception.

Figure 2: Terra Subterranea, or Shepard’s Monsters


A second appeal is that the theory finds a mathematical home in probability theory and statistics. The brain lives in an uncertain world, and even the brain’s own responses to identical stimuli are not the same every single time (that is, the brain itself is a “noisy” processor). In mathematics, the principles of sound inference in such uncertain contexts are provided by statistics. Couching theories of brain function in the language of probability and statistics allows psychologists to state their theories with more rigor than can be done in ordinary language. Perhaps most importantly, such theories allow them to generate quantitative predictions that can be tested against the data.

Figure 3: The eye lives between a noisy brain and an uncertain world.


The third appeal applies to the ‘unconscious’ side of the ‘unconscious inference’ thesis. That is, it seems pretty clear that the processes which generate our perceptual experiences are not consciously accessible to us (as discussed in post ten and post thirteen).

Hopefully this gloss on the unconscious inference theory of perception was half as fair as it was brief. At this point I don’t want to push too hard against it (for instance, you would be right to ask what it means for the brain to perform an inference). Rather, my goal was to showcase the most prominent species of the perception-as-interpretation thesis. More than one-hundred years after Helmholtz initially suggested the hypothesis that perception is unconscious inference, Fodor and Pylyshyn were able to describe the theory, without much overstatement, as the ‘Establishment theory' of perception.

Representations within interpretations
Enough with unconscious inferences: what about all the other theories I mentioned above, such as the view that the brain builds a ‘simulation’ of the world? I am going to avoid jumping down the historical rabbit hole of comparing/contrasting the often subtle differences in this panoply of psychological-level theories of perception. Rather, it will be more productive to extract a common denominator shared by all of these theories, something all of the advocates would agree upon. If such a common factor turns out to be useful and correct, then great. If not, then we will have eliminated an entire class of models of perception with one parsimonious swing of the blade. This seems much easier than starting by contrasting every such theory pair in detail.

The one theoretical commitment shared by all these theories of perception is that the brain constructs representations of the world, and the contents of such neuronal representations are the contents of experience. Our first priority will be to analyze this idea of neuronal ‘representation’: what the heck does it mean, and how far can it take us in our quest to understand visual experiences?

While I won’t analyze the notion yet, the notion of a ‘representation’ should be intuitively familiar to most of us. Three squiggly lines on a map represent water. A photograph of someone represents the person. I’ve already sneaked in the claim that the brain constructs a ‘portrait’ of the world: a portrait of something is one type of representation. The claim we will evaluate is that one component of the brain’s interpretation of a stimulus is an internal representation of the world constructed partly based on that stimulus.

Before heading into brains, however, in the next post I will finish this chapter by posting a broad range of quotations from the literature on the topics we have explored in the last eight posts. This will help us to see how these ideas of interpretation, simulation, representation, etc are used in practice.

References
Fodor, JA, and Pylyshyn ZW (1981) How direct is visual perception? Cognition 9: 139-196.

Helmholtz, H. von 1866 Concerning the perceptions in general. In Treatise on physiological optics, vol. III, 3rd edn (translated by J. P. C. Southall 1925 Opt. Soc. Am. Section 26, reprinted New York: Dover, 1962).

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

-------------------------------------
Table of Contents of my posts on consciousness.

How to run R code in Matlab

R (site here) is a great open-source environment for statistical analysis. But I'm a Matlab user. Luckily, it is pretty easy to run R code from Matlab. Since I just set it up in my Matlab environment, I thought I'd write out the recipe I followed. I have only done the following in Windows XP, and I used Matlab version 7.8. I think it will only work in Windows. It assumes you already have R and Matlab properly installed on your computer.

Of course, this doesn't mean I don't have to learn how to use R, it just means I get to do it all in Matlab (and note for fellow Matlab users, there is a great cheat sheet that shows how to translate between the two).

1. Install the R package rscproxy.
In R, enter:
>install.packages("rscproxy")
to install the package.

2. Install the R(D)Com server.
Download it here. The server allows Matlab to talk with R. I installed it using the default settings without checking or unchecking any boxes. Note this server is built for Scilab, which is an open source version of Matlab, but it seems to work for Matlab too.

3. Download the Matlab R-Link toolbox
Get MATLAB_RLINK.zip here, unzip the contents, and paste MATLAB_RLINK in Matlab's toolbox folder (or whatever folder you want). Be sure to add MATLAB_RLINK to your Matlab path.

4. Restart your computer.

5. Is it working?
To see if the toolbox is working, start Matlab and enter 'Rdemo' at the command prompt. This should evoke:
b =
1 4 9 16 25 36 49 64 81 100

c =
2 5 10 17 26 37 50 65 82 101

6. Have fun!
If Rdemo worked, you are ready to go!

For instance, enter the following in Matlab:
openR; %Open connection to R server
x=[1:50]; %create x values in Matlab
putRdata('x',x); %put data into R workspace
evalR('y<-sqrt(x)'); %evaluate in R
evalR('plot(x,y)') %plot in R
To close the connection to R, and the graphs opened from R, enter:
closeR;

7. Problems?
If the above doesn't work, go to C:\Program Files\R, open the (D)COM Server folder, go to 'bin', copy 'sciproxy.dll', and paste it in C:\Program Files\MATLAB\R2009a\bin (obviously you may have a different path to Matlab's binary folder). Close Matlab, and restart your computer.

If that doesn't help, I probably won't be able to help, but go ahead and ask as someone might know. The site where you downloaded R-Matlab has some useful Q&A so you might inquire there.

8. Acknowledgments
This is basically an updated version of Kevin Murphy's site. Please let me know if anything here becomes obsolete.

9. Caveat (added 6/18/12)

From the comments section:

After using R(D)COM and Matlab R-link for a while, I do not recommend it. The COM interface has trouble parsing many commands and it is difficult to debug the code. I recommend using a system command from Matlab as described in the R Wiki. This also avoids having to install all of the RAndFriends programs. 

Consciousness (13): The Interpreter versus the Scribe

Number thirteen in my series of posts on consciousness. Table of Contents is here.
---------------------------

While elaborating on the parallels between perception and language interpretation, we have unpacked many features of the nature of visual perception that should hold up even if we end up finding the view of perception-as-interpretation wanting. In this post I’ll briefly integrate the data and theory from the past seven posts into a more tidy and (hopefully) coherent story.

As we discussed in some detail in post ten, the contents of experience have properties that are, on the surface, quite different from the properties of the underlying neural machinery doing the experiencing. I can see an iridescent jewel two feet in front of me (that’s the content), but the vehicle doing the experiencing is neither iridescent nor two feet in front of me.

We can be intimately familiar with the contents of our experience while remaining in complete ignorance of facts about nervous systems. I hope I don’t offend my fellow neuroscientists when I claim that our species’ great artists, playwrights, musicians, and novelists have revealed more about the contents of experience than any neuroscientist. Yet most of these artists worked without knowing the most basic facts of neuroscience. The vehicles of experience are effectively invisible to us, while the contents of experience are as familiar as breathing. Anyone that has savored an authentic lobster roll from a rundown shack on the coast of Maine knows what it is like to revel in the contents of experience (and those who have not have yet to fully live).

In sum, the contents of our experience seem to be a neurally-constructed portrait of what is happening beyond the brain. The brain faces some rather severe obstacles if its goal is to make this portrait accurate. For one, a great deal of information is lost in the projection from the scene to the retina (a projection we discussed in some detail in post nine).

Consider the case in which a projection onto the retina is square-shaped. What can we say about the object that generated that projection? Assuming there are no distance cues present, the same square shape on the retina could be produced by a tiny square that is extremely close to the eye, a medium-sized square a moderate distance away, or a colossal square that is extremely far away. It could even be generated by non-square shapes transmitted through a distorting funhouse-type medium.

Purves and Lotto state the point nicely:

[T]he retinal output in response to a given stimulus can signify any of an infinite combination of illuminants, reflectances, transmittances, sizes, distances, and orientations in the real world. It is thus impossible to derive by a process of logic the combination of these factors that actually generated the stimulus[.]

In other words, given only retinal movies as data, the brain cannot determine with perfect accuracy the scene in the world that generated said movies. Given the often striking ambiguity of the source of a retinal projection, it is remarkable that our visual system usually locks in on a single perceptual response to a given stimulus. Even during bistable perception we typically experience one object at a time, not a superposition of two objects.

How does the brain settle on a unique percept when provided with an inherently ambiguous retinal projection? It seems the brain uses context (post eleven) as well as background assumptions and knowledge (post twelve) to help narrow down the range of reasonable interpretations. Bistable percepts merely serve to highlight those rare instances when these contributions from the brain are not sufficient to settle on a single interpretation for an extended period of time.

In general, while we know that the retinal movies strongly influence the brain’s construction of experience, our experience is obviously not a mere report or transcription of what is happening in the retinae. If it were, ambiguous stimuli wouldn’t spontaneously reorganize in such drastic ways such as we observe in the Spinning Girl and Rotating Necker Cube (post seven), the angles in Purves’ Plumbing would look the same, the tabletop dimensions in Turning the Tables would look identical (post twelve), the yellow and blue squares in Purves’ Cubes would look grey, Shepard's subterranean monsters would look identical in size (post eleven), etc..

Hopefully the previous seven posts have made it clear why psychologists often say that the brain constructs interpretations of stimuli in a context-sensitive way, based on background knowledge and assumptions, in the light of sometimes intense ambiguity of the actual source of the stimulus. If we were forced to choose between the false dichotomy of saying that experience is an interpretation of what is happening in the retinae versus a transcription of what is happening on the retinae, I think the choice is clear.

Richard Gregory (1966) summed up the view quite well when he said that “Perception is not determined simply by the stimulus patterns; rather it is a dynamic searching for the best interpretation of the available data.” Our visual experience is clearly the result of neuronal events downstream from the stimulus, a construction of an experience whose contents mostly include worldly events beyond the eyes. It is such worldly events that we must engage with, after all, and such engagement with the world determines whether we eat, reproduce, flee, or die.

Next up
In the next couple of posts we’ll persue the idea that perception is interpretation down more specific paths, looking at a prominent view that the mechanism of interpretation is a kind of unconscious inference, and finally we’ll end up heading into the brain, looking at the neuronal basis of these internal “portraits” of the world.

References
Gregory, RL (1966). Eve and Brain. London: Weidenfeld and Nicolson.

Purves, DP, and Lotto, RB (2003) Why we see what we do: An empirical theory of vision Sinauer Associates.


-------------------------------------
Table of Contents of posts on consciousness.

Consciousness (12): What the brain thinks it knows

Number twelve in my series of posts on consciousness. Table of Contents is here.
---------------------------

In this continuation of the previous two posts, let’s finish examining the parallels between linguistic interpretation and visual perception. Recall the first three similarities were: 1) applicability of the content/vehicle distinction, 2) the possibility of ambiguity, and 3) sensitivity to context. In this post we'll add one final point to the list.

4. Use of background knowledge/assumptions
As we discussed previously (post eight), interpreting a sentence draws on your background knowledge about the world. A sentence whose meaning is transparent to you (e.g., ‘George Washington was the first President of the US’) will be perplexing to someone who doesn’t share your knowledge of American history.

For another example, if someone were to utter ‘John was told what to do by the river,’ we (often implicitly) use our knowledge that rivers cannot talk to home in on the correct interpretation. If we were to replace 'river' with 'teacher' we would not be able to use our knowledge to determine the correct interpretation.

Even fallible assumptions about the structure of the world can bias us toward interpreting a sentence a certain way. For instance, we can modify a semantically "bistable" sentence from post eleven ('I saw the man with the telescope') in ways that will make folks much more likely to settle on one of the interpretations:

[1] The astronomer stared at the girls with his telescope.

While [1] is technically still ambiguous, the modifications make us much more likely to interpret the sentence as describing someone using his telescope to spy on some girls.

Our brain also seems to use background knowledge and assumptions about the world when generating its perceptual response to a stimulus. As an example, let’s consider the brain’s apparent knowledge of perspective.

Perspective refers to the depth cues that result when a scene is projected onto a flat sheet located at a particular location (such as a retina, as described in post nine). One such cue is linear perspective, which refers to the fact that a projection of two parallel lines that recede into the distance will converge to a common point rather than remain parallel. This can be seen in any stock image of railroad tracks, as in Figure 1. While we know the tracks are actually parallel to one another, when they project to a flat surface (such as a retina or the plate of sensors in your digital camera), they appear to converge.

Figure 1: Railroad tracks illustrating linear perspective.


The brain seems to "understand" linear perspective, as evidenced by the powerful experience of depth that results when artists recreate its effects on a two-dimensional sheet of paper or canvas. There are artists that, armed with nothing more than wits and a few pieces of chalk, are able to create a vivid impression of a rich three-dimensional scene on a sidewalk (Figure 2) or even an entire street (Figure 3). You can see the convergence of lines in these works, the use of linear perspective that helps create a striking impression of depth.

Figure 2: Taking the Plunge, a sidewalk chalk drawing by Julian Beever.


Figure 3:Lava Burst, as painted on the streets of Gelden, Germany.


Because our brain evolved partly to help us navigate three-dimensional terrain, it isn’t too surprising that it is so keyed in to such cues. Even when we know at a cognitive level that we are looking at a two-dimensional surface, our brains reflexively construct an experience of depth. The great psychologist Gordon Shepard, in his inimitable style, explains things as follows:

[W]e cannot choose to see a drawing merely as what it is—a pattern of lines on a flat, two dimensional surface. To the extent that that pattern of lines conforms with the rules of linear perspective, for example, that pattern automatically triggers the circuits in the brain that make the three-dimensional interpretation appropriate to such a perspective display. Any consciously adopted intentions to ignore such an interpretation are largely powerless against the swift deliverances of this underlying machinery. This should not surprise us. We have inherited this machinery from individuals who, long before the advent of picture making, interpreted—by virtue of this machinery--what was going on in the three-dimensional world around them with sufficient efficiency to survive and to continue our ancestral line.

In the normal world of perception in a three-dimensional world, such perspective cues are quite reliable indicators of the spatial structure of the world. Clever artists (including 3D cinematographers) and psychologists merely exploit the neuronal circuits which unquestionably believe such cues even in ethologically peculiar contexts.

Let’s finish by looking at a couple of rather stunning visual illusions psychologists have created to exploit the brain’s knowledge of perspective. First, consider the two tables in Figure 4. The tops of the two tables are actually the exact same size! If you were to rotate the left table top by 90 degrees and move it to the right, the two tabletops would overlap perfectly. Especially surprising is that the long edge of the table on the left (the edge that appears to be receding away from the viewer) is the same length as the front edge of the table on the right (the edge roughly parallel to the viewer).

Figure 4: Turning the tables.


It’s as if an understanding of the physics of projection is hard-wired into our visual system, and our brain automatically makes adjustments when it "thinks" an object’s size has been contracted because of viewing angle. Shepard says, "The fact that the retinal images of the two quadrilaterals interpreted as table tops are identical in length then implies that the real length of the table going back in depth must be greater than the real length of the crosswise table." In other words, because the table on the left is (apparently) receding into the distance, it must actually be longer than the table on the right, which runs parallel to the viewer.

Such compensation by our brain for such perspective-dependent distortions can also explain why an object such as a coin appears circular even when presented at an angle so it actually projects an oval to the retina.

A related illusion is shown in Figure 5, which appears to be a sort of display case for four pieces of plumbing. Each piece consists of two tubes connected in the middle by a ball. While the angles between the tubes looks quite different, they are actually all identical, as demonstrated in the bottom panel of the figure.

Figure 5: Purves’ plumbing.

Figure 5 contains strong cues about the spatial arrangement of the pieces relative to the viewer. For instance, the tubes in the red structure span from the back of the scene to the front, which would only be possible if the two tubes subtended an obtuse angle. While the structure happens to be projecting a right angle to the retina, this is because of its contingent orientation with respect to the viewer. Similarly, the angle between the tubes in the green piece clearly seems to be acute, even though it projects an identical 90 degree angle to our retina.

Again we find that the brain seems to compensate for the distorting filter of perspective, generating an experience to conform more closely with the actual angle than the stimulus (i.e., the angle projected to the retina).

Where next?
That's the last parallel I'll explore between language and perception. With that, we are ready to more closely evaluate the hypothesis that perception is a form of stimulus interpretation. We'll do that in the next post.

Sources of examples
‘John was told what to do by the river’ is from Norvig (1988), an article which also inspired sentence [1]. 'Taking the Plunge' was created by Julian Beever (http://users.skynet.be/J.Beever/pave.htm). The street carnage 'Lava Burst' was drawn by Edgar Mueller (http://www.metanamorph.com/). 'Turning the Tables' is from Shepard (1991). Purves' Plumbing is from Purves and Lotto (2003).

References
Norvig, P (1988) Multiple Simultaneous Interpretations of Ambiguous Sentences. Proceedings of the 10th Annual Conference of the Cognitive Science Society.

Purves, DP, and Lotto, RB (2003) Why we see what we do: An empirical theory of vision Sinauer Associates.

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

-------------------------------------
Table of Contents of posts on consciousness.

Consciousness (11): Ambiguity and Context in Perception and Language

Eleventh in my series of posts on consciousness. Table of Contents is here.
--------------------------------------------------

In this sequel to the previous post, we continue to examine the similarities between linguistic interpretation and perception, and explore the hypothesis that perception is stimulus interpretation.

2. Bistability
Just as we can experience bistable visual percepts, sentences can exhibit a “semantic bistability” in which there exist two equally reasonable interpretations. For example:

[1] Adam asked him to use the bathroom.

What is the meaning of [1]? Did Adam request that someone else go to the bathroom, or did Adam wish to use the bathroom?

[2] I saw the man with the telescope.

Who has the telescope, me or the man? There isn't enough information in sentences [1] or [2] to determine which interpretation is correct.

Similarly, ambiguous visual stimuli (discussed here and here) such as the Necker Cube don’t provide enough disambiguating information for our visual system to settle on one perceptual interpretation. Koch (2004) says:

Consider the twelve lines making up the Necker Cube. Due to the inherent ambiguity of inferring its three-dimensional shape from a two-dimensional drawing, the lines of the cube can be interpreted two ways, differing only in their orientation in space. Without perspective and shading cues, you are as likely to see one as the other. The physical stimulus--the line drawing--doesn't change, yet conscious perception flips back and forth between these two interpretations, in what is a paradigmatic example of a bistable percept.

Koch is claiming that because the visual stimulus does not adequately disambiguate the source object (i.e., the line drawing is ambiguous), our conscious experience flips back and forth between two plausible interpretations of the image.

Both types of bistability (i.e., linguistic and stimulus) can easily lead to errors. For instance, someone may understandably interpret sentence [2] to mean you were spying on someone, when in fact you really meant to say you innocently observed a guy that was carrying a telescope around. In the case of vision, if you see an actual wire frame of a cube (a real-life Necker Cube) your visual system will lock in on one or the other percept with about equal probability, and half the time it will be wrong.

3. Context dependence
The context in which a sentence is uttered can have strong effects on how we will interpret it. For example, in a conversation about money, you will likely interpret ‘I went to the bank’ differently than in a conversation about a canoe trip down a river.

Likewise, the visual experience produced by a localized stimulus depends strongly on the context of concurrent surrounding stimuli. For example, the top panels of Figure 1 show two cubes that are tiled with colored squares (a Rubik’s Cube sort of situation). The cubes seem to be placed in different sources of illumination: the cube on the left seems to be in yellow light, and the right cube in blue light. Remarkably, the blue tiles on top of the left cube are identical to the yellow tiles on top of the right cube. If you look at them in isolation, these tiles actually appear gray, as shown in the bottom half of Figure 1. Changing the visual context of these gray squares radically changes how we experience them.

Figure 1: Context dramatically affects color perception.


Another example of contextual influences on visual perception is shown in Figure 2, which depicts one monster chasing another through a sewer. While the monster chasing the other looks appreciably bigger, the two monsters are actually identical copies of one another, and project the same image to the retina. The context of the receding tunnel makes the monster on top seem much further away, and our brain somehow magnifies its apparent size based on such depth cues.

Figure 2: Big meanie chasing terrified rascal.


These examples demonstrate that our experience of something at a localized region of space is based partly on what is happening at other locations in our visual field. Our brain is quite sensitive to contextual factors in its construction of a percept.

In the next post we will finish this foray comparing perception and interpretation.

Sources of examples
The analogy between perceptual bistability and sentence ambiguity has been noted before. Norvig (1988) points to Hockett (1954) as the first. Hockett wrote, "The hearer, confronted with The old men and women stayed at home, is in much the same position as the observer who sees a picture of a hollow cube and can, almost at will, see first one corner and then another as closer to him." The sentence mentioned by Hockett can either mean that the old men and old women stayed, or that the women and the old men stayed.

Sentence [1] is from John Limber (personal communication), whose article Syntax and sentence interpretation (1976) has clear influences on this post. Sentence [2] seems to be ubiquitous in the linguistics literature, and its origins are opaque to me. I saw a reference to it in a 1961 memorandum from the RAND corporation, and I am trying to track it down. Figure 1 is from Shepard (1991). Figure 2 is from Purves and Lotto (2003) (there are many exceptional illusions at Purves' web site).

References
Hockett, CF (1954) Two models of grammatical description, Word, 386-399.

Koch, C (2004) The Quest for Consciousness: A neurobiological approach Roberts & Company Publishers.

Limber, J. (1976) Syntax and sentence interpretation, In R. Wales & E. C. T. Walker (Eds.), New
approaches to language mechanisms (pp. 151-181). Amsterdam: North Holland.

Norvig, P (1988) Multiple Simultaneous Interpretations of Ambiguous Sentences. Proceedings of the 10th Annual Conference of the Cognitive Science Society.

Purves, DP, and Lotto, RB (2003) Why we see what we do: An empirical theory of vision Sinauer Associates.

Shepard, RN (1991) Mind Sights, W.H.Freeman & Co Ltd.

-------------------------------------
Table of Contents of posts on consciousness.

Consciousness (10): Contents and Vehicles

Number ten in my series of posts on consciousness. Table of Contents is here.
------------------------------------

There is a natural tendency to describe bistable perception as the experience of competing interpretations of a stimulus. This tendency can be partly explained by the many similarities between linguistic interpretation and visual perception. Let’s consider three such similarities in this and the next post.

1. Applicability of a content/vehicle distinction
The first similarity is that a content/vehicle distinction (to be defined shortly) applies to language and to conscious experience. Let's take a look at each case.

Content and vehicle in language
As we discussed previously, to interpret a sentence is to determine what it means. Consider the following sentence:

[1]Rattlesnake bites are poisonous.

Imagine if a child asked an adult the meaning of [1], and the adult responded with ‘It is a four-word sentence written in black 10-point Times New Roman font.’ The child would be right to get annoyed.

What would a more appropriate answer look like? Minimally, interpreting [1] requires determining what the sentence expresses about the world outside of language. For instance, it tells us that a certain type of snake’s bite will harm humans. Similarly, the sentence ‘Fred got married’ tells you that something happened out in the real world, that some guy named Fred got married. To focus on the font size or color of the writing used to communicate the information would be to miss the point.

This distinction between the physical/structural features of a sentence, and the meaning it expresses, is well-known to everyone. It is often described by philosophers as the distinction between an expression’s content or meaning and the vehicle or medium of expression.

In general, the vehicle-properties of a sentence are different from its content-properties. Its vehicle-properties include physical features of the individual letters (e.g., their shape, properties of the ink, and the material on which they are presented) as well as structural features of the sentence as a whole (e.g., how many words it contains). The content-properties of a sentence, on the other hand, include those extralinguistic facts that the sentence tells you about. For instance, [1] tells you that rattlesnakes can bite, and that such bites are dangerous to people. When interpreting a sentence, we focus on such content-level properties, what is expressed about the world, and it is usually confused to focus on the vehicle properties.

The content/vehicle distinction also applies to individual words. The word ‘ice’ has certain vehicle-properties (it is three letters long, is written in a certain font, etc), while at the level of content it refers to the solid phase of water, out in the real world. The differences between the vehicle-properties and content-properties of a word are legend. The word ‘monosyllabic’ is not monosyllabic (also consider the word ‘palindrome’). See also Figure 1.

Figure 1: Sample words in which
content and vehicle disagree.


While the content/vehicle distinction isn’t the topic of polite conversation, it is blithely exploited by everyone that uses language. To talk with one another, we must be able to see past the vehicles of communication and respond to the content being expressed. If someone says ‘I got a puppy,’ we respond by asking more questions about their dog such as where they got it; we don’t focus on the vehicle-properties of the sentence. Indeed, if you were to systematically focus on the vehicle-properties of what people said, they would quickly lose all interest in talking to you (which is the fate of those deranged enough to focus incessantly on other people’s grammar or spelling). Such behavior indicates your interpreter is malfunctioning.

Every time we assess the truth or falsity of a declarative statement, we are implicitly examining whether the content of what someone says matches up to reality. For instance, ‘Fred did his math homework’ is true if Fred in fact did his math homework. It is false if he did not. Sentence [1] is true because it expresses something that indeed holds of rattlesnakes in the real world. We don’t determine if a claim is true or false by measuring the color of the font used to express it or the mean intensity of the sound waves when it was uttered. ‘Whales are fish’ is false not because it is written in Times New Roman font, but because whales aren’t fish.

Even the childhood rhyme ‘Sticks and stones’ plays on the content/vehicle distinction in a fun if oppressive manner. Sure, linguistic vehicles don’t harm your body (billboards notwithstanding), but obviously it’s the content of what is said that inflicts psychic pain when you are insulted.

Content and vehicle in experience
It seems a content/vehicle distinction also applies to conscious experience. While the brain is the organ (i.e., the vehicle) of conscious experience, what we actually experience (i.e., the contents of our experience) doesn’t seem neuronal at all. Indeed, we constantly experience things going on outside of our brains, things like seeing an ice cube out there, three feet in front of us; feeling a sharp pain in our toe, way down at our feet; hearing that song we like on the radio. All the while, the brain doing the experiencing, the vehicle that mediates such experiences, is locked up inside our skull.

These examples indicate that, as in language, the contents and vehicles of consciousness generally have quite different properties. When we experience an ice cube three feet in front of us, we don’t expect someone to find a literal cube of ice in our brains.

Daniel Dennett’s delightful essay Where am I? served to brand into my brain the stark divergence between the contents and vehicles of experience. Therein, Dennett describes how, through the wonders of neuroengineering, his brain was extracted from his skull and kept alive in a tank of cerebrospinal fluid (see Figure 2). Dennett’s brain was connected to his body through various transmitters and receivers (note the router attached to his brain in Figure 2), so his disembrained body could still get about normally. The signals from his optic nerves were transmitted to the brain so he could still see the world. His body could still move around because the motor commands produced by his brain were transmitted to microstimulators in his spinal cord.

Dennett then described what it was like the first time he visited his own brain (Figure 2):

I peered through the glass. There, floating in what looked like ginger ale, was undeniably a human brain, though it was almost covered with printed circuit chips, plastic tubules, electrodes, and other paraphernalia…I thought to myself: “Well, here I am sitting on a folding chair, staring through a piece of plate glass at my own brain . . . But wait,” I said to myself, “shouldn't I have thought, ‘Here I am, suspended in a bubbling fluid, being stared at by my own eyes’?” I tried to think this latter thought. I tried to project it into the tank, offering it hopefully to my brain, but I failed to carry off the exercise with any conviction…[W]hen I thought “Here I am,” where the thought occurred to me was here, outside the vat, where I, Dennett, was standing staring at my brain.

Dennett’s thought experiment vividly illustrates the extent to which contents and vehicles of experience can diverge. While his brain is still in a tank at some undisclosed location, Dennett enjoys a rich and varied mental life to this day. Of course, we are in a similar predicament every time we dream.

Figure 2: Where is Dennett?


Just as the content of a sentence involves reference to things in the extralinguistic world, the content of our visual experience involves reference to things outside of our eyes and brains. The visual stimulus, a projection of the scene onto the retinal movie screen, triggers an avalanche of neuronal processes that ultimately produces an experience of what is happening out there beyond the brain.

Using a little poetic license, we can say that once a scene is projected onto the retina, our brain then projects a scene back out into the world. The contents of this outwardly projected scene are the contents of conscious experience.

Caveats and such
It would be contentious to claim that the content/vehicle distinction at play in language is identical to the distinction in experience. They may simply be two species in the same genus. Hence, without a lot more argument, we should be clear to distinguish perceptual content/vehicles and linguistic content/vehicles. Regardless of this caveat, the parallels between linguistic meaning and the contents of experience probably provide the perception-as-interpretation view with a good deal of its traction.

Conceptually, the content/vehicle distinction has probably been around since humans started to think about language. The terminology is relatively new, however, and is likely due to Dan Dennett, as I discussed here.

The obvious question this discussion brings up is, “How a brain can be anything but a vehicle? How can a brain state have content?” Neuroscience has a lot to say about this question, but let’s not get too ahead of ourselves. We are going to revisit the content/vehicle distinction many times, but for now let’s continue delineating the analogies between linguistic interpretation and perception. We'll look at two more in the next post.

References
Dennett, D (1978) Where am I? Chapter 17 in Brainstorms: Philosophical Essays on Mind and Psychology, Montgomery, VT: Bradford Books


-------------------------------------
Table of Contents of posts on consciousness.

Consciousness (9): From texts to the grotesque cinema

Number nine in my series of posts on consciousness. All the posts are indexed here.
---------------------

We are examining the popular view that visual perception is a form of interpretation, specifically the interpretation of a stimulus. We should start by determining what, exactly, is a stimulus?

So as not to keep you waiting, the answer is roughly that a visual stimulus is visible light that is projected from the world to the retina. If that is unclear, or if you are interested in the biology, then keep reading.

In general, a stimulus is anything that can activate our sensory transducers, the cells that convert external signals into internal electrochemical signals that can be used by the rest of the nervous system. In the case of vision the transducers are in the eye, so let’s consider the general relationship between the world, the eye, and visual stimuli.

Our world is filled with objects that project light into our eyes. Let’s denote the set of such objects at a given time the scene. For example, the eye in Figure 1 is inspecting a scene that consists of two objects: a cube and a house. The eye seems to be looking directly at the cube, while the house is up to the left of the hovering orb.

Figure 1: An eye viewing a scene.
The light from the scene is projected to the retina by the eye's lens (Figure 2, top). The retina is a thin red sheet of tissue that coats the inside of the eye, biology’s grotesque movie screen.

Photoreceptors are probably the most important cells in the retina: they convert light into chemical signals that the rest of the nervous system can understand. However, the retina is much more than a sheet of photoreceptors. It is an extremely complicated neuronal processor in its own right. The retina contains multiple layers of neurons (Figure 2, bottom), and it is only the axons of the final layer of neurons that make their way through the optic nerve toward the brain.

Figure 2: Gross anatomy of the eye (top), and
cross-section of the retina (bottom).
The retina includes a special region, the fovea, which has an extremely high density of photoreceptors. When we look about the world, we typically direct our eye so the fovea aims at the most interesting parts of the scene, and this lets us take in more information from those regions. It’s like having a video camera with extremely high definition in the middle of the screen, but as you move away from the middle the picture gets quite fuzzy. Note that in Figure 1, the retina is painted onto the back of the eye in red, but the high-def fovea is represented by a small yellow region in the center of the retina.

Imagine gently peeling the retina from the inside of the eye and laying it flat on the page. Figure 3 is a graphical depiction of such a flattened retina, drawn so that the spacing of the grid lines represents the density of photoreceptors. The fovea is in the center of the graph, with finely-spaced grid lines that tell us we are in the high-def region. The density of photoreceptors quickly decreases as you move away from the fovea, as indicated by the more coarsely-spaced lines.

Figure 3: The retina from a functional
point of view.
You can directly experience this drop in resolution by trying to read this text while looking off to the side of the page. Even though you can still make out the coarse features of the page (e.g., there are words and pictures), it is extremely difficult to discriminate the finer-grained spatial properties such as individual letters and words.

There is one more notable feature of Figure 3. Namely, the top half of the retina is labeled ‘Bottom’ while the bottom half is labeled ‘Top.’ ‘Left’ and ‘Right’ also seem to be reversed. These labels are not mistakes, but highlight that an inverted image of the scene projects to the retina. For instance, an object located above the eye’s fixation point, such as the house in Figure 1, is projected to the bottom half of the retina.

I included the geometrical rationale for such image inversion in Figure 1 above. The light from the house travels from above, down to the eye, and the rays of light continue traveling down within the eye until they hit the lower half of the retina. For similar reasons, objects to the left of fixation project to the right-hand side of the retina.

All of this inversion business is illustrated in Figure 4, which shows the house/cube scene projected onto the retina. As we would expect, the fixated cube-face is projected squarely to the fovea. The image of the house, which arrives from the top-left part of the world, is projected to the lower-right quadrant of the retina. That is, the house projects to the quadrant labeled ‘Top Left,’ so-called because the image is projected from the top-left hand region of the world. Note that even the image of the house is inverted, so an upside-down house is projected onto the retina.

Figure 4: The scene from Figure 1 projected
onto the retinal graph from Figure 3. Note how
the projections are inverted.
Finally I think we have arrived at a respectable-enough understanding of visual stimuli. A visual stimulus consists of the images projected from the scene onto the retina. We should stipulate that only light that is able to activate the retina’s photoreceptors counts as a visual stimulus. Not all electromagnetic radiation (i.e., light) can activate photoreceptors. Photoreceptors are tiny light detectors that are sensitive to light within a certain narrow band of wavelengths, as shown in Figure 5. So, absorbing this wrinkle into our account of visual stimuli, and using the plural of ‘retina’ because we have two eyes, yields:

A visual stimulus is light, within the visible part of the electromagnetic spectrum, that is projected from the scene to the retinae.

So, with that definition, I think we have a respectable characterization of visual stimuli.

There is one practical detail I should add before closing this discussion. Because vision is a distal sense that involves the perception of things far away from our bodies, in practice there is quite a bit of flexibility in what researchers count as a stimulus. They will sometimes identify the stimulus as the object itself out in the world (e.g., the Necker Cube drawing), the light emitted from the object, or the light from the object that hits the surface of the eye. In practice the things that researchers count as visual stimuli depend on the question being asked and the experimental setup. In the future I may call any of the above ‘stimuli’ when it seems appropriate, though typically I use the word to refer to the image projected to the retina.

Figure 5: The electromagnetic spectrum,
with the visible spectrum as a subset.


-------------------------------------
Table of Contents of posts on consciousness.

I'm still stalking Mr B

I haven't forgotten that I owe further posts in the consciousness series. I've been working at them, but have so far held off on posting until their organization was clear in my mind.

The material is finally coming together in a way I like. Upshot? I have at least six posts in the queue, but am still organizing them and polishing them. I expect to have the first out next week and then they will come fairly quickly.

Admittedly, the material is exploding in my hands, so I plan to turn this entire project of stalking Mr B into a book proposal. Frankly, though, his name may need to be changed, as there is almost as much psychology as biology in the stuff I'm writing.