Phillips Collection, Washington, D.C./Bridgeman Art Library

Pierre-Auguste Renoir: Luncheon of the Boating Party, 1880–1881

The worlds we see, hear, feel, smell, and taste exist independently, but we know them only through the fabrications of our brains. The colors we see do not exist apart from our perception of them. The words and sentences we believe we are hearing are a jumble of sounds, whistles, grunts, and silences. From a variety of external signals our brains create something that is not there. In doing so, they help us understand and manipulate our environments. Our sensory worlds—vision, hearing, and touch—are created by combinations of physical characteristics in our environments that stimulate our eyes and ears and skin surfaces. These combinations simplify and stabilize our sensory worlds.

Wavelengths of light or varying pressures of sound waves have very definite physical characteristics that can be measured. We know that our retinas are sensitive to different wavelengths of light and that the spiral-shaped parts of the inner ear, called cochleas, are sensitive to different frequencies of sound.

Gordon Shepherd’s stimulating and informative new book, Neurogastronomy, describes how the brain creates our sensations of smell and taste. Unlike our other sensory experiences—seeing, hearing, touching—the sensory receptors responsible for the brain’s creation of smells and tastes do not react to specific forms, such as the objects and paintings that stimulate our visual system or the waves of sound that stimulate our auditory receptors.

Our sense of touch, for instance, relies on input from nerve cells located in the skin that sense everything from pain to temperature. Yet the process of flavor perception is multisensory and interactive. As Shepherd explains, “A common misconception is that the foods contain the flavors. Foods do contain the flavor molecules, but the flavors of those molecules are actually created by our brains.” Analogizing a flavorful food’s flavor to a colorful object’s color, he goes on:

Color arises as differences in wavelengths of light given off by an object; our brains transform those wavelengths into color to give it meaning for our behavior. Similarly, the smells that dominate the sense of flavor arise as differences between molecules; our brains represent those differences as patterns and combine them with tastes [from the mouth] and other senses to create smells and flavors that have meaning for our perceptions of food.

How then does the brain make sense of the molecules in the foods we eat, or of different kinds of molecules randomly floating on the surface of the wine in a wine glass? How does the brain invent the tastes and odors of Bordeaux wines and Chanel perfume? How does it make combinations of chemicals into smell and taste?

First, the taste receptors on the tongue respond to a broad range of molecules, neatly categorizing them into five types: sweet, salty, sour, bitter, and umami (from the Japanese—a savory or meaty taste). The distinctive balance of these five characteristics makes up the “taste profile” of a particular item of food. The molecules the brain perceives as bitter bind specifically to bitter receptors on the tongue, the molecules perceived as salty bind specifically to salt receptors, and so on, much as keys fit into specific locks.

The five categories of taste and the sensations associated with them are innate sensibilities. When a newborn’s tongue is touched with a cotton swab dabbed in a solution of sugar, for instance, the infant smiles. When the cotton is dabbed in a salty solution the infant is indifferent; when it is dabbed in an acid solution, the infant puckers its lips in apparent disgust. The basic taste receptors are hardwired from birth.

However, since our sense of taste is limited to sweet, salty, sour, bitter, and umami, it cannot account for the sensory richness of fine wines and foods. In fact, when we taste wine, we are not simply perceiving taste, but a synthesis of odor and taste, which together create flavor. While there are only five types of taste receptors on the tongue, there are, in our nasal passages, about a thousand receptors for different types of odors. When we smell, the brain converts the responses of our individual odor receptors into a two-dimensional spatial pattern in our brain cells that Shepherd calls the “smell image.”

This process is similar to how the visual system of our brains works. The three-dimensional scenes we see as we walk in a park or ride a bicycle in the street become flat two-dimensional images on the retinas of our eyes. Our retinas then send the flat two-dimensional images to our brains, and these images are used to recognize and sometimes later recollect the people and places we have visited or met.

Normally, we are not conscious of this flat two-dimensional image. The brain’s visual system immediately uses it to form a three-dimensional image that we can recognize from virtually any angle. Some individuals have difficulty recognizing three-dimensional images—for example, faces, a disorder known as prosopagnosia. The artist Chuck Close cannot recognize and remember people. He overcomes this handicap using a procedure that is analogous to the brain’s creation of smell images. The portrait paintings and photography for which he is famous are his way of remembering faces:


I don’t know who anyone is and have essentially no memory at all for people in real space, but when I flatten them out in a photograph, I can commit that image to memory in a way; I have almost a kind of photographic memory for flat stuff.

What makes Close’s greatly enlarged faces memorable to him is their very flatness. By flattening and magnifying them, he simplifies the complexity of a three-dimensional image, making recognition possible.

The response of the smell receptors in our noses is similarly mapped into the two-dimensional smell image. Just as the brain compares our retinal images of the faces of people we have met to the two-dimensional retinal images in the brain, the brain remembers and recognizes smells by comparing the two-dimensional images created from the patterns of our nasal receptor responses when stimulated by new sensations of smell.

There are two ways that we “sniff.” We take in the “odors” of wines, flowers, perfumes, and smoke by inhaling air directly through our nasal passages. This is called orthonasal smell. But a second source of the smells we recognize comes from the back of the mouth. When we chew bread, meat, and other foods, the chewing releases molecules on our tongue surface and into the back of our nasal passages, and the resulting smell contributes significantly to the flavor we experience.

This is called retronasal smell. Unlike simple tastes, which are hardwired from birth, our responses to retronasal smells are learned. This is what accounts for individual preferences. Since eating things with noxious smells and tastes can kill animals and people if they are not detected rapidly, our taste and smell receptors, which warn us of noxious substances, send both taste responses and the two kinds of smell responses directly to the highest cortical centers of our brains without passing through any intermediary cortical areas. Visual, auditory, and touch stimuli, although important in warning of danger in the environment, take a slower path, and go first through a central relay station in the brain, called the thalamus, before arriving in the higher centers of the brain, such as the cortex.

The fusion of the sense of taste with smell creates the flavors of our foods and our drinks, yet the critical importance of smell in producing flavor is not usually recognized because we are not aware that it is the combination itself that creates flavor. Neither taste nor smell alone has the quality of flavor. In fact, the most important smells influencing flavor are retronasal, because retronasal “smell images” interact in the brain with a wide range of stimuli, including sound, touch, and the mechanisms used in chewing food. Flavor is largely a consequence of smell; but very few people are aware that flavor is an invention of the brain that arises from smell and taste, and not simply from taste.

Take a piece of candy, put it on the tip of your tongue, and pinch your nose. If you successfully block any air from entering your nasal passages, you will not notice that the piece of candy is sweet. If you then release your nose, letting air into your nostrils, you will suddenly “experience the flavor of the candy.” As Shepherd notes, this simple experiment shows that there is no flavor without smell. The ability to identify the flavors of lemons, bananas, and strawberries comes from our sense of smell.

The flavors of foods, especially those flavors arising from sugar, salt, and fat, are essential to our desire to keep eating. We can rapidly become accustomed to tastes, and our desire to continue eating, say, chocolate cake or raspberry ice cream can accordingly rapidly diminish. Producers of fast foods are well aware of the need for new tastes, and therefore they often change the flavors of the foods they sell to the public. The cortical centers that are responsible for our eating splurges are the very same centers responsible for the craving for drugs such as cocaine.

Our conscious perceptual worlds, then, are a consequence of the brain’s combinations of sensory information. The brain combines taste and smell to give us flavor, just as it combines the amount of light reflected in different wavelengths to give us color. Nonetheless, the nature of the syntheses can be altered by varying sensory inputs. For example, the combined effect of what we see and what we hear was vividly demonstrated in the 1970s when the cognitive psychologist Harry McGurk and his research assistant John MacDonald discovered that the particular sound one hears is altered by the shape of the lips being observed. If one observes and listens to a speaker saying “Da-Da,” one perceives “Da-Da”; if one only listens, one perceives “Ba-Ba”; if one only observes, one perceives “Ga-Ga.” What we “hear” speakers saying is altered if we observe the motion of their lips.


Similarly, in a famous experiment from 2001, wine tasters used very different terms to describe the flavor of a red wine as opposed to a white wine. The experimenters then colored the white wine with a tasteless red dye. When the tasters were then asked to describe the result, a panel of fifty-four undergraduates enrolled in the Faculty of Oenology at the University of Bordeaux—all of whom had much experience in tasting wine—described the artificially colored wine using the same terms they had previously used to describe the true red wine.

Apart from what this tells us about the role that visual stimuli can play in influencing our perceptions of flavor, the experiment also has implications for our use of language in describing, or characterizing, our experience of flavor. As the group of experimenters put it in a separate paper in the journal Brain and Language:

All wine descriptive language is in fact organized around wine types…. What a wine taster does in front of a wine is not an analysis of its separate sensory properties but a comparison of all the cognitive associations he or she has from the wine (color, initial aroma, and taste) with the impressions he or she has already experienced when tasting other wines.

There are many reports of people who can smell colors or taste sounds. Indeed, the sensory input from our ears and our olfactory systems converge in a part of the brain that is activated by sound and smell, the first step in sensory synthesis. The brain creates even more complex combinations that are at the heart of our conscious sensory experiences. It links our knowledge from our senses of taste and smell to our knowledge of where we are, using the knowledge we gain through the physical senses of touch, sound, and sight; and this synthesis is combined with our feelings and emotions, such as a sudden feeling of melancholy when we revisit a place where we suffered disappointment in the past. Indeed, the amygdala, which is the emotional center of the brain, is linked to the hippocampus, which records our spatial position. And these, as well as the olfactory cortex, are linked to the striatum, the brain region that measures the rewards we receive from specific actions in the form of feelings of pleasure and pain and uses these estimates to control our behavior—eating more, or less, for example. The result is a grand synthesis of what we are experiencing and how we feel about it—but a synthesis that is an “invention” of our brains.

Memory, and the idea of a past, a present, and a future, are also a creation of our brains. In his own way, Marcel Proust captures the synthetic quality of memory when he describes tasting his famous tea-soaked madeleine (causing a retronasal stimulation), which led to a series of childhood recollections. Just as flavor is a creation of the brain that is neither simply taste nor smell, but a synthesis primarily of both kinds of stimuli, Proust’s description of his recollection suggests (as do many studies of memory) that memories, emotions, tastes, and smells are the consequence of a broad synthesis of the sensory information the brain receives from our eyes, ears, touch, and movement receptors.

As Shepherd notes, though Proust claims that involuntary memories appear suddenly, he describes how his recollections of the village of Combray came to him gradually, after a prolonged conscious effort. Proust writes:

Undoubtedly what is thus palpitating in the depths of my being must be the image, the visual memory which, being linked to that of taste, is trying to follow it into my conscious mind…. Ten times over I must essay the task….

And suddenly the memory revealed itself.

The process of remembering is similar to processing a photograph of a face in a darkroom. The first hints of the image may not be recognizable, but little by little the face begins to appear on the photographic sheet. Suddenly we recognize the face that we see—though the face has yet to appear in its entirety. Studies of the neurophysiology of memory reveal a similar slow “development” of a recollection. We needn’t see the entirely developed image to recognize the face. The brain can construct the full image from incomplete sensory information.

Thus Combray is a construction, as are most memories—in Proust’s recollection, Shepherd writes, “a retronasal smell recalled a visual scene.” It all began with a sensation of the flavor of his madeleine dipped in tea.

Consciousness itself, like memory, is an integration of past and immediate sensory experience. We are conscious of neither the past as such nor the immediate present, but of a synthesis of past and present that, like the colors we see and the flavors we taste, is a creation of the brain. In this way, the brain stabilizes our perceptual (and mental) worlds, making them knowable and to some degree predictable.