Visualizing Visual Adaptation

Summary

This article describes a novel method for simulating and studying adaptation in the visual system.

Abstract

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical or age differences, or a color deficiency or disease. This protocol describes a technique for incorporating sensory adaptation into the simulations. The protocol is illustrated with the example of color vision, but is generally applicable to any form of visual adaptation. The protocol uses a simple model of human color vision based on standard and plausible assumptions about the retinal and cortical mechanisms encoding color and how these adjust their sensitivity to both the average color and range of color in the prevailing stimulus. The gains of the mechanisms are adapted so that their mean response under one context is equated for a different context. The simulations help reveal the theoretical limits of adaptation and generate "adapted images" that are optimally matched to a specific environment or observer. They also provide a common metric for exploring the effects of adaptation within different observers or different environments. Characterizing visual perception and performance with these images provides a novel tool for studying the functions and consequences of long-term adaptation in vision or other sensory systems.

Introduction

What might the world look like to others, or to ourselves as we change? Answers to these questions are fundamentally important for understanding the nature and mechanisms of perception and the consequences of both normal and clinical variations in sensory coding. A wide variety of techniques and approaches have been developed to simulate how images might appear to individuals with different visual sensitivities. For example, these include simulations of the colors that can be discriminated by different types of color deficiencies^1,2,3,4, the spatial and chromatic differences that can be resolved by infants or older observers^5,6,7,8,9, how images appear in peripheral vision¹⁰, and the consequences of optical errors or disease^11,12,13,14. They have also been applied to visualize the discriminations that are possible for other species^15,16,17. Typically, such simulations use measurements of the sensitivity losses in different populations to filter an image and thus reduce or remove the structure they have difficulty seeing. For instance, common forms of color blindness reflect a loss of one of the two photoreceptors sensitive to medium or long wavelengths, and images filtered to remove their signals typically appear devoid of "reddish-greenish" hues¹. Similarly, infants have poorer acuity, and thus the images processed for their reduced spatial sensitivity appear blurry⁵. These techniques provide invaluable illustrations of what one person can see that another may not. However, they do not — and often are not intended to — portray the actual perceptual experience of the observer, and in some cases may misrepresent the amount and types of information available to the observer.

This article describes a novel technique developed to simulate differences in visual experience which incorporates a fundamental characteristic of visual coding — adaptation^18,19. All sensory and motor systems continuously adjust to the context they are exposed to. A pungent odor in a room quickly fades, while vision accommodates to how bright or dim the room is. Importantly, these adjustments occur for almost any stimulus attribute, including "high-level" perceptions such as the characteristics of someone's face^20,21 or their voice^22,23, as well as calibrating the motor commands made when moving the eyes or reaching for an object^24,25. In fact, adaptation is likely an essential property of almost all neural processing. This paper illustrates how to incorporate these adaptation effects into simulations of the appearance of images, by basically "adapting the image" to predict how it would appear to a specific observer under a specific state of adaptation^26,27,28,29. Many factors can alter the sensitivity of an observer, but adaptation can often compensate for important aspects of these changes, so that the sensitivity losses are less conspicuous than would be predicted without assuming that the system adapts. Conversely, because adaptation adjusts sensitivity according to the current stimulus context, these adjustments are also important to incorporate for predicting how much perception might vary when the environment varies.

The following protocol illustrates the technique by adapting the color content of images. Color vision has the advantage that the initial neural stages of color coding are relatively well understood, as are the patterns of adaptation³⁰. The actual mechanisms and adjustments are complex and varied, but the main consequences of adaptation can be captured using a simple and conventional two-stage model (Figure 1a). In the first stage, color signals are initially encoded by three types of cone photoreceptors that are maximally sensitive to short, medium or long wavelengths (S, M, and L cones). In the second stage, the signals from different cones are combined within post-receptoral cells to form "color-opponent" channels that receive antagonistic inputs from the different cones (and thus convey "color" information), and "non-opponent" channels that sum together the cone inputs (thus coding "brightness" information). Adaptation occurs at both stages, and adjusts to two different aspects of the color — the mean (in the cones) and the variance (in post-receptoral channels)^30,31. The goal of the simulations is to apply these adjustments to the model mechanisms and then render the image from their adapted outputs.

The process of adapting images involves six primary components. These are 1) choosing the images; 2) choosing the format for the image spectra; 3) defining the change in color of the environment; 4) defining the change in the sensitivity of the observer; 5) using the program to create the adapted images; and 6) using the images to evaluate the consequences of the adaptation. The following considers each of these steps in detail. The basic model and mechanism responses are illustrated in Figure 1, while Figures 2 - 5 show examples of images rendered with the model.

Protocol

NOTE: The protocol illustrated uses a program that allows one to select images and then adapt them using options selected by different drop-down menus.

1. Select the Image to Adapt

1. Click on the image and browse for the filename of the image to work with. Observe the original image in the upper left pane.

2. Specify the Stimulus and the Observer

1. Click the "format" menu to choose how to represent the image and the observer.
2. Click on the "standard observer" option to model a standard or average observer adapting to a specific color distribution. In this case, use standard equations to convert the RGB values of the image to the cone sensitivities³².
3. Click on "individual observer" option to model the spectral sensitivities of a specific observer. Because these sensitivities are wavelength-dependent, the program converts the RGB values of the image into gun spectra by using the standard or measured emission spectra for the display.
4. Click on "natural spectra" option to approximate actual spectra in the world. This option converts the RGB values to spectra, for example by using standard basis functions³³ or Gaussian spectra³⁴ to approximate the corresponding spectrum for the image color.

3. Select the Adaptation Condition

1. Adapt either the same observer to different environments (e.g., to the colors of a forest vs. urban landscape), or different observers to the same environment (e.g., a normal vs. color deficient observer).
   1. In the former case, use the menus to select the environments. In the latter, use the menus to define the sensitivity of the observer.
2. To set the environments, select the "reference" and "test" environments from the dropdown menus. These control the two different states of adaptation by loading the mechanism responses for different environments.
   1. Choose the "reference" menu to control the starting environment. This is the environment the subject is adapted to while viewing the original image.
      NOTE: The choices shown have been precalculated for different environments. These were derived from measurements of the color gamuts for different collections of images. For example, one application examined how color perception might vary with changes in the seasons, by using calibrated images taken from the same location at different times²⁷. Another study, exploring how adaptation might affect color percepts across different locations, represented the locations by sampling images of different scene categories²⁹.
   2. Select the "user defined" environment to load the values for a custom environment. Observe a window to browse and select a particular file. To create these files for independent images, display each image to be included (as in step 1) and then click the "save image responses" button.
      NOTE: This will display a window where one can create or append to an excel file storing the responses to each image. To create a new file, enter the filename, or browse for an existing file. For existing files, the responses to the current image are added and the responses to all images automatically averaged. These averages are input for the reference environment when the file with the "user defined" option is selected.
   3. Select the "test" menu to access a list of environments for the image to be adjusted for. Select the "current image" option to use the mechanism responses for the displayed image.
      NOTE: This option assumes the subjects are adapting to the colors in the image that is currently being viewed. Otherwise select one of the precalculated environments or the "user defined" option to load the test environment.

4. Select the Spectral Sensitivity of the Observer

NOTE: For the adaptation effects of different environments, the observer will usually remain constant, and is set to the default "standard observer" with average spectral sensitivity. There are 3 menus for setting an individual spectral sensitivity, which control the amount of screening pigment or the spectral sensitivities of the observer.

1. Click on the "lens" menu to select the density of the lens pigment. The different options allow one to choose the density characteristic of different ages.
2. Click on the "macular" menu to similarly select the density of the macular pigment. Observe these options in terms of the peak density of the pigment.
3. Click on the "cones" menu to choose between observers with normal trichromacy or different types of anomalous trichromacy.
   NOTE: Based on the choices the program defines the cone spectral sensitivities of the observer and a set of 26 postreceptoral channels that linearly combine the cone signals to roughly uniformly sample different color and luminance combinations.

5. Adapt the Image

1. Click the "adapt" button.
   NOTE: This executes the code for calculating the responses of the cones and post-receptoral mechanisms to each pixel in the image. The response is scaled so that the mean response to the adapting color distribution equals the mean responses to the reference distribution, or so that the average response is the same for an individual or reference observer. The scaling is multiplicative to simulate von Kries adaptation³⁵. The new image is then rendered by summing the mechanism responses and converting back to RGB values for display. Details of the algorithm are given in ^26,27,28,29.
2. Observe three new images on the screen. These are labeled as 1) "unadapted" — how the test image should appear to someone fully adapted to the reference environment; 2) "cone adaptation"- this shows the image adjusted only for adaptation in the receptors; and 3) "full adaptation"- this shows the image predicted by complete adaptation to the change in the environment or the observer.
3. Click the "save images" button to save the three calculated-images. Observe a new window on the screen to browse for the folder and select the filename.

6. Evaluate the Consequences of the Adaptation

NOTE: The original reference and adapted images simulate how the same image should appear under the two states of modeled adaptation, and importantly, differ only because of the adaptation state. The differences in the images thus provide insight into consequences of the adaptation.

1. Visually look at the differences between the images.
   NOTE: Simple inspection of the images can help show how much color vision might vary when living in different color environments, or how much adaptation might compensate for a sensitivity change in the observer.
2. Quantify these adaptation effects by using analyses or behavioral measurements with the images to empirically evaluate the consequences of the adaptation²⁹.
   1. Measure how color appearance changes. For example, compare the colors in the two images to measure how color categories or perceptual salience shift across different environments or observers. For example, use analyses of the changes in color with adaptation to calculate how much the unique hues (e.g., pure yellow or blue) could theoretically vary because of variations in the observer's color environment²⁹.
   2. Ask how the adaptation affects visual sensitivity or performance. For example, use the adapted images to compare whether visual search for a novel color is faster when observers are first adapted to the colors of the background. Conduct the experiment by superimposing on the images an array of targets and differently-colored distractors that were adapted along with the images, with the reaction times measured for locating the odd target²⁹.

Results

Figures 2 - 4 illustrate the adaptation simulations for changes in the observer or the environment. Figure 2 compares the predicted appearance of Cezanne's Still Life with Apples for a younger and older observer who differ only in the density of the lens pigment²⁸. The original image as seen through the younger eye (Figure 2a) appears much yellower and dimmer through the more densely pigmented lens (

Discussion

The illustrated protocol demonstrates how the effects of adaptation to a change in the environment or the observer can be portrayed in images. The form this portrayal takes will depend on the assumptions made for the model — for example, how color is encoded, and how the encoding mechanisms respond and adapt. Thus the most important step is deciding on the model for color vision — for example what the properties of the hypothesized channels are, and how they are assumed to adapt. The other important steps are...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Computer
Images to adapt
Programming language (e.g., Visual Basic or Matlab)
Program for processing the images
Observer spectral sensitivities (for applications involving observer-specific adaptation)
Device emmission spectra (for device-dependent applications)