Multiresolution Representations

According to multiresolution theory, the neural representation of the visual world is made up of a collection of component neural images. Each component neural image is constructed by mechanisms that are sensitive to a narrow band of spatial frequencies and orientations.

To help explain this idea of multresolution representations we will use the phenomenon of pattern adaptation. Center the Flash movie below and follow the directions to see a demonstration.

Let's look at what happens to contrast sensitivity after adaptation to a grating of a particular frequency. (At the risk of being repetitive, I ask you once again to consider how you would implement an experiment to make the measurements shown in the figure below.)

This figure shows the CSF after adaptation to a 7.5 cpd grating (dots). The effect of adaptation is to decrease the sensitivity to spatial frequencies maximally at the adaptation frequency. Frequencies near the adaptation also demonstrate reduced sensitivity but frequencies further away do not have lower contrast sensitivity.

This would indicate that the CSF is composed of multiple mechanisms and that adaptation to a particular spatial frequency affects only a subset of those mechanisms near the adaptation frequency.

The idea is shown conceptually in the figure on the right. The CSF is thought to be the envelope of sensitivities from multiple channels. Adaptation to a particular spatial frequency will lower the sensitivity of those mechanisms that are sensitive to the adaptation frequency. This would cause the type of "notch" seen above in the CSF.

Each one of these narrow mechanisms creates a neural representation of the visual world. These multiple representations are then combined later in the visual system to for the complete neural representation.

This figure shows tuning curves for simple cortical cells. The firing rate for these cells for gratings of various frequencies are shown for three cells. Again we see a bandpass response that is characteristic of cells that have receptive fields with fixed sized inhibitory and excitatory regions.

It has been proposed that such cells may underlie the narrow band channels that make up the CSF.

Returning to the demonstration of pattern adaptation...

When you look at a grating of a particular spatial frequency, a number of sub-mechanisms produce neural representations (or in simpler terms, a number of narrowly tuned mechanisms respond). The ensemble of these representations determine the apparent spatial frequency. When you adapt to a particular spatial frequency, those mechanisms most closely tuned to the stimuli undergo the most adaptation. Upon subsequent stimulation, the response from these mechanisms are reduced due to the adaptation. Therefore, the ensemble response will be shifted due to the reduced input from these mechanisms.

After adapting to a slightly higher spatial frequency, the response to the test frequency is shifted to lower spatial frequencies since the response to the higher frequencies is reduced. That is the ensemble response is now shifted towards activity in channels sensitive to lower spatial frequencies. The same argument applies to adaptation to a frequency slightly lower than the test frequency. Now there will be more activity in channels tuned to slightly higher spatial frequencies than the adapted channels.

In the case of the adaptation to the tilted gratings, the same type of argument applies. As we saw in the last chapter, cortical cells are characterized by orientation selectivity. A particular simple cell, for example, responds best to a particular orientation and responds less well to similar orientations. The determination of the the perception of orientation is again due to the ensemble responses from these cortical units. Adapting to a particular orientation will shift the ensemble response.

Pattern Discrimination and Masking

More evidence to support multiresolution representations comes from studies of pattern discrimination.

The filled squares show the CSF for the experimental conditions used in this experiment. The open circles plot the reciprocal of the logarithm of the contrast needed to differentiate a square wave pattern from a sine wave pattern of the same fundamental frequency. If you remember, a square wave can be constructed by adding harmonic functions. For a square wave pattern, the second component sine wave has an amplitude 1/3 that of the first component and a frequency three times that of the first.

According to multiresolution theory, the components of these harmonics would be analyzed in separate channels. If you are increasing the contrast of a square wave pattern from zero contrast, you would not be able to tell that it is a square wave until this second component reaches threshold. Let's assume that is what is happening. That is, the open circles show the level of detection when the second component of the square wave reaches threshold. The fundamental component for each point is then located at a frequency that is 1/3 that of the plotted point and the contrast for the fundamental is 3 times higher than the plotted point. So if we shift the data to the left to frequencies 1/3 lower (in linear units) and if we shift the data down to higher contrasts that are 3 times higher (remember, contrast decreases as you move up the sensitivity axis), we will see that the data aligns with the CSF. The arrow shows the shift in the figure.

So our theory is supported that the separate components of the pattern are analyzed in independent channels with different resolutions. In this experiment, the components of the stimulus are analyzed separately. In the next experiment we will see that the when the components are more similar, they will influence the detectability of each other due to interactions within channels. When the components are different, they are analyzed separately and do not influence their detectability.

In the experiment shown below (B), the masking stimulus is one cycle of grating of various spatial frequencies. The test stimulus is a 2 cpd grating. The contrast needed to detect the test stimulus is what is being measured and plotted on the y-axis. This contrast is measured for a number of different masks of various spatial frequencies on contrasts. On the x-axis is plotted the contrast of the mask. Each curve represents a mask of a different spatial frequency. The curves have been displaced vertically in order to reveal the shape of each curve.

Let's look at the curve labeled 2.0. This mask has a spatial frequency of 2 cpd which is the same as the test grating. The first point on this curve on the left shows the contrast needed to detect the test grating when the mask has very low contrast. So it is basically showing the threshold for detecting the test grating. As the contrast of the mask increases, you see that the threshold decreases. That is, the masking stimulus is facilitating the detection of the test stimulus. As the contrast of the mask increases past this point the thresholds increase to the right until we see that now the mask is impairing the ability to detect the target and the thresholds are much larger. This type of curve is sometimes called a "dipper function". Where the curve dips, we see facilitation. Where the curve rises to the right we see masking of the test by the masking stimulus.

Now look at the other curves. As the difference between the spatial frequency of the mask and test increases, we see a decrease in facilitation. This support multiresolution theory because it is consistent with the idea that stimuli with widely different spatial frequencies are encoded by different component images. Although not shown here other evidence indicates that the masking effect too is reduced as the difference between the spatial frequencies of the two patterns is increased.

Multiresolution theory has become a mainstay in many models of visual processing. However, the theory does face challenges from experiments that show interactions between channels with different orientation and spatial frequency tuning. The question facing researchers is to characterize these effects and identify the locus of these interactions in order to achieve better understanding of the spatial processing of visual information,

Continue on to
Hyperacuity and Vernier Acuity