Visual Representations... Chapter 3 (J. B. Pelz)

Visual Representations in a Natural Visuo-motor Task


Introduction

The results of the block-copying task in Chapter 2 are interpreted as evidence of the high cost of working memory relative to eye movements. Subjects chose to make frequent eye movements to the model area rather than copy multi-block patterns from memory, even though memorizing multi-block sub-patterns is well within the limits of working memory. The conclusion is not that eye movements are used to the exclusion of information held in working memory, but that subjects strike a balance between the two based on their relative cost and the amount of information in the model pattern needed to complete the task. Given this analysis, we expect that the trade-off between working memory and frequent eye movements would be affected by varying those parameters. If subjects are able to dynamically adjust the balance between memory and eye movements, increasing the cost of frequent model references would shift the balance toward fewer eye movements and higher memory strategies. If the frequent model references in the control condition serve to gather information necessary to perform the task (rather than being simple artifacts of experimental design, because the eyes are so much faster than the required hand movements), then decreasing the information content of the model pattern should lead to fewer model references. Working memory load is reduced, so fewer eye movements would be needed to maintain the desired balance.

In order to investigate the significance of the frequent model references observed in the main experiment, both types of manipulation were explored: 1) increasing the cost of the model references by increasing the distance between the three areas, and 2) reducing the information content of the model by eliminating color or position information.

Manipulating Task Parameters

Increasing the Cost of Frequent Model References

To test the effect of varying the relative cost of frequent model references, the model, resource, and workspace areas were rearranged to increase the cost of the frequent gaze changes. Placing a larger distance between the three areas required much larger gaze changes than in the control configuration. Figure 3.1 shows the configuration, labeled the 'far' condition. The model and resource are separated by ~40deg., and the workspace and model are separated by ~70deg.. This angular separation required large gaze shifts made up of eye, head, and torso movements that are more costly than are the equivalent movements in the 'near' condition.

One measure of the increased cost of eye movements in the far condition is the time it takes to perform each block move. Figure 3.2 shows the average duration of block moves as a function of strategy for five subjects in the near and far conditions (error bars show between-subject s.e.m.). Performing the task in the larger space required more time for all strategies, but the increase was not constant across strategies. As seen in Figure 3.3, the PD strategy (with no model fixations) showed the smallest average increase (285 msec), and the >MPMD strategy (with at least three model references)

Figure 3.1 Position of model, resource, and workspace areas for the control and 'far' configurations.

had the largest increase in block move duration (885 msec). An average 'temporal cost' of the model references was derived by subtracting the average increase of a PD block move from the other values, and normalizing by the number of model references. The result, 420 msec (s.e.m. = 50 msec), is an indication of the average penalty suffered for each model reference.

Nine subjects performed the block-copying task in the 'far' condition. All of the subjects had previously performed the task in the control configuration. Subjects performed from 96 to 208 block moves (mean = 160). Figure 3.4 shows the relative frequency of strategies selected for the control and 'far' conditions averaged over the nine subjects (error bars show between-subject standard error). Figure 3.5 shows the relative frequency of strategies for each of the nine subjects. All subjects showed a decrease in low-memory strategies (MPMD and >MPMD), and seven of the nine subjects increased use of the high-memory PD strategy (one subject never used the PD strategy). It is useful to examine the shift in strategies used by the subjects between the control and 'far' conditions. Figure 3.6 shows the average change in the relative frequency of each strategy across four subjects. Each value represents the difference between the relative frequency in the far and control conditions. The mean response is a reduction in the low memory >MPMD and MPMD strategies, balanced by increases in the higher memory MPD and PD strategies, which are performed with fewer model references.

Figure 3.2 Duration of individual block moves averaged over five subjects in 'near' and 'far' conditions.

Figure 3.3 Added cost of model fixations in the 'far' condition, averaged over five subjects.

Figure 3.4 Mean relative frequency distribution for control and 'far' conditions averaged across nine subjects.

Figure 3.5 Individual relative frequency distribution for control and 'far' conditions for nine subjects.

Figure 3.6 Change in relative frequency of strategies between the far and control conditions, averaged across nine subjects.

The number of multiple model fixations was reduced among eight of nine subjects. Note however that the reduction in eye movements (and the increased reliance on working memory) was not complete; none of the subjects completely eliminated MPMD strategies, nor did they copy large multi-block sub-patterns without referring back to the model. Figure 3.7 shows the average number of model references per block, averaged across subjects. The mean drops from 1.43 in the control ('near') condition to 1.17 in the 'far' condition (P < 0.001).

Figure 3.8 shows the individual subjects' performance in the two conditions. Eight of nine subjects made fewer model references in the 'far' condition, though even the subject who had the lowest number of looks per block in the 'far' condition (subject LD) averaged 6.4 model references per 8-block trial, providing strong evidence that the frequent model references are not artifacts of the control experiment's design.

The distribution of model references is broader in the near condition than in the far and there is large variability between subjects. When each subject's change in model references between near and far conditions is compared, the common change in task performance is more obvious.

Figure 3.7 Number of model references per block moved in the near and far conditions, averaged across nine subjects.

Figure 3.8 Number of model references per block moved in the near and far conditions for nine subjects.

Manipulating the Information Content of the Model

The experiment described above demonstrated that subjects' choice of strategy can be manipulated by adjusting the 'cost' of frequent model references. Our understanding of the model fixations as a mechanism to bind values from the fixation point into working memory suggests that reducing the amount of information needed from the model area may also reduce the frequency of model references, even in the 'near' condition. Fixations in the model area gather information about the color and position of each block. The amount of information could be reduced by 1) using a monochrome model, or 2) using a simple, predictable shape so that the relative position of each block is predetermined.

The Monochrome Condition: Eliminating Color Information

In the preliminary version of the experiment performed on the Macintosh, the model pattern and all blocks in the resource area were of a single color, eliminating the need to determine the color of each block to be copied -- the only relevant information remaining in the model pattern was the position of each block. The blocks were manipulated with a computer mouse. Blocks could be 'picked up' from the resource area by positioning the mouse cursor over a block and holding the mouse button down. The block could then be moved across the screen and 'dropped' in the workspace by releasing the mouse button. In order to reduce fine positional control requirements, blocks dropped in the workspace 'snapped' into position in a regular grid. Eye movements were recorded with an SRI Dual-Purkinje image tracker, and cursor movement was recorded throughout the task to indicate 'hand' position.

Eliminating color information from the model resulted in a decrease in model references. Figure 3.10 shows the shift in strategy averaged over four subjects in the `monochrome' and control conditions, and Figure 3.10 shows the change in strategy between the two conditions. As in the `far' condition (refer to Figure 3.6), low-memory strategies (MPMD+) decrease, and the frequency of the high-memory PD strategy increases. While the change in frequency of those two extreme strategies was the same in the `far' and `monochrome' conditions, the changes in the intermediate memory strategies (MPD and PMD) were opposite for the two manipulations. The frequency of the MPD strategy (where the subject presumably gathers both color and position in a single model reference) increased in the far condition, where the cost of model fixations is increased. The monochrome condition, on the other hand, led to more frequent use of the PMD strategy. Because the model pattern is made up of a single color, there is no need for a model fixation before a block is picked up. Figure 3.11 shows the number of model references per block in the control and monochrome conditions, averaged across four subjects. The average number of model references was reduced from 1.6 to 1.0, due largely to the decrease in MPMD strategies balanced by an increase in PD strategies, consistent with the interpretation that the frequent model references observed in the control condition are used to gather information necessary to complete the task, and not simply an artifact of the experimental design.

The Linear Condition: Reducing Position Information

The monochrome condition reduced the information content of the model by eliminating color as a variable. The informational demands can also be reduced by constraining the location of the blocks, rather than their color. In a pilot experiment,

Figure 3.9 Mean relative frequency distribution for control and 'monochrome' conditions averaged across four subjects.

Figure 3.10 Average change in relative frequency of strategies between the monochrome and control conditions for four subjects.

Figure 3.11 Number of model references per block in the control and monochrome conditions, averaged across four subjects

some of the colored model patterns formed a single, horizontal line, as shown in Figure 3.12. This reduced the position information needed to duplicate the pattern; after the first block was placed in the workspace, the pattern could be completed by moving consecutive blocks to the 'end' of the pattern. In the pilot experiment, the 'linear' model patterns were interspersed with the normal two-dimensional model patterns, appearing only once in each 24-trial block. Figure 3.13 shows the average relative frequency of each strategy for the linear trials and the remaining 23 trials with two-dimensional models. Figure 3.14 a) - i) shows the strategies for each of the nine subjects for the control and linear trials. The increase in the relative frequency of the PD strategy seen in Figure 3.13 approached significance (P< 0.1) as did the decrease in the MPD strategy (P<0.1).

The modest increase in the relative frequency of PD block moves in the pilot experiment (in which the linear models were interspersed infrequently with normal model configurations) suggested that subjects were changing their strategies in real-time based on the reduced spatial information content of the linear model (subjects were not told when the linear patterns would appear). Based on the results of the pilot experiment, another experiment was run in which the linear models were presented in 24-trial blocks to see whether subjects would adapt their strategies when the position information was consistently reduced. Four new subjects performed the block-copying task with control and 'linear' models, each presented in blocks of 24 trials. Figure 3.15 shows the mean across the fours subjects and the between-subjects standard error. The individual subjects' strategies are shown in Figure 3.16 a) - d). There was a larger shift toward lower memory strategies when the linear trials were presented in 24-trial blocks, but the strategy shifts are masked to some degree by the idiosyncratic performance

Figure 3.12 Model configuration for the 'linear' condition.

Figure 3.13 Mean relative frequency distributions for two-dimensional and linear models when linear models were interspersed with 2D patterns (one in 24 models were linear).

(see legend, next page)

Figure 3.14 Individual relative frequency distributions for control and 'pilot-linear' conditions

of the subjects. When the results are analyzed by examining the shift in each subject's strategy between the two conditions the strategy shifts are more evident. Figure 3.17 shows the average change in strategy use (i.e., frequencylinear - frequencycontrol) across the four subjects, along with the standard error between-subjects. This analysis of the linear condition showed a significant decrease in >MPMD strategies (P<0.03), and significant increases in PMD (P<0.05) and PD (P<0.05) strategies. The drop in the frequency of the MPMD strategy did not reach significance (P<0.12).

Figure 3.18 shows the mean number of model references for the four subjects in the near (control) and liner conditions. The mean number of model references per block drops from 1.43 in the control condition to 1.21 in the blocks of linear trials. Error bars show the between-subjects s.e.m.. Most of the variability between subjects is due to large mean differences between the subjects rather than variability in the change between the control and linear conditions. Figure 3.19 shows the average number of model references for each of the four subjects. A paired sample t-test shows the decrease of 0.22 model references per block to be significant (P<0.03).

Figure 3.15 Mean relative frequency distributions in control and 'linear' conditions, averaged across four subjects.

Figure 3.16 Individual relative frequency distributions for control and 'linear' conditions.

Figure 3.17 Mean change in strategies between control and linear conditions, averaged over four subjects.

Figure 3.18 Mean number of model references in the control and liner conditions, averaged across four subjects.

Figure 3.19 Individual number of model references in the control and liner conditions.

Discussion

When the distance between the model and workspace is increased from ~15deg. to ~70deg., subjects had to make large eye, head, and torso movements to return gaze to the model area. Moving the model away increased the time required for all block moves, but strategies with model references required more time. The differential cost

((tstrat - tPD)/Nmodel references) of eye movements to the model area was about 400 msec more than in the control condition, so there was a significant cost associated with model references in the 'far' condition. Subjects adapted their strategies to the 'far' condition, making fewer model references. The relative frequency of block moves completed with two or more model references (MPMD + >MPMD) fell significantly, balanced by increases in MPD and PD strategies, indicating that the color (and in some cases the position) of a block was held from previous model fixations. The average number of model references per block decreased by 15% in the 'far' condition. Increasing the cost of model references caused a significant reduction in eye movements to the model, but subjects still relied on repeated model references to complete the task. Although the control experiment described in Chapter 2 showed that subjects are capable of copying several blocks from memory, the mean number of model references per block did not fall below 1.0, so it is clear that when given the option of working from multi-block subpatterns held in working memory, or making frequent eye movements to the model, they choose to minimize working memory and rely instead on eye movements.

When subjects performed the block-copying task with monochrome blocks (leaving only information about a block's position relevant) subjects again shifted their strategies and made fewer model references. The largest changes were significant decreases in >=MPMD block moves and increases in PD moves. There was also a shift from MPD to PMD strategies. The mean number of model references per block fell from 1.6 to 1.0. The drop in frequent eye movements to the model area when color was eliminated supports the interpretation that the frequent model references in the control condition are often used to gather color and position information independently.

In the control condition, there was no significant difference between the relative frequency of MPD and PMD strategies, while in the monochrome condition the frequency of PMD sequences were more than double that of MPD sequences. In the monochrome case, where only the position of blocks in the model area is relevant, the PMD strategy is more efficient than the MPD strategy because position needs to be held for a shorter period of time before the block is placed in the workspace. This evidence that subjects are adapting their strategies based on the details of the specific task is an example of the task-dependency of visual behaviors. The eye movements are still used to serialize the task when the model blocks are all the same color, but the subtasks, and the order in which they are executed, is driven by the constraints of the task.

Linear models carry less information than do two-dimensional models because there is no positional uncertainty about the remaining blocks once the first block is placed in the workspace (presuming that the blocks are copied in order, from left to right, which was invariably the case). When linear models appeared unexpectedly and at a frequency of only 1 in 24 trials, there was a small increase in PD strategies, coming mostly at the expense of MPD block moves. When subjects knew to expect the linear model configuration, and they appeared in blocks of 24 trials, there was a more pronounced change in strategy use by the subjects. >MPMD moves vanished and the frequency of MPMD and MPD strategies fell, while the frequency of PMD and PD strategies increased. Subjects made significantly fewer fixations in the model area in the linear condition, as one would expect given the interpretation of the two model looks in the MPMD strategy as separate references to color and position. When the subjects' choice of strategies is examined in detail, however, it is clear that the interpretation is not simple. Specifically, the increase in the frequency of PMD sequences is puzzling. In the control experiment, the PMD sequence was understood to occur when the subject remembered the color of a block in the pattern from a previous fixation. After a block of that color was picked up, the block's position was determined (or confirmed) in the model reference between the pickup and the drop. But in the linear condition, once the first block is put in place in the workspace, the position of each subsequent block is determined. The relatively high frequency of the PMD strategy in the linear trials suggests that subjects are not working with the block moves as completely separate actions, but instead are 'thinking ahead' to the following block. Alternatively, it may be that it is just as easy to use visual information gathered in the model to guide the placement of the block as it is to use relative position information from the partially completed workspace.

The three experiments presented in this chapter support the interpretation of frequent model references as a strategy to reduce working memory load. They also show that the tradeoff between memory load and frequent eye movements is flexible, and that subjects can dynamically alter that tradeoff based on task demands. The difference between linear trials that occurred infrequently among 2D model patterns in the pilot experiment and those run in blocks of trials suggests that subjects adapt their strategies based on recent experience. It would be possible to investigate this interpretation of subjects' behavior further in a saccade-contingent experiment where all remaining (i.e., uncopied) blocks in the model are changed between each block move. If this is done with the linear models, it would be possible to eliminate the benefit of looking ahead, and may influence subjects' behavior. It would be instructive to see whether subjects would shift their strategies from PMD to MPD, a strategy that reduces the number of looks to the model area, but does not suffer from changes in the model area between block moves.

Chap. 4: The Coordination of Eye, Head, & Hand Movements in a Natural Task


==> "Visual Representations in a Natural Visuo-motor Task"

By: Jeff B. Pelz
Center for Imaging Science, Rochester Institute of Technology
Department of Brain and Cognitive Sciences, University of Rochester

1995