Visual Representations... Chapter 5 (J. B. Pelz)

Visual Representations in a Natural Visuo-motor Task


The experiments described in the previous chapters demonstrate a tradeoff between frequent eye movements and working memory in the block-copying task. In a sense, eye movements and working memory have the same logical status in terms of the value of variables currently required for the task. Requiring the subject to perform a secondary task that requires the allocation of limited central resources may affect the tradeoff between eye movements and working memory since some of the variables in working memory may be allocated to the secondary task.

The nature of working memory is not yet well understood; early concepts of working memory held that general working memory served as a unitary short-term store [e.g., Miller 1956]. This view was supported by evidence that cognitive and memory tasks in different modalities interfered with one another. But the interference was not complete, and experiments demonstrated that the performance tradeoff in dual tasks was not linear, suggesting that while all 'short-term memory' loads are somehow related, they did not simply share a unitary store. For example, Brooks [1968] performed a series of experiments probing the interaction of verbal and visual stimulus/response combinations. In one experiment the stimulus was visual; subjects were briefly shown a block capital letter, then were instructed to 'go around' the periphery of the letter (from memory) saying 'yes' for each corner that was on the top or bottom of the letter, and 'no' otherwise. They responded in one of two ways; they either spoke aloud, or pointed manually to the words 'yes' and 'no' printed on a piece of paper. Subjects were more accurate when they responded verbally to the visual stimulus held in memory. Next, Brooks presented subjects with a sentence. Subjects were required to hold the sentence in memory, 'scan' the sentence, and report 'yes' for each noun, and 'no' for all other words in sequence. In this case, the manual response was more accurate than the verbal response.

Taking such results into account, Baddeley and Hitch proposed a more complex model of working memory made up of a number of semi-independent memory subsystems controlled centrally by a limited capacity 'executive' [Baddeley and Hitch 1974, Hitch and Baddeley 1976]. Their model included separate stores for verbal and visual information, with the central executive responsible for coordinating and controlling the peripheral subsystems. In this model, verbal information is held in an 'articulatory loop' capable of holding and rehearsing up to 6 or 7 items without loading the central executive. The articulatory loop is paired with a 'visuo-spatial sketch pad' (VSSP) responsible for temporary storage of spatial information.

The Dual-Task Paradigm

Brooks' experiments provided support for a subsystem/central executive mechanism like that proposed by Baddeley and Hitch by demonstrating modality-specific stimulus/response interference in a single task. Other evidence comes from dual-task experiments in which subjects perform a primary task along with a concurrent secondary task. In classical 'psychological refractory period' (PRP) experiments, two isolated stimuli are presented with variable stimulus onset asynchronies (SOA), and the reaction times are recorded. The general result is that the reaction time to the second response is delayed as the SOA is decreased [Pashler et al. 1993]. The first task apparently utilizes resources that are needed to initiate the second task. There is evidence that the second response is delayed because the 'response selection' stage of the task is postponed, suggesting that the 'bottleneck' limiting performance in dual-tasks is a limited central resource that performs response selection for diverse tasks [Pashler et al. 1993].

Yet in the dual-task literature many tasks appear not to interfere and Pashler was unable to find interference with some saccadic eye movement tasks. The alternative conceptualization to a central bottleneck is diagrammed in Figure 5.1, after Wickens [1986, Figure 3.7]. Human processing resources are visualized in three dimensions. The left face of the solid shows stimulus space, divided into modality/code space; visual or auditory stimuli can carry spatial or verbal information. The 'length' of the solid represents stages of processing, from encoding through response. The response can be either manual or vocal. Multiple tasks utilizing resources separated within this space are thought to interfere less than tasks whose resource requirements overlap. The block-copying task would appear along the far-top edge because the task provides visual/spatial input and requires manual responses.

Pashler points out that attempts to examine complex tasks using the dual-task paradigm have suffered because they "can only be analyzed in a rather coarse way (e.g. aggregating performance errors over many seconds)" [Pashler et al. 1993, p. 52]. Thus a central bottleneck might not be evident in the data measured at a coarse timescale. The block-copying task, however, allows us to detect even subtle

Figure 5.1 Multi-dimensional structure of cognitive processing resources proposed by Wickens [1986].

interference because movements are monitored at such a fine scale. The analyses in the previous chapters allow interference in oculomotor performance, strategy choice (and therefore memory use) and eye/head/hand coordination to be detected.

People are generally not aware of their eye movements, and many so-called 'dual-task' experiments require concurrent eye movements, even though the eye movements are rarely acknowledged as a concurrent task. Kowler [1995] showed that saccadic and pursuit eye movements invariably require the allocation of attentional resources, which suggests that the programming of eye movements should both affect, and be affected by, concurrent tasks, at least at a fine temporal scale. It may be that interference has not been detected in many dual-tasks involving eye movements because of the very brief periods involved in individual eye movements. Any interference could be hidden if the programming of eye movements can be interleaved with other tasks so that the two tasks are performed serially.

What we have found so far suggests that the block-copying task can be viewed as a sequential program of interleaved perceptual and motor actions; in order to move each block the subject must program, initiate, and monitor a large number of eye, head, and hand movements, while gathering information via frequent fixations. Thus our paradigm supports the central bottleneck idea. If this is the case we expect to see some effect of even a simple, unrelated task if we look at the individual movements at a fine timescale. The block-copying task is an interesting area for investigating to what extent all of these programs require overlapping central processing resources, and to what extent they can be carried out independently. Frens & Erkelens [1991] reported that hand movements are affected by an auditory stimulus presented near the time of a visual target for a hand movement. The primary goal of this chapter is to examine whether we would observe similar interference in a task which does not involve the same sensory or motor modalities but may require common central attentional resources.

Concurrent Cognitive Load

Two cognitive load conditions were investigated. The two differed in difficulty and in the extent to which the secondary, cognitive load task "overlapped" with the primary task.

The "Attend" Condition

In the 'attend' condition, the subject was instructed to perform the block-copying task, and at the same time perform a secondary task with limited working memory and attentional resource demands. The task was otherwise unrelated to the primary task. Before the trial began, the subject heard a "target character" (e.g., "F"). While performing the primary block-copying task, subjects heard a 1.1 Hz pseudo-random stream of characters (e.g., "A, K, F, V, ..."), and were instructed to repeat the target character aloud whenever they heard it in the speech stream. The target character and speech stream were created using the PlainTalk(TM) speech generation capabilities of a Macintosh 7100/66AV computer. The target character was selected at random from a 21 character set (A-Z, excluding C, D, P, T, and X). The characters making up the speech stream were selected from the same character set, with the probability of selecting the target character set to 0.20 per spoken character. The target occurred an average of 3.5 times per trial (based on an average rate of 1.1 Hz and an average trial time of 16 seconds). The 'attend' condition was run in two configurations; 'near-attend' trials were performed in the control configuration (see Figure 2.1). In the 'far attend' condition the model, resource, and workspace were moved farther apart on the working plane, increasing the cost of model references, as shown in Figure 3.1.

The "Say Other Color" Condition

The more complex cognitive load condition was labeled the 'say other color' condition. The subject was instructed to say a block color aloud as each block was copied, with the restriction that the color had to be different than the color of the block being copied. For example, while copying a blue block, the subject could say "red," "green," or "white," but not "blue." This task was expected to interfere more directly with the primary block-copying task. The 'say other color' condition was run only in the 'near' (control) configuration.


Effects on motor coordination:

Performing the block-copying task requires the coordination of eye, head, and hand movements. Frens and Erkelens [1991] showed that gross motor (hand) movements were affected by an auditory stimulus, and Kowler [1995] has shown that saccadic and pursuit eye movements require the allocation of attentional resources, so it may be the case that programming head and hand movements also place demands on limited central resources. Referring back to the schematic timeline in Figure 2.18, the complexity of the extended block-copying task is evident. The caption "Pickup Block #1" actually requires a series of actions:

* Identify a resource block in the peripheral view

* Program eye, head, and head movements to that block

* Initiate the movements

* Monitor the movements and program corrections if needed

Placing the block in the workspace also requires a complex series of motor actions, especially in the common case where an intermediate model area fixation is programmed between the pickup and drop. If we think in terms of a sequential program in which the motor actions have to be interwoven with the information gathering model fixations, then the need to coordinate these complex actions, along with the result indicating that the addition of a cognitive load influenced the subjects' strategy, leads to the question of whether motor behavior is affected by the addition of a secondary cognitive task. Some aspects of motor behavior considered earlier can be used to study this question. The cross-correlation analysis introduced in Chapter 4 can be used to look for changes in motor behavior when an additional cognitive load is added on top of the primary task. Figure 5.2 shows the temporal position at which the peak of the horizontal gaze/head cross-correlation occurred for subject sc. There was a significant drop in the gaze/head waveform latency in the cognitive load condition for sc, who showed the greatest change in correlation under the cognitive load conditions. Not surprisingly, subjects' responses were idiosyncratic. Because the cross-correlation is a function of extended waveforms rather than the onset of individual movements, it is not possible to tell from this analysis whether the change in position of the peak is a result of a timing shift, a change in the gaze and/or head's waveform, or both. It is important to note that the primary and 'attend' tasks do not involve the same sensory or motor modalities, so any interference must occur at a higher level.

Figure 5.2 Temporal position of peak gaze/head correlation for subject sc in control and cognitive load conditions.

It was shown in section 4.2.3 that some subjects program the eyes and head to independent targets while moving blocks from the resource area to the workspace. This sophisticated behavior likely requires attentional resources. If that is the case, then the addition of a secondary cognitive load may interfere with the targeting of eye and head. Table 4.3 listed the relative frequency of targeting categories used by four subjects in the control condition. Figure 5.3 shows the effect of the two cognitive load conditions on the subjects' head targeting behavior in the 'attend' and 'say other color' conditions. As in their performance in the control condition, subjects' responses to the added cognitive load varied. Subject eb, who displayed the widest distribution of eye/head targeting strategies in the control condition, shifted towards a tighter linkage between eye and head (decreasing 'diagonal' head movements, and increasing 'separate horizontal & vertical' and 'curved' movements) under the added cognitive load. Except for subject mh (who did not make any diagonal head movements in the control condition), the frequency of diagonal (dissociated) head movements fell dramatically in the 'say other color' condition. The decrease in diagonal head movements (i.e., dissociated eye and head targets) under the 'attend' and 'say other color' conditions suggests that independently targeting eye and head movements requires the allocation of central resources. When the subject is required to perform the cognitive load along with block-copying task, those central resources are apparently too scarce to also support the independent targeting of eye and head movements.

Again, subjects' response to the addition of the cognitive load was idiosyncratic, but other metrics of eye/head coordination also showed evidence that eye and head movements are more closely tied under the cognitive load conditions. The number of

a) b)

c) d)

Figure 5.3 Relative frequency of eye/head targeting behavior in Control, Attend, and Say other color conditions for subjects a) eb, b) jw, c) mh, & d) sc.

Figure 5.4 Frequency of downward, vertical head movements per trial under control and cognitive load conditions for subject sc.

vertical head movements from the model to the workspace that accompanied model to workspace gaze changes was counted. While there was considerable variability between sessions, Figure 5.4 shows that the number of vertical head movements between the model and workspace per trial increased in the 'attend' and 'say other color' conditions, again indicating a tighter linkage between eye and head under the cognitive load conditions.

So, while one might think that head movements are controlled by low-level motor routines, it appears that when an additional cognitive load is added, the eye and head become more tightly linked, temporally and spatially. Other aspects of motor behavior were affected as well. The RMS measure of head movement amplitude used in Chapter 4 can also be used to look for signs that the added load influences motor actions. Even the 'attend' condition, in which the subject was only required to remember a single target character and respond to a 1.1 Hz speech stream, affected subject sc's motor performance. Figure 5.5 shows the head's RMS amplitude along the three rotational degrees of freedom. The RMS amplitude increased along all three axes. The increases between control and 'attend' were significant in all cases (two-tailed, P<0.005), as were the increases from control to 'say other color' and 'attend' to 'say other color' (two-tailed, P<0.001). These effects suggest that many aspects of task behavior are to some degree affected by central processing limitations.

Figure 5.5 Amplitude of head movements under control and cognitive load conditions for subject sc.

Effects on Eye Movement Strategies

It was demonstrated in Chapter 3 that there is a trade-off between eye movements and working memory. The added cognitive load conditions presumably reduces the available working memory resources, and therefore might result in increased eye movements during the block-copying task. Figure 5.6 a) - d) show the relative frequency of each strategy for the four subjects in the control and 'attend' conditions. There was wide variation between subjects. Figure 5.7 a) - c) show the strategies for the three subjects who ran in the 'say other color' condition. As expected, there was a more dramatic shift in strategy use in this condition.

Because of the variation in strategies between subjects (in both control and cognitive load conditions) it is helpful to examine the change in observed strategies between the control and cognitive load conditions. Figure 5.8 shows the average change in relative frequency of each strategy in the two cognitive load conditions. The plotted values are the average differences between the control and cognitive load conditions (i.e., frequencyattend - frequencycog. load). There was a significant shift from the high-memory PD strategy to the low-memory MPMD strategy in the 'attend' condition. More dramatic shifts were observed in the relative frequency of MPMD and PD strategies, and other strategy shifts become evident in the 'say other color' condition which was expected to interfere more directly with task performance. The relative frequency of >MPMD block moves (with 3 or more model references) increased in this condition, and the frequency of the single model reference MPD and PMD strategies fell. The idiosyncratic response to the addition of the cognitive load is evident in Figure 5.9 a) - d), which show the change in strategy use for individual subjects.

Figure 5.6 Relative frequency of strategies in control and 'attend' conditions.

Figure 5.7 Relative frequency of strategies in control and 'say other color' conditions

Figure 5.8 Change in relative frequency for each strategy in the control and two cognitive load conditions; "attend" and "say other color." In the "attend" condition

Figure 5.9 Change in relative frequency for each strategy in the control and cognitive load.

Figure 5.10 Convergence of PD strategy use under added cognitive load.

When viewed a different way, however, some aspects of subjects' performance is seen to converge toward a common reference. Figure 5.10 shows the relative frequency of the PD strategy for the three subjects. While there was a wide range of PD strategy frequencies in the control condition (subject jw had the highest fraction of PD moves, and the fewest number of model references of any subject run in the block-copying experiments), the range narrowed in the 'attend' condition, and virtually disappeared in the 'say other color' condition as jw's use of PD block moves fell to the same value as the other subjects.

Number of Model References Per Block

The shift caused by the addition of the cognitive load can also be analyzed in terms of average number of model references per block. Figure 5.11 shows that measure for the four subjects who performed the 'attend' condition and the three who performed the 'say other color' condition. Two of the four subjects in the 'attend' condition showed significant increases in the number of model references in the 'attend' condition, the other two did not show a significant change. The most dramatic increase was seen in the subject who had relied most heavily on working memory in the control condition: Subject jw increased from 1.0 to 1.5 model references per block. All three subjects who completed the 'say other color' condition showed significant increases in the number of model references per block compared to the control condition. As in Figure 5.10, the convergence under added cognitive load is evident.

Figure 5.11 Number of model references per block for subjects in control and cognitive load conditions.

Concurrent Cognitive Load in the 'Far' Condition

The results of the cognitive load experiments indicate that the changes are a function of the balance between the use of frequent eye movements and working memory. The 'far' condition was shown in Chapter 3 to shift that trade-off toward heavier reliance on working memory, so it is of interest to learn how subjects react to the combination of demands in the 'far attend' condition. The combination of the 'far' configuration and the 'attend' condition is of interest because the two manipulations have been seen to shift the balance between working memory and eye movements in opposite directions; subjects reduced the number of model references in the 'far' condition, and increased them in the 'attend' condition. Figure 5.12 shows the relative frequency of strategy use for four subjects in the far control (i.e., the far condition with no added cognitive load) and the far attend conditions. Subject jw, who showed the highest fraction of PD block moves in the control and far conditions, showed very little difference in the three conditions. Subjects eb and sc, who used a significant number of PD block moves in the 'far' condition, reduced their use of the high-memory strategy dramatically when the cognitive load was added, replacing those with more >=MPMD and PMD block moves. Subject mh, who did not use the PD strategy in the 'attend' condition still showed an increase in MPMD block moves. The changes are reflected in the average number of model references per block copied for the same conditions, shown in Figure 5.13. The average number of model references per block fell from 1.41 in the control condition to 1.17 in the 'far' condition (see section 3.2.1). Adding the relatively simple 'attend' task to the 'far' condition resulted in a compromise, significantly increasing the average number of model references per block to 1.30 (P < 0.05).

Figure 5.12 Relative frequency of strategies in 'far-control' (i.e., far with no added cognitive load) and 'far-attend' conditions.

Figure 5.13 Average number of model references per block for the control, far, and far attend conditions.


As with eye movements, taking the task into account allows issues about the cognitive control of head movements to be addressed. Subjects' responses to the added cognitive load were idiosyncratic, but the addition of a secondary task affected several measures of eye/head motor performance. The peak of the cross-correlation between gaze and head was reduced for some subjects, indicating that the eye and head were more tightly linked when the cognitive load was added, and the frequency of block-moves in which the eye and head were targeted independently was reduced dramatically. Subject eb, who showed independent targeting of the eyes and head in approximately 20% of PMD sequences in the control condition, fell to only half that number in the 'attend' condition, and to almost zero in the 'say other color' condition. The amplitude of head movements also increased for some subjects. These change in eye/head dissociation and head amplitudes are also consistent with the eye and head becoming more tightly linked in space and time.

These experiments have shown that head targeting, which might be expected to be controlled by lower-level peripheral motor routines, is also limited at a central level. Kowler [1995] has shown that programming saccadic eye movements requires attentional resources. These results suggest that head movements and the coordination of eye and head movements also require central resources. When the demands on the limited central resources are increased, the motor systems apparently 'fall back' to simpler, linked movements that require less processing 'overhead.' This suggests that the same shared central resource utilized in the primary task to maintain the color and position information about the blocks is also responsible for coordinating the interwoven perceptual and motor actions necessary to complete the task. Even though the economies of parallel computation are used to great advantage, it appears that complex behaviors are limited at a central level.

The concurrent cognitive load also affected eye movement strategies. When subjects were required to attend to a speech stream in the 'attend' condition, there was a small but significant shift in strategy use by three of four subjects, increasing the relative frequency of double-look MPMD block moves, and decreasing PMD and PD moves. The 'say other color' condition lead to more dramatic shifts in the three subjects who completed the task; >=MPMD block-moves were increased still further, while PD block moves dropped, especially in the subject who had the highest frequency of PD block moves in the control condition. This change in the high-memory strategy points to an interesting aspect of subjects' performance under control and cognitive load conditions. The large between-subject variation in the frequency of PD strategies virtually disappeared in the 'say other color' condition, as all three subjects converged to the same low frequency of PD block moves. The dramatic decrease in PD strategies by the subject who had previously relied on them the most heavily was not paralleled by a similar shift in strategies by the other subjects. While all subjects made more frequent model references in the 'say other color' condition, these changes also resulted in a convergence to a much smaller range of model references per block. The range of values for the three subjects who completed both cognitive load conditions was 0.6 in the control condition, dropped to 0.3 in the 'attend' condition, and to 0.15 in the 'say other color' condition. This experiment demonstrates that the addition of a secondary task, even one unrelated to the primary task, affected subjects' choice of strategy and the degree to which they relied on working memory in copying the model pattern. The convergence of different subjects' performance may suggest that there is some common minimum performance to which all subjects fall when presented with sufficient extra load. The 'far attend' condition demonstrates subjects' ability to adapt behavior based on combinations of constraints. When the model, resource, and workspace areas were separated by ~70deg., subjects adapted their strategies, relying more on memory and less on frequent eye movements to the model pattern. When the relatively simple 'attend' task was added to the 'far' condition, subjects returned to an intermediate number of model references, balancing the constraints imposed by the 'far' configuration and the added cognitive load.

Chap. 6: General Discussion

==> "Visual Representations in a Natural Visuo-motor Task"

By: Jeff B. Pelz
Center for Imaging Science, Rochester Institute of Technology
Department of Brain and Cognitive Sciences, University of Rochester