Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

The overall goal of this protocol is measure parameters of attention and visual processing for almost arbitrary stimuli. This is accomplished by modelling data from temporal order judgements, TOJs. In these, participants judge the presentation order of two targets, the resulting data is analyzed with a novel model that allows to measure how attention affects stimulus encoding rates.

Bundesen’s theory of visual attention provides various use for parameters, such as attentional weight, and processing rates. Typically these are estimated in item report tasks where letters or digits are used, using anything other than letters or digits is difficult and requires intensive training and therefore is rarely done. For example, it’s difficult to imagine how one would measure visuals salience and pop-out displays with this method.

Therefore we propose a novel TVA-based method, which uses temporal order judgment that can be conducted with almost arbitrary stimuli. Temporal order judgments have been used to assess attention for a long time, but combined with TVA, they become more powerful, providing readily interpretable parameters. In the temporal order judgment task, two stimuli, here identified as A and B are shown in close succession or simultaneity.

The perception of temporal order can be modeled with an independent channels model. In such a model there are independent processing channels for each target. An order comparator monitors the channels and records the count when a particular target, here A, arrives at it before the other target.

Before this is demonstrated, we briefly describe the form in which the experimental data is typically represented. The relative frequency of a particular type of judgment, here A first, is plotted depending on the SOA, the time interval between the presentations of the two targets. Now, with a large negative SOA, here this means stimulus A leads, it is most likely that A arrives first at the comparator.

Only very rarely will stimulus B arrive earlier. Therefore, we obtain a data point representing a relative frequency close to one. For a large positive SOA, that is stimulus B leads, the comparator registers A first results only very rarely.

Therefore we obtain a data point close to zero. If the SOA is zero, the targets are presented simultaneously, and a data point at the point five chance level is recorded. When the SOAs varied over a range of values, the resulting pattern can be described with a psychometric function.

Now, when attention is directed to a stimulus, here, stimulus A, it arrives earlier at the comparator, consequently, here shown for SOA zero, the probability of A first judgments is increased. This shifts the whole psychometric function. The SOA at the which the red curve crosses the point five level is often taken to quantify the influence of attention.

The temporal auto judgment seems to be the natural way to assess perceptual latency, and many questions about the influence of attention and time perception have been answered with this method. However, the TOJ has a central weakness, and this is its relative nature. Temporal auto perception might tell us that stimulus A is perceived before stimulus B, but it doesn’t tell us why.

It might be because stimulus A is processed faster, or because B is processed slower. To increase the explanatory power a TOJ model is derived from TVA, which models the encoding processes of visual stimuli. In each channel, encoding is assumed to proceed according to TVAs exponential race model.

The probabilities of encoding each target within a certain duration are transformed into the probability of encoding one target before the other. The perception of temporal order. Importantly, the two rate parameters inherited from TVA allow to answer questions such as, was it the attended stimulus that was processed faster, or was it the unattended which was slowed down?

Alternatively, the rates can also be expressed as relative attentional weights. Select stimuli according to the research question. In general, it must be possible to show two targets at different locations on the screen.

Later in this protocol, we show results from experiments with pop-out displays, natural images and letter targets. Here is an example of how action space and background objects in natural images can be used. The photograph with both targets present is laid over one with one without them.

Objects are virtually removed by making the upper image locally transparent. With this procedure, images with both, none, and either of the two objects present, are created to establish TOJs with background first, action space first, and simultaneous presentation. To plan the design and sample size of the experiment, a Bayesian power analysis can be performed.

Repeatedly simulate and fit the data with the intended model, experiment design and hypothesized parameters. The proportions of simulations for which a success criterion is reached, for example, a difference in attentional weights, is used to estimate the experiment’s power. Use an experiment builder or psychophysical presentation library to implement the experiment.

We supply a for the open source experiment builder OpenSesame, for TOJ with letters, digits and shapes, in which only the trials table needs to be specified. We also supply an OpenSesame example for TOJs with natural images. For each condition, create trials for all planned SOAs.

Exemplary SOA distributions are shown in the representative results. The two targets are defined as probe and reference. The reference is always shown at zero, and is the unattended target, whereas the probe stimulus is subject to the attention manipulation.

In trials with negative SOA, present the probe stimulus first, and after the SOA the reference stimulus. For trials with positive SOAs, present the reference stimulus first, and after a delay, the probe according to the SOA. For the SOA of zero, present both targets simultaneously.

In neutral conditions, the assignment of probe and reference is arbitrary, but required for analyzing the data. Create repetitions of all SOAs. Randomize or systematically vary influences that are not of influence, such as stimulus location or target identity.

The number of repetition depends on the attended power. Roughly 800 trials can be presented within one hour. If more repetitions are needed, consider splitting the experiment into several sessions.

Welcome the participants and inform them about the general procedure of the experiment. Obtain the consent to participate. Ensure they have normal or corrected to normal vision.

Provide a quiet booth for the experiment. Adjust chair, chin rest and keyboard, and so on, to ensure optimal conditions. Make the participants aware that the experiments requires attention and mental focus, and can be fatiguing.

Ask them to take short breaks when required. Present on-screen instructions detailing the presentations and response collection. Remind the participants that the task is to report the perceived temporal order of the targets, and guess in trials where they cannot tell the order at all.

To avoid eye movements during the trials, ask the participants to fixate a central marker whenever it is shown on the screen. Ask them to rest their head on the chin rest. Ask them to take short breaks if necessary.

Perform a short training with ten to 20 trials. Confirm that the participants understood the task. Let them explain it.

If they have no further questions, leave the booth for the main experiment. After converting the trial-by-trial data into counts of probe first judgments, parameters can be estimated by running the desired Beyesian hierarchical model. After the process is complete, plot convergence diagnostics for all parameters and confirm that the chains have converged and that they provide a sufficiently large effective sample size.

Then, plot and assess the parameters of interest. Exemplary results are shown in the next section of this video. The first experiment measured the influence of visual salience on processing speed.

Participants judged which of two target line segments, left or right, in a background pattern, flickered first. In half of the trials, the probe was a color pop-out. The data was fitted with TOJ model derived from TVA, which was described earlier.

The benefit from the salient stimulus can be seen in the attention condition, as an increased attentional weight of point five nine for the probe. In the control condition where no target was salient, the neutral weight of point five was obtained, hence the attentional weight of the probe in the attention condition was reliably higher. Zero, no difference, was not included in the 95 HDI.

The individual weights of the probe and there reference show that the salient stimulus advantage results from a 16 hertz processing rate reduction of the non-salient stimulus. Possibly the salient target suppresses processing of the non-salient one to some degree, and therefore benefits relatively. The second experiment investigated attentional advantages in the action space depicted in natural images.

Action space, probe and background objects reference, appeared for the TOJ. In a control experiment a baseline without action space advantages was recorded using upside down versions of the images, for which the scene layout perception is disturbed. The parameter posterius indicate an action space advantage in the attention condition.

The probe weight is higher than the neutral one of point five. Curiously, this is also found in the control condition, suggesting that the image inversion did not remove the potential action space advantage. Therefore, most likely the advantages caused by factors such as salience or visibility, which are not removed by the scene inversion.

The possibility that the experimental power was too low to detect the action space effect, is discussed in the written part of this protocol. A peripheral cue is used in the third experiment, leads to a large effect in an additional parameter, which models the delay between encoding processes. This experiment is discussed in detail in the written part of this protocol.

In conclusion, the protocol in this article describes how to conduct simple TOJs, and fit the data based on a fundamental model of stimulus encoding. In three experiments, we showed that the results can be evaluated in hierarchical Bayesian estimation framework. We found that salience in pop-out displays leads to increased attentional weights.

Also, increased weights were estimated for action space objects in natural images, however, due to the persisting advantage, when the spatial relations were disturbed by showing the images upside down, it is likely that another attentional benefit lead to the weight increase. A peripheral cue in experiment three led to a large value for a parameter that models an additional delay between the encoding processes. Advantages of the protocol are, the simplicity of the TOJ task, which can utilize almost arbitrary stimuli.

The thorough theoretical underpinning by TVA, and the Bayesian evaluation scheme. Although salience plays an important role in many studies, only few of them have tried to quantify visual salience. Quantifying visual salience however, would enable us to compare different feature dimensions, like orientation, color, or motion.

Here, we investigated the influence of color contrast on the attentional weights of formal model derived from TVA, allowed us to measure salience quantitatively, and in a psychologically sound way. For the success of this protocol it is crucial that there are only two stimuli that generate temporal signals at the target locations. Explicitly modeling additional stimuli in TOJs, such as peripheral cues, is goal of future research.