Timing

One of the most frequent questions we get is about the timing accuracy of Gorilla. The short answer is that we use high resolution timers for accurate reaction times and frame counting for accurate stimuli presentation times - techniques that have only recently become available in all major browsers.

This paper gives a good overview of the issues involved in online research, and contains a detailed section on timing. As we'll see, the main sources of inaccuracy come from peripheral devices, which affect both native applications (i.e. installed on a computer in the lab) and browser-based ones alike.

Browser-based timing

Much of the criticism of browser-based timings stem from the inaccuracy of JavaScript's setTimeout(), setInterval() and Date.now() functions. Within the last few years, more accurate techniques have become available:

  • performance.now() offers microsecond-precision timestamps. This was first introduced in Chrome in 2012, and iOS was the last to adopt it in 2015.
  • requestAnimationFrame() allows us to run logic every time the screen is about to be refreshed, and should be synchronised to the refresh rate. This was first introduced in Firefox in 2012, and Android was the last to adopt it in 2013.

Both features are now standard in all major browsers.

Displaying Stimuli

The key issue with displaying stimuli is that we are fundamentally limited to the refresh rate of the display. Most modern displays refresh at 60Hz, which is every 16.67 milliseconds. This means that stimuli can only ever be displayed for multiples of 16.67 milliseconds - without specialist equipment, you cannot show a stimuli image for 40ms; you can either have 33.3ms (two frames) or 50ms (three frames).

This issue affects native programs and browser-based ones equally, as they are both ending up on the same display.

Gorilla uses requestAnimationFrame() to count frames and ensure that a stimuli is shown for as accurate a time as possible, by evaluating for each frame whether it would be more accurate to display for another frame (because there is more than 8ms left to display) or to stop on this frame (because there is less than 8ms left to show).

Measuring Reaction Times

Whenever a new screen is displayed in Gorilla, a timestamp is recorded using performance.now(). When a response is recorded (e.g. a keypress or a mouse click), a second timestamp is taken, and the reaction time is then the difference between the two.

Exactly how accurate the response is depends chiefly on the latency of the input device. The default USB polling rate in Windows is 120Hz, or every 8.3ms. Therefore, there could be up to an 8.3ms error on keyboard presses or mouse clicks for peripherals connected via USB.

Touchscreens have higher latency still. The fastest devices are now at around 20ms, and getting faster all the time.

Validating Timing Accuracy in Gorilla

We are in the process of commissioning research that tests the timing accuracy of Gorilla. Consequently, we hope to have some published research on this matter in the near future. Gorilla was launched in 2016, and it will take time to develop this body of evidence.

Nevertheless, UCL are using Gorilla for teaching first year undergraduate students. One of the first research methods lectures was on the IAT. 108 participants took part in an IAT that is included in the Starter Project: Examples project folder. This IAT investigates whether participants have an implicit association between young people and good words, and old people and bad words.

The results of this test replicated previous findings that the participants do have such an implicit association.