We were chatting to a client, and in a moment of frustration with their current tools they said 'I just want something as easy to use as Survey Monkey, for putting reaction time tasks online. Is that too much to ask?'
To which we responded 'Oh, so you want a bigger and better monkey! Maybe a Gorilla?'
The name stuck.
(Pedants: yes, yes, Gorillas are apes, not monkeys. Five points to Griffindor!)
Online Timing Accuracy
When conducting research online, rather than in the lab, it's important to understand how timing accuracy changes.
The schematic below shows the main differences:
Modern techniques refers to the
Performance.now() function common in all major
browsers which provides microsecond timing. We do not have control
of the underlying operating system, so we cannot make the same OS-level timing calls that native software
can. While clearly there will always be some experiments that require extremely high fidelity timing, the
precision offered by Gorilla is appropriate for a wide range of research.
To talk through each element in the picture above:
This article, Woods et al., 2016, is an excellent summary of the strength and weakness of online research.
This article, Hilbig, 2015, presents the effect of lab- versus web-based research on reaction times.
A detailed technical overview of the timing techniques employed in Gorilla is located here.
Anonymity and Ethics
In compliance with BPS (The British Psychological Society) requirements, identifying data, demographic information and performance data are all stored separately. They are downloaded separately from the metrics tab and joined together outside Gorilla using the Private IDs provided.
Our database architecture supports double-blind studies; you can join demographic data with performance data while remaining blinded.
If using Gorilla in conjuction with a third party recruitment service, it may be that you do not collect any identifying data.
GDPR: General Data Protection Regulation
Gorilla is fully compliant with GDPR.
Gorilla is built around the existing BPS (The British Psychological Society) and NIHR (National Institute for Health Research) standards which were far more stringent than the Data Protection Act. Moreover, GDPR does not apply to data that “does not relate to an identified or identifiable natural person or to data rendered anonymous in such a way that the data subject is no longer identifiable.” The majority of our recruitment policies anonymise participants.
Data Protection and Security
Gorilla is fully compliant with data protection and security policies.
Microsoft Azure guarantees that our servers will be working 99.95% of the time. There are 525,600 minutes in a year. That 0.05% when our servers could be down - outside of our control - equates to ~263 minutes a year. This is equivalent to ~2 minutes a month or ~44 seconds a day. At scale, very rare events happen surprisingly often.
Microsoft Azure performs far above this threshold, nevertheless server downtime is a reality of internet research, and we want to give you the information you need to make an informed decision.
Due to the reality of the possibility of server downtime, we recommend launching experiments in small enough batches that you can afford to lose every participant that is currently active. On our side – as long as you haven’t included participants at the start node – no Gorilla fees would be due. If you are paying participants through a participant recruitment service, you may need to check their policy.
The Code Editor
Go here for more information about the code editor.
We have a seat licensing model. Each person signs up with their own email address and effectively has their own account. Each user then has complete control over any task, questionnaire, experiment and associated data that they have authored. This model fits with BPS (The British Psychological Society) requirements around data security; data is only accessible by the person that owns the experiment or those that they are collaborating with.
Users can also collaborate on projects. When sharing projects the level of access (read, write, admin) can also be set.
We don't currently have the idea of student accounts and supervisors. Any account holder is able to publish their experiments and the onus is on them to ensure they have done so in compliance with their institutions ethics and code of conduct.
What happens when my licence expires?
When your licence expires, your account will revert to a Pay-per-Participant account. All you data, task, questionnaires and experiments will be maintained. You will still have access to all the editing tools and the previewing tools. You just won't be able to collect more data without first purchasing pay-per-participant tokens.
Gorilla is an ideal environment for teaching Research Methods, as students can get valuable experience in operationalising experiments, collecting data, and analysing the data collected.
The Experiment Tree makes the experimental design clear, which can often help students understand whether their experiment is adequately controlled.
We have a suite of tools that allow teachers to manage classrooms. These allow you to:
For Masters students who may not have the time or inclination to learn to code, Gorilla offers a user-friendly environment in which to author completely novel tasks.
Our blog entry from December 2016 describes the UCL experience of teaching 1st year undergraduates with Gorilla.
We're often asked to provide draft text for an ethics application.
We will use Gorilla (www.gorilla.sc) to collect data for our study. Gorilla is a cloud software platform specifically for the behavioural sciences.
To refer to Gorilla in an ethics application, grant application or article for publication, please link to the main website. We also recommend stating the date window within which data was collected, so that someone reading the study could cross-references this with our release notes.
We used the Gorilla Experiment Builder (www.gorilla.sc) to create and host our experiment (Anwyl-Irvine, Massonnié, Flitton, Kirkham & Evershed, 2018). Data was collected between 01 Jan 2017 and 15 Jan 2017. Participants were recruited through [Facebook / Prolific / Research Now].
Published Paper (Preferred Citation)
Anwyl-Irvine, A.L., Massonié J., Flitton, A., Kirkham, N.Z., Evershed, J.K. (2019). Gorilla in our midst: an online behavioural experiment builder. Behavior Research Methods. Doi: https://doi.org/10.3758/s13428-019-01237-x
Anwyl-Irvine, A., Massonnié, J., Flitton, A., Kirkham, N. and Evershed, J. (2018). Gorilla in our Midst: An online behavioral experiment builder bioRxiv, 438242 doi: https://doi.org/10.1101/438242
A list of publications and pre-prints that cite Gorilla can be found here.