Support Home

Due Diligence

  • Overview
  • What is Gorilla?
  • The Business Case for Gorilla
  • Testimonials
  • Anonymity and Ethics
  • Data Protection and Security
  • GDPR Compliance
  • Legal
  • The Code Editor
  • Licensing
  • Teaching
  • Online Timing Accuracy
  • Server Downtime
  • Ethics Applications
  • Publications and Referencing

Overview


This document provides information on a range of topics that may need to be reviewed by stakeholders as part of the process of getting a departmental licence. If you have further questions, please don't hesitate to contact us by email.

The sections that are most often asked for are:

  • Anonymity and Ethics
  • Data Protection and Security
  • GDPR Compliance
  • Legal Documents
  • Ethics Applications

What is Gorilla?


The Gorilla Experiment Builder is a cloud based research platform that allows researchers and students to quickly and easily create and deploy behavioural (reaction-time) experiments online.

In a nutshell, how does it work?

Simply login to Gorilla using your browser to access:

  • A questionnaire builder
  • A task builder
  • An experiment configuration tool

Create your component questionnaires and tasks and then link them together into an overall experimental protocol.

Then deploy it online and recruit your participants. You can do this via social media (i.e. Facebook) or paid services via our trusted partners (Prolific, mTurk, SONA and more) or by email.


How is this different to a survey tool?

Survey tools allow you to ask explicit questions and get subjective responses from the respondent.

In contrast, a behavioural study typically has lots of trials that capture the respondents behaviour in terms of accuracy and reaction times. From these raw metrics, secondary metrics can be derived to get an objective measure of behaviour.

Gorilla is built for running behavioural studies. The task builder is fully tooled – you don’t have to code – so you can create your study quickly and easily. You then configure the experiment protocol in the Experiment Tree tool.

This allows you to add randomisation, counterbalancing, repeats and delays to create a wide range of experiments designs.

Find out more about our tools here.


What else?

We know that novel research sometimes needs completely new functionality! As well as the tooled environment Gorilla allows you to augment the functionality we provide by adding scripts (snippets of JavaScript) or build a task from scratch in the Code Editor.


The Business Case for Gorilla


The Gorilla Experiment Builder is a cloud based research platform that allows researchers and students to quickly and easily create and deploy behavioural (reaction-time) experiments online.

In a nutshell, how does it work?

Simply login to Gorilla using your browser to access:

  • A questionnaire builder
  • A task builder
  • An experiment configuration tool

Create your component questionnaires and tasks and then link them together into an overall protocol.

Then deploy it online and recruit your participants. You can do this via social media (i.e. Facebook) or paid services via our trusted partners (Prolific, mTurk or SONA) or by email.

How much does it cost?

Build for free. Pay per participant.

This means it’s completely free to sign up to Gorilla and to use our questionnaires, task and experiment creation tools.

We only charge for data collection. Our current prices are shown here.

This model allows universities to teach students how to create tasks and experiments—unimpeded by cost, subscription plans or number of users.

Once researchers or students have built a study they’re happy with, they simply deploy it online for data to be collected.

Can you give me an example? Each student could send the link to a handful of friends, and once the data is in, the entire class can analyse the data. One study with 200 participants would cost just £150 + VAT.

Do you do departmental licences? We offer departmental licences starting at £2,000 + VAT that covers student usage. You can find out more in our separate pricing documents.


What are the benefits of Gorilla?


What are the benefits for students and teaching staff?

  1. Students can design and create experiments —without learning to code. This means they’re focused on what’s most important: the psychological methodology and operationalisation of their experiments; instead of losing hours of valuable time to coding.
  2. Students enjoy their research methods course – increasing their engagement with and ratings of a large component of their course
  3. Students have more time to analyse their data—improving their quantitative and analytical skills.
  4. Students are more able to conduct quantitative analysis—increasing their employability with employers who place high value on this skill. Gorilla removes the barrier to quantitative research online, improving the balance between qualitative and quantitative research.
  5. Students can more easily share resources and submit assignments for marking—thanks to Gorilla’s teaching tools. This lightens the load for teaching staff, by dramatically reducing the amount of time spent teaching students to code or supervising complex experiments.

What are the benefits for Researchers?

Researchers can design, create and complete studies in record time—and for a fraction of the cost of traditional methods

This means that entire studies can be condensed into days, rather than weeks—and for very little cost. Productivity is uplifted, and dead ends that might have taken months to uncover are quickly resolved.


What about Technical Staff and Supervisors?

Because Gorilla is so easy to use, technical staff have fewer support queries—freeing them up to help elsewhere.

When queries do arise, resolving them is faster (and can be done from anywhere) thanks to Gorilla’s online interface.


Any other benefits?


Gorilla reduces the cost of computing and equipment

Gorilla operates on any device with a browser. Which means students can use their own laptops for lab work. Over time, this puts less pressure on the department to provide expensive computers.


Gorilla reduces the cost of facilities

Data can be collected online, which also reduces the need for testing rooms. This frees up valuable real-estate within the department and reduces facilities costs.


Gorilla reduces the need for on-site attendance

For online courses, students can work from any location, which means they don’t have to come to the department to get practical lab experience. This convenient way of working frees up space and reduces facilities and staff costs.


What support is provided by Gorilla?


Sample tasks available in Gorilla can be cloned for easy, time-saving adaptation. Our support pages also provide everything you need to use the software.

We also offer technical email support to paying clients. However, we encourage students to problem solve themselves, as much as possible. In this way, they learn more, while accurately reflecting their skill level.


Why researchers and students like Gorilla:

'Gorilla is designed for psychologists, so it has all the relevant options. I've found it really easy to use and very flexible.'

Hannah Betts, University College London

'Gorilla is quite simply a revolutionary product for the field of psychology. With so much of the field collecting data over the Internet, the unrivalled flexibility of Gorilla to create any behavioural task quickly and easily is invaluable. On top of that, it provides many complex tasks ready-made, as well as integration with many participant pools like SONA and Prolific.ac. In short, Gorilla really will transform your research and is a fantastic investment.'

Alex Jones, Swansea University

'Massively impressed with Gorilla. Logged on for the first time yesterday lunchtime. Had a demo experiment ready for students to adapt by 6pm the same day.'

Jenni Rodd, Senior Lecturer, UCL

'I use Gorilla in my research on typical and atypical social abilities. This combines experimental tasks and questionnaire measures, in a way that is not possible using other platforms. I found the platform intuitive, easily to learn, but also very powerful. And I have also been extremely impressed with the enthusiasm and (personalized) support from the Gorilla Team, and believe the platform has potential to be the market leader in online experimentation, both in and outside of academic settings!'

Punit Shah, Kings College London

'Gorilla is a great product providing great value to researchers conducting online experiments. The workflow is very clear and cleverly designed. It is a very efficient, flexible and powerful way of creating online experiments. The user interface is super user-friendly and there is practically no learning curve. Thanks for the great support service, too.'

Wing Lau, Xi'an Jiaotong, Liverpool University

'Gorilla lets me take my experiments online quickly and efficiently. The clear and straightforward interface allows me to map out the logic of my experimental designs, whilst its modular nature helps me to build and test components as I go. Taking my research online will dramatically increase my sample size, and help me to reach groups under-represented in my field.

Alexandra Hendry, Kings College London

'I finally had some time to sit through the experiment again yesterday and it looks absolutely amazing! And I’m even more amazed that it took me less than two weeks to set it all up. Thanks for making this possible!'

Eva Port, University College London

Testimonials

Read Testimonials from clients here.


Anonymity and Ethics

In compliance with BPS (The British Psychological Society) requirements, identifying data, demographic information and performance data are all stored separately. They are downloaded separately from the metrics tab and joined together outside Gorilla using the Private IDs provided.

Our database architecture supports double-blind studies; you can join demographic data with performance data while remaining blinded.

If using Gorilla in conjuction with a third party recruitment service, it may be that you do not collect any identifying data. We do not collect IP addresses automatically, in order to ensure participant anonymity.


Data Protection and Security

Gorilla is fully compliant with data protection and security policies.

  • Hosting: Gorilla is hosted on Microsoft Azure.
    • Currently, all our instances are located in their North Europe region, which is within the EU (Republic of Ireland).
    • In future, we may need to expand to other regions as we take on more international clients. This will allow us to keep data storage to particular jurisdictions if that's required from a legal standpoint
    • Microsoft Azure is compliant with ISO/IEC 27001:2005. More details.
  • Traffic Encryption: All traffic to and from Gorilla is encrypted (TLS/SSL)
  • Database Encryption: The database is encrypted using industry-standard cryptography
  • Ownership: The experiment owner owns the research data that has been collected using Gorilla
  • Data Protection: Participant research data can be fully deleted by the researcher. Responsiblity for deletion of participant data falls to the researcher including accidental deletion of participant data. Once data has been deleted it cannot be recovered. Researchers are able to delete all data for an experiment or data pertaining to an individual participant. When this action is taken, data will be removed immediately from the database, and cleared permanently from our automated backups after 14 days.
  • Passwords: We use up-to-date cryptography techniques to handle passwords and user authentication. Passwords are 10 characters long and must contain a reasonable amount of entropy. They are stored as salted hashes in our database to prevent against rainbow table attacks. To prevent brute force attacks, after 3 failed login attempts, users have to wait for 10 seconds before they can try again.
  • GDPR: Gorilla is fully compliant with GDPR.
  • IP Address: IP addresses are not provided in the data download unless specifically included.

GDPR: General Data Protection Regulation


Gorilla is fully compliant with GDPR.

Gorilla is built around the existing BPS (The British Psychological Society) and NIHR (National Institute for Health Research) standards which were far more stringent than the Data Protection Act. Moreover, GDPR does not apply to data that “does not relate to an identified or identifiable natural person or to data rendered anonymous in such a way that the data subject is no longer identifiable.” The majority of our recruitment policies anonymise participants.


The Code Editor


Code in Gorilla is written in TypeScript, which is a typed superset of JavaScript that compiles to plain JavaScript. The reason we chose TypeScript over plain JavaScript is that TypeScript offers a wealth of useful features that make writing code easier, and because many of those features are destined to become part of the ES6 standard which is currently being ratified. This means that JavaScript will effectively become TypeScript, and so your code is future-proofed.

In terms of libraries, JQuery and Bootstrap are included by default.

Go here for more information about the code editor.


Licensing


We have a seat licensing model. Each person signs up with their own email address and effectively has their own account. Each user then has complete control over any task, questionnaire, experiment and associated data that they have authored. This model fits with BPS (The British Psychological Society) requirements around data security; data is only accessible by the person that owns the experiment or those that they are collaborating with. Users can also collaborate on projects. When sharing projects the level of access (read, write, admin) can also be set.

We don't currently have a system for student accounts and supervisors. Any account holder is able to publish their experiments and the onus is on them to ensure they have done so in compliance with their institutions ethics and code of conduct.

What happens when my licence expires?

When your licence expires, your account will revert to a Pay-per-Participant account. All your data, task, questionnaires and experiments will be maintained. You will still have access to all the editing tools and the previewing tools. You just won't be able to collect more data without first purchasing pay-per-participant tokens.

Teaching


Gorilla is an ideal environment for teaching Research Methods, as students can get valuable experience in operationalising experiments, collecting data, and analysing the data collected.

The Experiment Tree makes the experimental design clear, which can often help students understand whether their experiment is adequately controlled. Getting Started: When students are getting started it can be useful to provide them with an experiment where they can change the preconfigured manipulations. For instance, they may change the timing of a task.

  • Beginner: Next, students may start to tweak tasks an questionnaires. They may use different stimuli, to answer a different question. Or add questions to look at alternative correlations.
  • Branching Out: Next, students may start to author their own novel tasks in the code builder.
  • Expert: Finally, students may start to use the Scripting Tools or Code Editor.

We have a suite of tools that allow teachers to manage classrooms. These allow you to:

  • upload lists of class members
  • share resources with them, and
  • receive submissions from them.

For Masters students who may not have the time or inclination to learn to code, Gorilla offers a user-friendly environment in which to author completely novel tasks.

Case Studies

Read our Case Studies about UCL's and Birkbeck's experiences here:

Online Timing Accuracy


When conducting research online, rather than in the lab, it's important to understand how timing accuracy changes. The schematic below shows the main differences:

Modern techniques refers to the Performance.now() function common in all major browsers, which provides microsecond timing. We do not have control of the underlying operating system, so we cannot make the same OS-level timing calls that native software can. While clearly there will always be some experiments that require extremely high fidelity timing, the precision offered by Gorilla is appropriate for a wide range of research.

The latency between the cloud storage and local computer is irrelevant because Gorilla downloads all task information (i.e. stimuli) to the local machine in advance and collects local timestamps.

This article, Woods et al., 2016, is an excellent summary of the strength and weakness of online research.

This article, Hilbig, 2015, presents the effect of lab- versus web-based research on reaction times.

A detailed technical overview of the timing techniques employed in Gorilla is located here.

Server Downtime


Microsoft Azure guarantees that our servers will be working 99.95% of the time. There are 525,600 minutes in a year. That 0.05% when our servers could be down - outside of our control - equates to ~263 minutes a year. This is equivalent to ~2 minutes a month or ~44 seconds a day. At scale, very rare events happen surprisingly often.

Microsoft Azure performs far above this threshold, nevertheless server downtime is a reality of internet research, and we want to give you the information you need to make an informed decision.

  • Short server downtime: Some server downtime will be so short that it will not affect your participants. The participants computer will have stored a few trials ahead, so it’s possible that the server is back up again before the participant needs more information.
  • Longer server downtime: Some server downtime will be long enough that participants notice. It may still be sufficiently short that the participant has to refresh their browser, and in that case it might simply be a question of excluding a trial with a long reaction time or inter trial interval. In this situation, the server downtime has impact similar to your participant being momentarily distracted.
  • Critical server downtime: Some server downtime will be long enough that participants cannot continue at that time. Depending on the recruitment policy and experiment, it might be that they can continue later. On the other hand, it might be that for experimental reason, you can’t use the data.

The risk to your experiment of server downtime will depend on your recruitment method and your target participants.

  • If you are crowdsourcing participant for free - then just ignore this risk.
  • If you are using a recruitment service, then you can mitigate the risk by releasing your study in batches. Generally this isn't necessary - participant pools are so big and the risk is so small that it makes more sense to tollerate the attrition.
  • However, if you are recruiting from a small population (e.g. green-eyed bilinguals), then you may want to recruit in very small batches.

On our side – as long as you haven’t included participants at the start node – no Gorilla fees would be due. If you are paying participants through a participant recruitment service, you may need to check their policy.

Ethics Applications


We're often asked to provide draft text for an ethics application.

Gorilla

We will use Gorilla (www.gorilla.sc) to collect data for our study. Gorilla is a cloud software platform specifically for the behavioural sciences.

  • Hosting: Gorilla is hosted on Microsoft Azure within the EU (Republic of Ireland)
  • Traffic Encryption: All traffic to and from Gorilla is encrypted (TLS/SSL)
  • Database Encryption: The database is encrypted using industry-standard cryptography
  • Data Ownership: The experiment owner owns the research data that has been collected using Gorilla and has complete control over it
  • Data Protection: Gorilla is fully compliant with data protection legislation
  • BPS: Gorilla is fully compliant with BPS guidelines.
  • GDPR: Gorilla is fully compliant with GDPR.
  • IP Address: IP addresses are not provided in the data download unless specifically included.

Recruitment

  • Data Collection: Participants will take part via a desktop computer, laptop, tablet or phone from [anywhere in the world]
  • Consent: Participants will give consent within Gorilla [see supporting documentation]. Participant can opt to not give consent.
  • Recruitment Policy:
    • We will use an anonymous recruitment policy in Gorilla. Consequently, once data is collected it cannot be deleted as it cannot be identified. Participants can still withdraw from the experiment at any time by closing their browser.
      or
    • We will use a recruitment policy in Gorilla that provides participants with a unique an non-identifyable key [ABC123456] that allows them to withdraw their data after completing the experiment.
      or
    • We need to collect data from participants over several days and therefore want Gorilla to email participants to remind them to take part. Consequently participant email addresses will be uploaded to Gorilla.
      • To ensure complete confidentiality and data security, participants are first given a Public ID (ABC123456) which they can use to log in with.
      • Performance data is stored against a Private ID (X1Y2Z345).
      • The relationship between the email address and Public ID is stored separately from performance data.
      • The relationship between the Public ID and Private ID is stored separately from performance data.

Publications and Referencing


Referencing

To refer to Gorilla in an ethics application, grant application or article for publication, please link to the main website or to the About page.

We also recommend stating the date window within which data was collected, so that someone reading the study could cross-references this with our release notes.

Example Text

We used the Gorilla Experiment Builder (www.gorilla.sc) to create and host our study. Data was collected between 01 Jan 2017 and 15 Jan 2017.

44 Participants were testing online using Gorilla (gorilla.sc). Data was collected between 01 Jan 2017 and 15 Jan 2017.

Publications

A list of publications that cite Gorilla can be found here.

Interviews

We also have the following interviews from ambitious researchers that have embraced online research methods.

  • Juliet Usher Smith who ran an interventional study in doctors’ surgeries to measure how patients responded to their cancer risk score with the aim of promoting cancer prevention behaviours.
  • Kyle Jasmin who has pioneered online auditory research in order to access specific and hard to reach populations. His research has shown how people re-calibrate their speech perception system to compensate for specific weaknesses.
  • Claire Gothreau who is integrating behavioural methods into the political sciences. Her research studies the connection between masculinity threat and attitudes to reproductive rights.
  • Adrian Banks who studies how we think, and how we can improve our automatic thinking. His research has implication for how we guard ourselves against fake news.
  • Masa Vujovic who taught participants an artificial language in order to understand the visual and environmental cues that help people learn complex syntax.