A comparison of Tools for Behavioural Research

Musings on Online Behavioural Research

This is an extract from our Gorilla in our Midst paper published in Behavior Research Methods.

Is Gorilla right for me?

In deciding which tool to use, the key questions to ask yourself are:

  • Experiment Design: Do I want to code or use an easy-to-use graphical user interface?
  • Hosting: Do I want to procure, manage and maintain my own server or use a hosted service?
  • Data Security: Do I want to be resonsible for complying with data protection laws and regulations or use a hosted service?
  • Online / Lab: Do I want to want it to work online (in the browser) or as installed software?
  • Devices: Do I want it to run on tablets or phones?
  • Peripherals: Do I want to interface with peripherals devices?
  • Recruitment: Do I want it to work with widely used recruitment services (Prolific, mTurk, SONA)?

Table of tools used for behavioural science research

Type

Examples

$*

OS*

Description

Hosted experiment builder

Gorilla

$

CS

Gorilla contains a questionnaire builder, GUI task builder, Java Script code editor and an experiment design tool.

Secure and reliable experiment hosting and data collection are part of the service provided.

You can also host files from other task builders and libraries (i.e., jsPsych, Lab.js) that export to JavaScript with minor modification to connect to the Gorilla Server.

Participants can be directed to an external resource (i.e., Qualtrics) and then return them to Gorilla.

Hosted survey tools

Qualtrics

SurveyMonkey

Lime Survey

$

$

$

CS

CS

OS

These allows users to collect questionnaire-type data and present media to participants.

They are not designed for collecting reaction time data, for running behavioral science tasks or creating complex experimental designs.

Coding libraries

PsychoPy (Python)

jsPsych (JavaScript)

PsychToolBox (Matlab)

PyGaze (Python)

F

F

F

OS

OS

OS

These help behavioral and neuroimaging researchers create tasks.

These are built using programming languages. If web-compatible a server and database will be needed to host these online for data collection.

Task builders

E-Prime

Presentation

PsychoPy Builder

Open Sesame

PsyToolKit

Lab.js

$

$

F

F

F

F

CS

CS

OS

OS

OS

OS

These are task creation tools. Many of these interface with neuroimaging equipment and eyetrackers.

Some are more code based (i.e., PsyToolKit), whereas others provide pre-built tools (i.e., PsychoPy Builder).

Some provide the ability to export JavaScript files (e.g., PsychoPy Builder and Lab.js) for online hosting via a 3rd party hosting solution.

Free tools are often supported by community forums, whereas the paid solutions have help desks.

Hosted task builders

Inquisit

Testable

PsyToolKit on the web

$

$

F

CS

CS

OS

These are online task creation tools allowing you to build a task for use online, and also provide integrated hosting for that task.

Some are more code based (i.e., Inquisit), whereas others are more tooled (i.e., Testable). The platform provides the hosting and data collection service for you.

Hosting solution

Pavlovia

F/$

CS

This is a grant funded and integrated hosting solution for PsychoPy Builder. You can also host files from other task builders and libraries that export to JavaScript.

Hosting libraries

JATOS

TATOOL

The Experiment Factory

F

F

F

OS

OS

OS

Hosting these libraries requires procuring and installing the source code on your own server that you may need to pay for. You will have to manage any updates to the library and implement any missing functionality that you need (e.g., integration with recruitment services). Additionally, you will need to maintain the server itself, and perform your own system administration, security and backups.

*Key: $, Paid for; F, Free to the user, often department or grant funded; OS, Open source; CS, Closed source