A com­par­i­son of Tools for Behav­iour­al Research

Musings on Online Behav­iour­al Research

This is an extract from our Gorilla in our Midst paper pub­lished in Behav­ior Research Methods.

Is Gorilla right for me?

In decid­ing which tool to use, the key ques­tions to ask your­self are:

  • Experiment Design: Do I want to code or use an easy-to-use graph­i­cal user interface?
  • Hosting: Do I want to procure, manage and main­tain my own server or use a hosted service?
  • Data Secu­ri­ty: Do I want to be respon­si­ble for com­ply­ing with data pro­tec­tion laws and reg­u­la­tions or use a secure and com­pli­ant hosted service?  Paid for service usually offer better com­pli­ance with infor­ma­tion secu­ri­ty requirements.
  • Online / Lab: Do I want to want it to work online (in the browser) or as installed software?
  • Devices: Do I want it to run on tablets or phones?
  • Periph­er­als: Do I want to inter­face with periph­er­als devices?
  • Recruit­ment: Do I want it to work with widely used recruit­ment ser­vices (Pro­lif­ic, mTurk, SONA)?
  • Support:  Do I want access to a ded­i­cat­ed support desk and train­ing materials?

Table of tools used for behav­iour­al science research

Type

Exam­ples

$*

OS*

Descrip­tion

Hosted experiment builder

Gorilla

$

CS

Gorilla con­tains a ques­tion­naire builder, GUI task builder, Java Script code editor and an experiment design tool.

Secure and reli­able experiment hosting and data col­lec­tion are part of the service provided.

You can also host files from other task builders and libraries (i.e., jsPsych, Lab.js) that export to JavaScript with minor mod­i­fi­ca­tion to connect to the Gorilla Server.

Par­tic­i­pants can be direct­ed to an exter­nal resource (i.e., Qualtrics) and then return them to Gorilla.

Hosted survey tools

Qualtrics

Sur­vey­Mon­key

Lime Survey

$

$

$

CS

CS

OS

These allows users to collect ques­tion­naire-type data and present media to participants.

They are not designed for col­lect­ing reac­tion time data, for running behav­ioral science tasks or cre­at­ing complex exper­i­men­tal designs.

Coding libraries

Psy­choPy (Python)

jsPsych (JavaScript)

Psy­ch­Tool­Box (Matlab)

PyGaze (Python)

F

F

F

OS

OS

OS

These help behav­ioral and neu­roimag­ing researchers create tasks.

These are built using pro­gram­ming lan­guages. If web-com­pat­i­ble a server and data­base will be needed to host these online for data collection.

Task builders

E‑Prime

Pre­sen­ta­tion

Psy­choPy Builder

Open Sesame

Psy­ToolK­it

Lab.js

$

$

F

F

F

F

CS

CS

OS

OS

OS

OS

These are task cre­ation tools. Many of these inter­face with neu­roimag­ing equip­ment and eyetrackers.

Some are more code based (i.e., Psy­ToolK­it), whereas others provide pre-built tools (i.e., Psy­choPy Builder).

Some provide the ability to export JavaScript files (e.g., Psy­choPy Builder and Lab.js) for online hosting via a 3rd party hosting solution.

Free tools are often sup­port­ed by com­mu­ni­ty forums, whereas the paid solu­tions have help desks.

Hosted task builders

Inquisit

Testable

Psy­ToolK­it on the web

$

$

F

CS

CS

OS

These are online task cre­ation tools allow­ing you to build a task for use online, and also provide inte­grat­ed hosting for that task.

Some are more code based (i.e., Inquisit), whereas others are more tooled (i.e., Testable). The plat­form pro­vides the hosting and data col­lec­tion service for you.

Hosting solu­tion

Pavlovia

F/$

CS

This is a grant funded and inte­grat­ed hosting solu­tion for Psy­choPy Builder. You can also host files from other task builders and libraries that export to JavaScript.

Hosting libraries

JATOS

TATOOL

The Experiment Factory

F

F

F

OS

OS

OS

Hosting these libraries requires procur­ing and installing the source code on your own server that you may need to pay for. You will have to manage any updates to the library and imple­ment any missing func­tion­al­i­ty that you need (e.g., inte­gra­tion with recruit­ment ser­vices). Addi­tion­al­ly, you will need to main­tain the server itself, and perform your own system admin­is­tra­tion, secu­ri­ty and backups.

*Key: $, Paid for; F, Free to the user, often depart­ment or grant funded; OS, Open source; CS, Closed source