A com­par­i­son of Tools for Behav­iour­al Research

Mus­ings on Online Behav­iour­al Research

This is an extract from our Gorilla in our Midst paper pub­lished in Behav­ior Research Methods.

Is Gorilla right for me?

In decid­ing which tool to use, the key ques­tions to ask your­self are:

  • Exper­i­ment Design: Do I want to code or use an easy-to-use graph­i­cal user interface?
  • Host­ing: Do I want to pro­cure, man­age and main­tain my own serv­er or use a host­ed service?
  • Data Secu­ri­ty: Do I want to be reson­si­ble for com­ply­ing with data pro­tec­tion laws and reg­u­la­tions or use a host­ed service?
  • Online / Lab: Do I want to want it to work online (in the brows­er) or as installed software?
  • Devices: Do I want it to run on tablets or phones?
  • Periph­er­als: Do I want to inter­face with periph­er­als devices?
  • Recruit­ment: Do I want it to work with wide­ly used recruit­ment ser­vices (Pro­lif­ic, mTurk, SONA)?

Table of tools used for behav­iour­al sci­ence research

Type

Exam­ples

$*

OS*

Descrip­tion

Host­ed exper­i­ment builder

Gorilla

$

CS

Gorilla con­tains a ques­tion­naire builder, GUI task builder, Java Script code edi­tor and an exper­i­ment design tool.

Secure and reli­able exper­i­ment host­ing and data col­lec­tion are part of the ser­vice provided.

You can also host files from other task builders and libraries (i.e., jsPsych, Lab.js) that export to JavaScript with minor mod­i­fi­ca­tion to con­nect to the Gorilla Server.

Par­tic­i­pants can be direct­ed to an exter­nal resource (i.e., Qualtrics) and then return them to Gorilla.

Host­ed sur­vey tools

Qualtrics

Sur­vey­Mon­key

Lime Sur­vey

$

$

$

CS

CS

OS

These allows users to col­lect ques­tion­naire-type data and present media to participants.

They are not designed for col­lect­ing reac­tion time data, for run­ning behav­ioral sci­ence tasks or cre­at­ing com­plex exper­i­men­tal designs.

Cod­ing libraries

Psy­choPy (Python)

jsPsych (JavaScript)

Psy­ch­Tool­Box (Mat­lab)

PyGaze (Python)

F

F

F

OS

OS

OS

These help behav­ioral and neu­roimag­ing researchers cre­ate tasks.

These are built using pro­gram­ming lan­guages. If web-com­pat­i­ble a serv­er and data­base will be need­ed to host these online for data collection.

Task builders

E‑Prime

Pre­sen­ta­tion

Psy­choPy Builder

Open Sesame

Psy­ToolK­it

Lab.js

$

$

F

F

F

F

CS

CS

OS

OS

OS

OS

These are task cre­ation tools. Many of these inter­face with neu­roimag­ing equip­ment and eyetrackers.

Some are more code based (i.e., Psy­ToolK­it), where­as oth­ers pro­vide pre-built tools (i.e., Psy­choPy Builder).

Some pro­vide the abil­i­ty to export JavaScript files (e.g., Psy­choPy Builder and Lab.js) for online host­ing via a 3rd party host­ing solution.

Free tools are often sup­port­ed by com­mu­ni­ty forums, where­as the paid solu­tions have help desks.

Host­ed task builders

Inquisit

Testable

Psy­ToolK­it on the web

$

$

F

CS

CS

OS

These are online task cre­ation tools allow­ing you to build a task for use online, and also pro­vide inte­grat­ed host­ing for that task.

Some are more code based (i.e., Inquisit), where­as oth­ers are more tooled (i.e., Testable). The plat­form pro­vides the host­ing and data col­lec­tion ser­vice for you.

Host­ing solution

Pavlovia

F/$

CS

This is a grant fund­ed and inte­grat­ed host­ing solu­tion for Psy­choPy Builder. You can also host files from other task builders and libraries that export to JavaScript.

Host­ing libraries

JATOS

TATOOL

The Exper­i­ment Factory

F

F

F

OS

OS

OS

Host­ing these libraries requires procur­ing and installing the source code on your own serv­er that you may need to pay for. You will have to man­age any updates to the library and imple­ment any miss­ing func­tion­al­i­ty that you need (e.g., inte­gra­tion with recruit­ment ser­vices). Addi­tion­al­ly, you will need to main­tain the serv­er itself, and per­form your own sys­tem admin­is­tra­tion, secu­ri­ty and backups.

*Key: $, Paid for; F, Free to the user, often depart­ment or grant fund­ed; OS, Open source; CS, Closed source