How To: Experiment Tree


Welcome to the Experiment 'How To' Guide


Here you can learn about the basic features of building Experiments in Gorilla by exploring the list of questions on the left.

Not sure where to start? Try one of these quick-start shortcuts:

Looking for more information on a specific Experiment Tree Node? Check out the Experiment Tree Node Tooling Reference Guide.

If you can't find an answer to your question here please get in touch with us via our contact form. We are always happy to help you, simply tell us a little about what you are trying to achieve and where you are getting stuck.

What is the Experiment Tree?


In Gorilla you create Experiments using the Experiment Tree.

Gorilla uses a graphical drag-and-drop interface to represent your Experiments, which take the form of a tree or flowchart.

You create Experiments by combining together your Questionnaire and Task components as 'Nodes' which you link together to form your experiment tree.

A simple experiment may consist of a consent Questionnaire, a demographics Questionnaire and a test Task.

For a more advanced experiment; there are also powerful Control Nodes such as the Randomiser Node, Branch Node, and Order Node, that support complex experimental designs, all without touching a single line of code!

How do I Start a New Experiment?


A new experiment can be created in a project by pressing the 'Create' button and selecting 'Experiment' and 'Create New'.

On the next screen, choose a name for your Experiment and then press 'OK.'

Alternatively, you can choose to clone an experiment from an existing project such as a sample project or one from your library.

In both cases, you will then be taken to the Design Tab of your newly created (or cloned) Experiment.

Learn more about the Experiment Tree interface here

image of creating new experiment menu

Pro Tip!

When choosing a name for your experiment, try to make it something unique and memorable - a name you would easily associate with the experiment contents.

You will use this name when searching for your experiment in your library. It is also the name people will see if you share your experiment with them, so its important that they would be able to recognise it easily too!

You can add descriptions to your Experiment in the Settings menu, use this feature to add a short reminder of what your experiment is about.

How do I use the Experiment Tree?


The Experiment Tree interface is divided into 4 major sections each found in a separate Tab:

  • Design: This is where you build your experiment. Find out more here.
  • Recruitment: This is where you set your recruitment options. Find out more here.
  • Participants: This is where you view the status of participants you have recruited. Find out more here.
  • Data: This is where you download your experimental data collected from your participants. Find out more here.

Each major section represents a stage in your overall experimental design. Usually you will progress through each of these sections one-after-another from left-to-right.

When you first enter an Experiment you will be presented with the Design Tab as is shown in the image below. The Design Tab is where you create your experimental design.

From this page you can navigate to any of the tabs for your experiment.

Image below shows the Design Tab of the Experiment Tree, with an example of a simple experiment:
Image of Design Tab of Experiment Tree


To learn how to use the Experiment Tree to design and build a simple Experiment, like the one in the example above, click here.

How do I use the Design tab?


In Gorilla you build your Experiments in the Design Tab of the Experiment Tree:

Image below shows the Design Tab of the Experiment Tree:

Understanding the design tab interface:

  1. Name of your Experiment and Description: you can change these in the settings under Name and Description
  2. Settings & Preview Experiment
  3. Version Bar: Version History, current experiment status: either Edit, or Cancel Changes/Commit Version
  4. Design Bar: Add New Node Element, New Node Icons, Check for Errors
  5. Design Space

How do I Build an Experiment in Gorilla


In Gorilla you build your Experiments in the Design Tab of the Experiment Tree:

When you create a new experiment for the first time you'll notice that unlike the Questionaire and Task builders the experiment tree already contains two Nodes: a Start Node and a Finish Node.

When building a new experiment the first step is to add some Nodes, here's how:

  1. Click the Edit button.
  2. Click the Add New Node button, found on the left just above the building area.
  3. A New Node menu will appear.
  4. Select the type of Node you wish to add into your Experiment.
  5. Click 'OK'.

Or

  1. Click the Edit button.
  2. Find the icon of the Node you want to use on the Design Bar
  3. Click the icon.

Once you have added at least one node, you can clone that Node (and its settings) by clicking in the bottom left-hand corner of the Node.

Image below shows the Add New Node Menu:


Image below shows the Icons available on the Design bar:


Image below shows the a Node with the Clone Icon highlighted:

There are currently 15 different Experiment Tree Nodes to choose from, allowing you to present and/or gather data from your participants in a variety of different ways.

What are Nodes?


Experiment Tree Nodes are the building blocks of experiment creation. Building a Longitudinal study or creating a training study? Sophisticated experimental designs are now seconds away!

There are currently 15 different Experiment Tree Nodes for you to choose from, allowing you to perform randomisation, branching and counterbalancing without touching a line of Code! Simply choose from our available Experiment Tree Nodes, drag and drop them into your experiment Tree, and link them together along with your Task and Questionnaire Nodes.

You can create any experiment design you wish by simply combining Experiement Tree nodes in different ways and combinations.
Learn how to add Experiment Tree Nodes into your Experiment Tree Design here.

Broadly speaking the Experiment Tree Nodes fall into three categories: Study Nodes, Core Nodes and Control Nodes.

Below are the Experiment Tree Nodes you will find in Gorilla's Experiment Builder, click on an individual Node to view the dedicated Tooling Reference Guide page:



You can find out more detailed information about each Experiment Tree Node, and how to set them up, in the Tooling Reference Guide.

How do I add Tasks and Questionnaires to an Experiment?


Task

The task node is blue with a in the top left corner. It has a single connection point.


Double clicking on the nodes will open the node modal screen shown below.


It includes the standard save and remove buttons as well as preview and options. Additionally, if your task has any manipulations available, these can be set from here as well.

Questionnaire

The questionnaire node is green with a in the top left corner. It has a single connection point.


Double clicking on the nodes will open the node modal screen shown below.


It includes the standard save and remove buttons as well as preview and options. Additionally, there is the choice to randomise the elements of the questionnaire. The questionnaire can be fully randomised or have all nodes but the first randomised. This is particularly useful if the first element in the questionnaire is a markdown item which provides the instructions to the participant.

Note: If you make changes to your Task or Experiment after you have added it to the Experiment Tree, you will need to update the Task/Questionnaire Node to the latest version

To do this, click on the Node, click 'Options' in the bottom left-hand corner, then 'Update to latest version'. If your Task/Questionnaire is not the latest version, an orange warning triangle will appear next to the 'Options' button.

What are Core Nodes?


Core Nodes are structural elements of your task. This includes Start Nodes, Finish Nodes, Reject Nodes, Checkpoint Nodes, and Delay nodes.

These primarily control how participants exit and enter your task.

Reject Nodes allow you to reject participants who are not suitable for your experiment, or who withdraw their participation. Checkpoint nodes allow you to monitor how far along a participant is in your experiment, which can be useful for longitudinal studies or for managing attrition.

Core Nodes can be added from the design bar in the same way as Task and Questionnaire Nodes, and have Node modal screens that may require configuration.

One Start Node and one Finish Node are automatically added to any new Experiment. However, it is possible to have multiple Start or multiple Finish Nodes. For more information about this, see the Start and Finish Node pages in the Tooling Reference Guide.

To learn more about each Node, and how to set them up, see the Experiment Tree Nodes Tooling Reference Guide.

What are Control Nodes?


Control Nodes allow you to manipulate the path of participants through your experiment.

Some Control Nodes, such as the Repeat Node and Switch Node affect a single path of participants. These nodes allow you to, for example, ensure participants repeat a task, or are able switch between tasks.

Other Control Nodes, such as Branch Nodes, Randomiser Nodes and Counterbalance Nodes, allow you to divide participants into different conditions. Different participants can then be shown different tasks/questionnaires, or different versions of the same task/questionnaire.

Control Nodes can be added from the design bar in the same way as Task and Questionnaire Nodes, and have Node modal screens that may require configuration.

To learn more about each Node, and how to set them up, see the Experiment Tree Nodes Tooling Reference Guide.

Recruitment


Gorilla does not recruit participants for you, however, you can link an external recruitment service to your Gorilla Experiment, create a link to distribute, or invite participants you already know to participate. The recruitment section is where you configure the the method by which participants will access your experiment, optionally restrict the devices, browsers or location they can take part from, and control how many participants you wish to recruit.

Recruitment Policy

The recruitment policy you choose determines how participants will access your experiment. There are several options here: a simple link that you can put on your website or post to social media, uploading a CSV of email addresses and inviting them all to take part, or interfacing with other recruitment systems such as Prolific.ac, MTurk, SONA, or Research Now.

Click here for a full list of recruitment policies.

Recruitment Target

The recruitment target is the number of participants you wish to recruit. All participants who are marked as included on your participants page will count towards this total.

If you are on an Unlimited account, you can choose to set your recruitment target as Unlimited. This will continue to accept participants until you end the experiment by setting the recruitment policy to Disabled.

If you are on a Pay-as-you-go or Standard account, you must set a recruitment target. This will transfer the appropriate number of tokens from your account to the experiment.

Participant Tokens can be either ‘Fulfilled’ or ‘Reserved’. Participants who have been included (complete, manually included, or included at start) are considered ‘Fulfilled’, and all live or non-included participants are considered ‘Reserved’.

When you have a recruitment target set, the experiment will continue to accept participants until the combined number of fulfilled and reserved participants reaches your recruitment target. The Recruitment Status is then 'Full'. If any participant holding a ‘Reserved’ token is subsequently excluded (either by a Reject node, a Time Limit or manual exclusion), their reserved token will be returned, and the experiment will subsequently accept another participant until the total is reached. However, their token will not be returned if they have previously been included. The Recruitment Status will then be 'Complete'.

To find out more about the recruitment target, take a look at the How To: Participant Tokens guide.

Important Information!

Only the Project Owner can change the recruitment target settings. Only the Project Owner's tokens can be assigned to the experiment. Collaborators are not able to interact with these settings, so cannot contribute any of their own tokens or use their unlimited licence (if they have one).

Requirements

You can optionally restrict your participants by device type, connection speed, browsers or geographic location. Any participants not meeting these criteria will be shown an error message.

Time Limit

Time Limit, found on your experiment recruitment page, allows you to automatically reject participants who do not complete your experiment, or who take longer to complete it than is considered reasonable.

Once a Time Limit is set, in hours and minutes, participants who reach the Time Limit will be automatically rejected, but will be allowed to finish their current task, before being redirected to the Finish Node. You may wish to set a Time Limit because ‘Live’ participants reserve tokens, contributing towards your recruitment target. This means that participants who drop out without finishing your experiment can prevent more participants from entering your experiment until they are rejected.

Whilst you can reject participants manually, this requires monitoring your recruitment progress closely. Instead, you may choose to set a Time Limit to automate this process.

We suggest setting a Time Limit that is far longer than it could reasonably take to complete your experiment. For example, If your experiment should take 15 minutes to complete, you might set your time limit at 2 hours.

For this reason, we do not recommend using Time Limits for longitudinal studies. In a longitudinal study, the reasons for taking a long time to complete a study are much more numerous, which makes the padding you'd want to give the time limit excessively large and hard to estimate. When you can see your attrition and rejection numbers, you may wish to revise your Time Limit, and would then have to manually include the participants you’d automatically rejected.

Additionally, depending on ethics and your recruitment service, you will likely still have to pay participants who only complete the first half of your study for completing the first section, so you may wish to make use of their data.

Note: When using the Time Limit with recruitment services that offer a similar Time Limit, make sure that your Gorilla experiment Time Limit matches the time limit set in the recruitment service.

I've launched my experiment can I still make changes?


Can I change recruitment policies at any time?

It is possible to change a recruitment policy at any time. However, switching between policies that do or don't require public ids can cause disruption to any current participants. For example, if participants have originally been sent a simple link and the recruitment policy is subsequently changed to require a public id or login, those simple links will no longer work. Consider only changing the recruitment policy once a trial of the experiment has run successfully, or sending out updated invites to existing participants.

How can I change the requirements of my task?

By default, participants can perform an experiment on any device from anywhere in the world. If necessary, it is possible to restrict the circumstances under which a participants can perform an experiment. These requirements consist of: limiting device types to phones, tablets and/or computers; limiting to a geographical location via a 2-letter country code; limiting the browser used to Chrome, Safari, Edge, Firefox and/or Internet Explorer; and limiting to a minimum connection speed. Any participant who doesn't meet the criteria below will be shown a default page explaining why they cannot proceed. If they log in later and meet the criteria (e.g. because they have switched from their phone to their tablet), they will be able to proceed as normal.

Participants


Participants

The participants screen allows you to observe and manage the participants who have been invited to, are registered to or have completed your task. If you are using a simple link or pilot recruitment policy, this list of participants will be populated as people first log into the task and will indicate their progress. If you are using the email shot, email id or supervised recruitment policies, the pre-prepared list of participants, email addresses or public ids will appear in 'Participants' before the participant has logged in for the first time. If necessary, there will be an option on this page to 'activate' the participant, which will send them the initial recruitment email and any login details.
Participants can also be rejected, included or deleted from this page. To find out more, see the How To: Participant Tokens guide.

The Data Tab: Consort Data


The data tab now includes information about the state of participants at each Node. This means that you can see where participants have dropped out, been rejected, and gain detailed attrition data.

e.g. a participant has been through three nodes before being rejected. The participant will be shown as entering and exiting three nodes, and then entering the node at which they were rejected and shown as rejected at that node.

The Number of Participants who have entered the node

The Number of Participants who are still live.

The Number of Participants who were rejected

The Number of Participants who were deleted

The Number of Participants who have exited the node

Here we have 17 participants entering the node and 16 exiting, with one remaining live. It may be that the participant has left the experiment at this point, and we may wish to manually reject them. This will set the number of ‘Live’ participants to 0, and the number of rejected participants to 1.

The Data Tab: How Do I Download my Metrics


Your experiment Data page allows you to download data from the various Task and Questionnaire Nodes of your experiment in the form of a metrics spreadsheet.

In compliance with BPS (The British Psychological Society) and NIHR (National Institute of Health Regulations) we store data from each node separately. This way demographics data and performance data are always kept separately.

Data is presented in long-format, with one row per-event. There is an option, however, to download questionnaire data in short-format, with one row per-participant.

Note: Script widget data will not be displayed in short form. If you have used a script widget in your questionnaire, download the long-form version.

Image below shows the Data Tab of the Experiment Tree:

To download data from all Nodes:

  1. From the Data Tab, click ‘Manage Experiment Data’. This will open the data-download menu.
  2. Select your preferred Filetype.
  3. Select the Timeframe you wish to collect data from.
  4. Click ‘Generate Data’.
  5. Click ‘Download Data’.

To download data from one Node:

  1. From the Data Tab, click on any Node in your Experiment Tree to open its individual data-download menu.
  2. Select your preferred Filetype.
  3. Select the Timeframe you wish to collect data from
  4. Click ‘Generate Data’.
  5. Click ‘Download Data’.
Image below shows the data-download menu

For more information on data format and analysis, take a look at the How To: Metrics Guide.