Randomisation and Attrition in Gorilla


What follows is a description of some of the complexities surrounding randomisation in Gorilla.

In short, there are two types of randomisation: with replacement and without replacement. These differ in how they allocate participants to different conditions and it’s important to understand these differences.


RANDOM WITH REPLACEMENT (RANDOM)


The best way to think about random with replacement is that it is like a coin toss, or dice roll. Each coin toss is independent of what has happened before.

If we have a randomiser going to two groups, it’s like tossing a coin for each participant. On average, we would expect a 50:50 ratio of head to tails, but you could get runs of all heads by sheer (bad) luck. We are unlikely to end up with a ratio of exactly 50:50, but it will probably be close. The larger the sample, the closer the ratio will get.

This is called WITH REPLACEMENT because each time a participant is assigned a group, Heads or Tails, all options are still available for the next participant. Each allocation is independent of all previous allocations.


RANDOM WITHOUT REPLACEMENT (BALANCED)


The best way to think about the balanced randomiser is like a deck of cards. Each draw is dependent on what has been drawn before.

The colour of the card (red or black) determines which branch participants go down. Balanced with a 2:2 ratio means we have 4 cards, 2 are black and 2 are red. Cards are handed out in a random order (BBRR). In each lot of 4 cards, we will have the 2:2 ratio exactly. For the next four participants, the process is repeated. So over 12 participants we would get 3 lots of 4 cards, with 6 Red and 6 Black. For instance, the three lots could be; BRBR then RRBB then RBBR. Importantly, consider the 1st lot. Once we’ve had BRB, the last card has to be R. There is no chance of it being anything else.

A 2:2 ratio is different from 1:1 ratio. With 1:1 there are only 2 cards, 1 red and 1 black, so there are only two possible orders (BR and RB). So over 12 participants we might get BR, RB, BR, RB, BR, BR.

A 10:10 ratio means we have 20 cards, 10 red and 10 black. We could (by chance) have a run of 10 red followed by 10 black, then 10 black and 10 red. In that unlikely event, the only times we have a balanced set of participants is at 20 and 40 Ns. For example, if we only had 15 Ns, the first 10 would be allocated to red, leaving us with only 5 allocated to black.

With a 2:2 ratio, one group can only get 2 ahead of the other group. If you have already sampled 100 participants equally into black and red and you only want to sample two more, it may be that these last two both get red cards (RRBB), leading to 50 black and 52 red.

With 1:1 ratio this goes down to 1. If you have already sampled 100 participants equally into black and red and you only want to sample two more, one will be black and one red.

With 10:10 ratio, this goes up to 10. This is important because a larger ratio increases your chance of unequal conditions if you end recruitment early.

This is called RANDOM WITHOUT REPLACEMENT because in each lot of participants (2, 4, 10) cards are dealt out without replacing the cards that have been taken by previous participants. At the end of each lot we will have dealt participants the exact ratio we set. Allocations are dependent on all previous allocations.


ATTRITION


In Gorilla the randomiser node, order node and counterbalance node have no knowledge of subsequent attrition. Attrition is when a participant drops out of your experiment part way through. This means that even with balanced randomisation, you may end up with unequal groups if participants drop out. Drop-out can be caused by a range of factors: participants can get bored, have their attention called elsewhere, dislike the task, or stop participating for whatever reason.

Worked example:

  1. Imagine we have a simple between subject experiment with two groups. The Randomiser leads to Group A and Group B via two checkpoint nodes. We want 12 participants overall and the randomiser is set to 2:2 Balanced (without replacement).
  2. We launch the experiment.
  3. Participants come into our experiment and get assigned as follows: AABB, ABAB, BABA
  4. Our experiment is now full and participants can no longer join. Great!
  5. Participants start completing our experiment. Participants AABB, BB, BABA all complete. 2 remain live both on the A branch. It might be that these participants have got bored and wandered off.

Two scenarios can happen next.


Scenario 1: We have a Timelimit set, no manual intervention

  1. After the appropriate time, both remaining live participants are automatically rejected.
  2. Our experiment is no longer full. New participants join.
  3. Gorilla generates a new block of participants (BBAA). Remember, the randomiser has no knowledge of subsequent attrition.
  4. The first participant starts and immediately drops out handing back their token. That’s the first B of our new BBAA lot used.
  5. The next participant who joins takes up the next B on our BBAA lot and goes down the B branch and completes the task.
  6. The next participant goes down the A branch and completes the task.

Summary:

At the end we have 12 completes, 5 on the A branch and 7 on the B branch.

We might report this as: 12 participants randomised to condition (5 condition A, 7 condition B) completed the study. 14 participants were recruited overall, 2 participants dropped out of the A condition.


Scenario 2: We don’t have a Timelimit set, manual intervention.

  1. Gorilla has sent us an email saying our experiment is full. We leave it 2 hours, so allow people to complete. After this time we are happy to manually reject Live participants as they have probably dropped out.
  2. We look at our participants dashboard and see that we have 6 completes on the B branch and 4 completes on the A branch.
  3. We edit our experiment and change the randomiser ratio from A2:B2 to A2:B0.
  4. We reject the two live participants.
  5. Recruitment resumes, and we get two participants that are sent down the A branch by our new randomiser ratio and complete the task.

Summary: At the end we have 12 completes, 6 on the A branch and 6 on the B branch.

We report this as 12 participants were randomly assigned to groups and recruited until there were 6 participants that completed each condition.


Conclusion

In scenario 2 (above), we ended up with the ideal number of participants in each condition. But we have lost information about attrition. In scenario 1 we end up with small differences in group size, but we have the information about attrition. Neither are ideal, but good science is often about compromises.

Before we look at possible solutions, lets take a deeper look at attrition and why it is important.


ASYMMETRIC ATTRITION


To take an extreme example, imagine we have two groups, one with a positive mood induction (kittens and puppies) and one with a negative mood induction (spiders and snakes) followed by a probabilistic discounting task. Our hypothesis is that a poor mood makes people favour money now over money later, whereas a good mood pushes this horizon further out.

In an ideal condition, we’d recruit 50 participants to each group and they would all complete. But this is the real world, and that’s unlikely to happen. Importantly, we'd expect a lot more attrition in the negative mood group.

Three scenarios can happen next.


Scenario 1: We have a Timelimit set, no manual intervention.

By the time we have 100 completes, our methods section might read as follows:

100 participants completed the study. They were randomised to condition (70 condition A, 30 condition B). Overall, 200 participants were recruited;10 participants dropped out of the A condition and 90 dropped out of the B condition.

This tells the reader that we had significant asymmetric attrition and that the negative mood induction group are probably systematically different to the positive mood induction group. They have self-selected for being less sensitive to spiders and snakes.


Scenario 2: We have don't a Timelimit set, manual intervention.

100 participants were randomly assigned to groups and recruited until there were 50 participants that completed each condition.

The asymmetric attrition information is hidden from the reader.


Scenario 3: Recruit in 3 phases changing the randomiser ratio

An alternative, and elegant approach is to split our recruitment into 2 phases.

In phase 1 we collect 50 Ns worth of data at a 4:4 ratio. We then determine the attrition rate in each group and adjust the ratios accordingly.

In phase 2 we collect 50 Ns worth of data at the 4:8 or 4:12 ratio – whatever fits our attrition rate. By doing so we aim to get both groups to 50 completes at about the same time.


QUOTAS


We have launched a new feature in Gorilla called Quota Nodes. This allow us to set a quota on a branch of our experiment. Once the quota is full, participants will no longer be sent down that branch. This will give us the effect of setting randomiser branches to 0 once we have enough participants for that condition without the need for manual intervention.

This feature should be used with care so that we don’t hide asymmetric attrition from ourselves or others.


ATTRTITION SENSITIVE NODES


(coming soon)

We are working on an improvement to Randomiser, Order and Counterbalance nodes that allows them to be sensitive to subsequent attrition.

Going back to the original example:

  1. Imagine we have a simple between subject experiment with two groups. The Randomiser leads to Group A and Group B via two checkpoint nodes. We want 12 participants overall and the randomiser is set to 2:2 Balanced.
  2. We launch the experiment.
  3. Participants come into our experiment and get assigned as follows: AABB, ABAB, BABA
  4. Our experiment is now full and participants can no longer join. Great!
  5. Participants start completing our experiment. Participants AABB, BB, BABA all complete. 2 remain live both on the A branch. It might be that these participants have got bored and wandered off.
  6. After the appropriate time, both these participants are automatically rejected.
  7. Our experiment is no longer full.
  8. New participants join.
  9. Gorilla checks to see if any branch assignments have been returned due to attrition. They have (AA).
  10. These assignments are handed out to new participants until they are consumed. This approach will give us complete information on how many participants were recruited, how many completed and how much attrition there was in each condition. This feature is coming soon. On the face of it, it’s simple. But we also need to factor in Version Control across experiments and how to manage this.