Check out these top three considerations when piloting your study.

At BeOnline 2020 Emily Breeze gave an insightful talk about piloting research. An often overlooked element of the research process was brought to the forefront of our minds as she took 15 minutes to explain three important considerations to take into account when piloting.

The primary functions of piloting a study are to test and validate research protocols, data collection instruments, sample recruitment strategies and other experimental techniques, in preparation for a larger study.

1: Two Step Piloting

Emily suggests using two different groups to pilot your study: one involving colleagues/family/friends and a second one involving a small representative sample.

The benefit of using the first pilot sample is that it’s cheap and convenient, and you can ask your participants to intentionally try and do silly things in your experiment to highlight any bugs.

It’s important to also use the second sample however as it will be more representative and your participants will not have the opportunity to ask you questions so will likely throw up errors that your first sample missed.

If the population of prospective participants is very small, you may need to think about possible practice effects resulting from people taking part in both the pilot and the real experiment. If this is a problem then you might need to relax the criteria you require from the participants in your sample.

2: Thorough Piloting

It’s important to pilot everything in your experiment, even though it might be tempting just to pilot the main task element. Things like questionnaires and consent forms also need piloting to ensure that the instructions are clear and that things work as expected.

It would be really annoying to have the perfect experiment set up only to find out that the participants weren’t able to select any of the check boxes in the questionnaire!

Not only is it useful to check how participants can respond but also to examine the data output so that everything you need is being recorded correctly in a way that you can analyse later on.

3: Exclusion Criteria

Finally, piloting is important for informing our decisions about exclusion criteria. The data of participants who meet the exclusion criteria are removed from the analysis.

By testing out your experiment on real people you might discover reasons to exclude participants which you had not thought of before. You might find for example that certain participants are pressing keys randomly or as quickly as possible to get through the trials quickly. By running the pilot you can get an idea of a cut-off time below which you can assume that participants have not paid proper attention to the study. Excluding such participants is essential if we want to extract good quality data from our experiment. The benefit of specifying exclusion criteria before running the real experiment is that you can pre-register them, making any accusations of cherry-picking data in the final analysis more difficult.

Conclusion

In short, piloting your study is an essential part of the research process. It may feel boring, especially when you’re so excited to start collecting data, but the positive impact on the quality and usefulness of the resulting data definitely makes it worthwhile.

Always remember:

1. Pilot two groups if possible, colleagues/friends and a small, preferably representative sample

2. Pilot every aspect of your experiment, not just the main task

3. Use the piloting to inform your exclusion criteria

   Are you interested in learning more tips about online research methods? Watch over 40 BeOnline talks from behavioural researchers here.