Are you con­fi­dent your online experiment will get you the data quality you need?

Check out these top three con­sid­er­a­tions when pilot­ing your study.

At BeOn­line 2020 Emily Breeze gave an insight­ful talk about pilot­ing research. An often over­looked element of the research process was brought to the fore­front of our minds as she took 15 minutes to explain three impor­tant con­sid­er­a­tions to take into account when piloting.

The primary func­tions of pilot­ing a study are to test and val­i­date research pro­to­cols, data col­lec­tion instru­ments, sample recruit­ment strate­gies and other exper­i­men­tal tech­niques, in prepa­ra­tion for a larger study.

1: Two Step Piloting

Emily sug­gests using two dif­fer­ent groups to pilot your study: one involv­ing colleagues/family/friends and a second one involv­ing a small rep­re­sen­ta­tive sample.

The benefit of using the first pilot sample is that it’s cheap and con­ve­nient, and you can ask your par­tic­i­pants to inten­tion­al­ly try and do silly things in your experiment to high­light any bugs.

It’s impor­tant to also use the second sample however as it will be more rep­re­sen­ta­tive and your par­tic­i­pants will not have the oppor­tu­ni­ty to ask you ques­tions so will likely throw up errors that your first sample missed.

If the pop­u­la­tion of prospec­tive par­tic­i­pants is very small, you may need to think about pos­si­ble prac­tice effects result­ing from people taking part in both the pilot and the real experiment. If this is a problem then you might need to relax the cri­te­ria you require from the par­tic­i­pants in your sample.

Sug­gest­ed pilot pipeline

2: Thor­ough Piloting

It’s impor­tant to pilot every­thing in your experiment, even though it might be tempt­ing just to pilot the main task element. Things like ques­tion­naires and consent forms also need pilot­ing to ensure that the instruc­tions are clear and that things work as expected.

It would be really annoy­ing to have the perfect experiment set up only to find out that the par­tic­i­pants weren’t able to select any of the check boxes in the questionnaire!

Not only is it useful to check how par­tic­i­pants can respond but also to examine the data output so that every­thing you need is being record­ed cor­rect­ly in a way that you can analyse later on.

3: Exclu­sion Criteria

Finally, pilot­ing is impor­tant for inform­ing our deci­sions about exclu­sion cri­te­ria. The data of par­tic­i­pants who meet the exclu­sion cri­te­ria are removed from the analysis.

By testing out your experiment on real people you might dis­cov­er reasons to exclude par­tic­i­pants which you had not thought of before. You might find for example that certain par­tic­i­pants are press­ing keys ran­dom­ly or as quickly as pos­si­ble to get through the trials quickly. By running the pilot you can get an idea of a cut-off time below which you can assume that par­tic­i­pants have not paid proper atten­tion to the study. Exclud­ing such par­tic­i­pants is essen­tial if we want to extract good quality data from our experiment. The benefit of spec­i­fy­ing exclu­sion cri­te­ria before running the real experiment is that you can pre-reg­is­ter them, making any accu­sa­tions of cherry-picking data in the final analy­sis more difficult.

Con­clu­sion

In short, pilot­ing your study is an essen­tial part of the research process. It may feel boring, espe­cial­ly when you’re so excited to start col­lect­ing data, but the pos­i­tive impact on the quality and use­ful­ness of the result­ing data def­i­nite­ly makes it worthwhile.

Always remem­ber:

1. Pilot two groups if pos­si­ble, colleagues/friends and a small, prefer­ably rep­re­sen­ta­tive sample
2. Pilot every aspect of your experiment, not just the main task
3. Use the pilot­ing to inform your exclu­sion criteria

Are you inter­est­ed in learn­ing more tips about online research methods? Watch over 40 BeOn­line talks from behav­iour­al researchers here.