Piloting Studies
Top three con­sid­er­a­tions to max­imise data qual­i­ty in behav­iour­al experiments

Are you con­fi­dent your online exper­i­ment will get you the data qual­i­ty you need?

Check out these top three con­sid­er­a­tions when pilot­ing your study.

At BeOn­line 2020 Emily Breese gave an insight­ful talk about pilot­ing research. An often over­looked ele­ment of the research process was brought to the fore­front of our minds as she took 15 min­utes to explain three impor­tant con­sid­er­a­tions to take into account when piloting.

The pri­ma­ry func­tions of pilot­ing a study are to test and val­i­date research pro­to­cols, data col­lec­tion instru­ments, sam­ple recruit­ment strate­gies and other exper­i­men­tal tech­niques, in prepa­ra­tion for a larg­er study.

YouTube

By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

1: Two Step Piloting

Emily sug­gests using two dif­fer­ent groups to pilot your study: one involv­ing colleagues/family/friends and a sec­ond one involv­ing a small rep­re­sen­ta­tive sample.

The ben­e­fit of using the first pilot sam­ple is that it’s cheap and con­ve­nient, and you can ask your par­tic­i­pants to inten­tion­al­ly try and do silly things in your exper­i­ment to high­light any bugs.

It’s impor­tant to also use the sec­ond sam­ple how­ev­er as it will be more rep­re­sen­ta­tive and your par­tic­i­pants will not have the oppor­tu­ni­ty to ask you ques­tions so will like­ly throw up errors that your first sam­ple missed.

If the pop­u­la­tion of prospec­tive par­tic­i­pants is very small, you may need to think about pos­si­ble prac­tice effects result­ing from peo­ple tak­ing part in both the pilot and the real exper­i­ment. If this is a prob­lem then you might need to relax the cri­te­ria you require from the par­tic­i­pants in your sample.

Sug­gest­ed pilot pipeline

2: Thor­ough Piloting

It’s impor­tant to pilot every­thing in your exper­i­ment, even though it might be tempt­ing just to pilot the main task ele­ment. Things like ques­tion­naires and con­sent forms also need pilot­ing to ensure that the instruc­tions are clear and that things work as expected.

It would be real­ly annoy­ing to have the per­fect exper­i­ment set up only to find out that the par­tic­i­pants wasn’t able to select any of the check boxes in the questionnaire!

Not only is it use­ful to check how par­tic­i­pants can respond but also to exam­ine the data out­put so that every­thing you need is being record­ed cor­rect­ly in a way that you can analyse later on.

It’s impor­tant to also use the sec­ond sam­ple how­ev­er as it will be more rep­re­sen­ta­tive and your par­tic­i­pants will not have the oppor­tu­ni­ty to ask you ques­tions so will like­ly throw up errors that your first sam­ple missed.

If the pop­u­la­tion of prospec­tive par­tic­i­pants is very small, you may need to think about pos­si­ble prac­tice effects result­ing from peo­ple tak­ing part in both the pilot and the real exper­i­ment. If this is a prob­lem then you might need to relax the cri­te­ria you require from the par­tic­i­pants in your sample.

3: Exclu­sion Criteria

Final­ly, pilot­ing is impor­tant for inform­ing our deci­sions about exclu­sion cri­te­ria. The data of par­tic­i­pants who meet the exclu­sion cri­te­ria are removed from the analysis.

By test­ing out your exper­i­ment on real peo­ple you might dis­cov­er rea­sons to exclude par­tic­i­pants which you had not thought of before. You might find for exam­ple that cer­tain par­tic­i­pants are press­ing keys ran­dom­ly or as quick­ly as pos­si­ble to get through the tri­als quick­ly. By run­ning the pilot you can get an idea of a cut-off time below which you can assume that par­tic­i­pants have not paid prop­er atten­tion to the study. Exclud­ing such par­tic­i­pants is essen­tial if we want to extract good qual­i­ty data from our exper­i­ment. The ben­e­fit of spec­i­fy­ing exclu­sion cri­te­ria before run­ning the real exper­i­ment is that you can pre-reg­is­ter them, mak­ing any accu­sa­tions of cher­ry-pick­ing data in the final analy­sis more difficult.

Final­ly, and this is often an over­looked ben­e­fit, pilot­ing is impor­tant for inform­ing our deci­sions about exclu­sion cri­te­ria. The data of par­tic­i­pants who meet the exclu­sion cri­te­ria are removed from the analy­sis. By test­ing out our exper­i­ment on real peo­ple we might dis­cov­er rea­sons to exclude par­tic­i­pants which we had not thought of before. You might find for exam­ple that cer­tain par­tic­i­pants are press­ing keys ran­dom­ly or as quick­ly as pos­si­ble to get through the tri­als quick­ly . Exclud­ing such par­tic­i­pants is essen­tial if we want to extract good qual­i­ty data from our experiment.

Con­clu­sion

In short, pilot­ing your study is an essen­tial part of the research process. It may feel bor­ing, espe­cial­ly when you’re so excit­ed to start col­lect­ing data, but the pos­i­tive impact on the qual­i­ty and use­ful­ness of the result­ing data def­i­nite­ly makes it worthwhile.

Always remem­ber:

  1. Pilot two groups if pos­si­ble, colleagues/friends and a small, prefer­ably rep­re­sen­ta­tive sample
  2. Pilot every aspect of your exper­i­ment, not just the main task
  3. Use the pilot­ing to inform your exclu­sion criteria

Are you inter­est­ed in learn­ing more tips about online research meth­ods? Reg­is­ter for the 2021 BeOn­line con­fer­ence now!

Sid Prab­hu-Naik

Sid is a PhD stu­dent based in the Depart­ment of Exper­i­men­tal Psy­chol­o­gy at UCL. He is work­ing part time with Gorilla cre­at­ing a suite of fun games to col­lect research data to bet­ter under­stand some of the cog­ni­tive mech­a­nisms behind lan­guage devel­op­ment. He is also look­ing at how aspects of gam­i­fi­ca­tion itself can con­tribute to more moti­vat­ed, atten­tive, and ulti­mate­ly suc­cess­ful learn­ing strategies.