Online vs lab-based behavĀ­ioral research: Letting go of the illuĀ­sion of control

Some researchers have resistĀ­ed the move to online research over the last few years, but the COVID crisis has forced many to switch to online methods. However, one quesĀ­tion keeps coming up: ā€œJo, is there a good way to monitor the parĀ­ticĀ­iĀ­pant enviĀ­ronĀ­ment when testing remotely?ā€

The fear of losing control of the testing enviĀ­ronĀ­ment when taking research online is real, so let’s address it.

But first, let’s look at the benĀ­eĀ­fits of online and lab research to get on the same page before we look into what’s posĀ­siĀ­ble in terms of enviĀ­ronĀ­menĀ­tal monĀ­iĀ­torĀ­ing — and what I think may be a better approach.

BenĀ­eĀ­fits of behavĀ­ioral online research: Speed, reach, scale

  1. Online data colĀ­lecĀ­tion can be comĀ­pletĀ­ed at an incredĀ­iĀ­ble speed. The tools for online research are now so good, that it can take only a few hours to create a study. Long gone are the days of painfulĀ­ly coding both the parĀ­ticĀ­iĀ­pant and server-side expeĀ­riĀ­ence. Couple behavĀ­ioral science softĀ­ware with any number of parĀ­ticĀ­iĀ­pant recruitĀ­ment serĀ­vices and you can see the data come flying in.
  2. Go large scale and say goodbye to underĀ­powĀ­ered studies. As you no longer need to sit in the lab with each parĀ­ticĀ­iĀ­pant, mulĀ­tiĀ­ple parĀ­ticĀ­iĀ­pants can comĀ­plete your experiment simulĀ­taĀ­neĀ­ousĀ­ly, leading to much larger samples. With experiment softĀ­ware, you can get data from thouĀ­sands of parĀ­ticĀ­iĀ­pants in a day.
  3. Extend your reach and recruit the parĀ­ticĀ­iĀ­pants you need. Do you need a more diverse sample? Or a really speĀ­cifĀ­ic group of parĀ­ticĀ­iĀ­pants? InteĀ­grate your experĀ­iĀ­menĀ­taĀ­tion platĀ­form with a recruitĀ­ment service like ProĀ­lifĀ­ic or SONA and reach groups that you couldn’t have done in the lab.

BenĀ­eĀ­fits of behavĀ­ioral lab research: Proxy for parĀ­ticĀ­iĀ­pant attention

So with all these benĀ­eĀ­fits, why do we stay in the lab? Control!

As researchers, we like to feel like we’re in control in the lab. We want control over the enviĀ­ronĀ­ment because (1) we had it in the lab and (2) we use it as a proxy for parĀ­ticĀ­iĀ­pant attenĀ­tion.

We may fear that remainĀ­ing online removes this sense of control. It seems scary to have to trust that our parĀ­ticĀ­iĀ­pants will pay attenĀ­tion to the task we give them — espeĀ­cialĀ­ly if we’re not there to keep things on track. It’s scary to think about all the reasons why we may need to exclude parĀ­ticĀ­iĀ­pants and to come up with a list of pre-defined excluĀ­sion criteria.

But in reality, these are things we should be thinkĀ­ing about anyway. Perhaps we don’t have full control in the lab after all — perhaps the control is just an illuĀ­sion. You had control of the enviĀ­ronĀ­ment, but you never had control of their mind.

The illuĀ­sion of control: The mind is free

When a parĀ­ticĀ­iĀ­pant comes into the lab we can interĀ­act with them and watch them comĀ­plete the task. We can make sure they are in a quiet, disĀ­tracĀ­tion-free room, and sit in a senĀ­siĀ­ble workspace.

Yet, we cannot control where their attenĀ­tion is focused. They may look like they are paying attenĀ­tion to the task, but perhaps they are dayĀ­dreamĀ­ing or just not taking it seriĀ­ousĀ­ly, and you can often only see this in the data later on in the research process.

Online we can ask parĀ­ticĀ­iĀ­pants to find a quiet space, but we can never be sure if they have done this. Again this is someĀ­thing that we wouldn’t necĀ­esĀ­sarĀ­iĀ­ly spot until we look at the data.

EnviĀ­ronĀ­menĀ­tal MonĀ­iĀ­torĀ­ing is problematic

Of course, we could with the consent of our parĀ­ticĀ­iĀ­pants, interĀ­leave task trials with short bursts of recordĀ­ing the backĀ­ground audio (with the audio zone) and video of the home enviĀ­ronĀ­ment (from the webcam).

However:

  1. Your parĀ­ticĀ­iĀ­pants may not like this at all! They could rightly be worried about secuĀ­riĀ­ty. And the very act of asking for this sets up an antagĀ­oĀ­nisĀ­tic relaĀ­tionĀ­ship with your parĀ­ticĀ­iĀ­pant. I’ve written before about the imporĀ­tance of making parĀ­ticĀ­iĀ­pants a research partner and treatĀ­ing them with respect.
  2. Your ethics comĀ­mitĀ­tee might not like this at all! We’re now colĀ­lectĀ­ing perĀ­sonĀ­alĀ­ly idenĀ­tiĀ­fyĀ­ing data and colĀ­lectĀ­ing data that isn’t necĀ­esĀ­sarĀ­iĀ­ly relĀ­eĀ­vant to the task. In terms of secuĀ­riĀ­ty, a good rule of thumb is to collect the minimum data posĀ­siĀ­ble. This goes against that rule of thumb.
  3. You will now have to watch and listen to all these files (it can’t be autoĀ­matĀ­ed), and you might regret your choice espeĀ­cialĀ­ly when there are better and more autoĀ­matĀ­ed ways to achieve the same results. Read on!

PilotĀ­ing and pre-regĀ­isĀ­terĀ­ing is the way to go

So, someĀ­thing can happen both in lab research and online research… and we want to deal with the issue. The best way to do this is through strong pilotĀ­ing of your study and working out objecĀ­tive excluĀ­sion criĀ­teĀ­ria based on data quality. From this, we can pre-regĀ­isĀ­ter our criĀ­teĀ­ria strengthĀ­enĀ­ing the trust other sciĀ­enĀ­tists can have in our work.

You could pilot your study this way: Once you’ve designed your parĀ­ticĀ­iĀ­pant expeĀ­riĀ­ence in the testing platĀ­form, do some user testing. Get 10 parĀ­ticĀ­iĀ­pants to take part while you watch over zoom. You’ll get incredĀ­iĀ­ble feedĀ­back about what is clear, and what’s conĀ­fusĀ­ing and this will allow you to make your parĀ­ticĀ­iĀ­pant expeĀ­riĀ­ence better. I know we love quanĀ­tiĀ­taĀ­tive research, but qualĀ­iĀ­taĀ­tive research has its place, espeĀ­cialĀ­ly when it comes to user testing.

Next, collect a small set of data remoteĀ­ly, and use the perĀ­forĀ­mance data to idenĀ­tiĀ­fy objecĀ­tive quanĀ­tiĀ­taĀ­tive excluĀ­sion criĀ­teĀ­ria. Time spent on the instrucĀ­tions. Number of missed trials. Maximum and minimum response threshĀ­olds. This allows you to objecĀ­tiveĀ­ly exclude trials and exclude parĀ­ticĀ­iĀ­pants that are behavĀ­ing difĀ­ferĀ­entĀ­ly and which you assume to be disĀ­tractĀ­ed at that moment.

Finally, to ensure you aren’t cherry-picking the data, pre-regĀ­isĀ­ter these objecĀ­tive criĀ­teĀ­ria and then apply them rigorously.

Pre-regĀ­isĀ­terĀ­ing eleĀ­ments of our study is someĀ­thing that does give us some control over our research – thinkĀ­ing about these things ahead of data colĀ­lecĀ­tion and analyĀ­sis is incredĀ­iĀ­bly imporĀ­tant. More insight into mainĀ­tainĀ­ing data quality when you can’t see your parĀ­ticĀ­iĀ­pants can be found in Jenni Rodd’s BeOnĀ­line 2020 lecture.

Level up: Gamify tasks to maxĀ­iĀ­mize data quality

I’ve written before about how to harness parĀ­ticĀ­iĀ­pant engageĀ­ment and attenĀ­tion to maxĀ­iĀ­mize data quality when testing online. In a nutĀ­shell, you harness parĀ­ticĀ­iĀ­pant attenĀ­tion by making your task interĀ­estĀ­ing and engagĀ­ing parĀ­ticĀ­iĀ­pants in your research question.

Top tips include making your parĀ­ticĀ­iĀ­pant a research partner and making your task fun. You can even conĀ­sidĀ­er gamĀ­iĀ­fiĀ­caĀ­tion — it’s easier than you might think!

The pain of face-to-face testing can be over

Many types of behavĀ­ioral science research involves working with one parĀ­ticĀ­iĀ­pant at a time, and bringĀ­ing them to the lab. Maybe you can book 2 parĀ­ticĀ­iĀ­pants per day, so to get a sample of 100 parĀ­ticĀ­iĀ­pants, that will be 50 days – but that’s only if every person turns up. Add in weekĀ­ends and no-shows, you’re looking at around 2 months of data collection.

Instead, imagine putting your study online and colĀ­lectĀ­ing data from 500 parĀ­ticĀ­iĀ­pants in one hour. Even if you had to exclude say 10% due to poor data quality, that’s still 450 parĀ­ticĀ­iĀ­pants in one hour. The amount of time and stress saved is immense!

Many Ph.D. stuĀ­dents are funded using public funds, and so this time saving is also a cost-saving and allows PhDs to focus on better experiment design or on a task that will benefit their future research objectives.

The flexĀ­iĀ­bilĀ­iĀ­ty embedĀ­ded in online research also allows for a more repĀ­reĀ­senĀ­taĀ­tive sample. Often face-to-face lab research will be missing out on parĀ­ticĀ­iĀ­pants who are unable to attend the lab during the working day. Going online allows people to comĀ­plete your study at a time that suits them, meaning you can get reach parĀ­ticĀ­iĀ­pants that othĀ­erĀ­wise will not have been accountĀ­ed for.

UnnatĀ­urĀ­al behavĀ­ior in artiĀ­fiĀ­cial situations

From another perĀ­specĀ­tive, maybe too much control over parĀ­ticĀ­iĀ­pants is a bad thing – we put parĀ­ticĀ­iĀ­pants in an artiĀ­fiĀ­cial sitĀ­uĀ­aĀ­tion, one that may be very new to them, and then sit and watch them comĀ­plete a task. This may mean we are no longer getting a measure of ā€œnaturalā€ human behavĀ­ior, but how they respond in difĀ­ferĀ­ent circumstances.

A parĀ­ticĀ­iĀ­pant comĀ­pletĀ­ing a task online cannot have their behavĀ­ior altered by our presĀ­ence in the same way it could in the lab. In fact, in real life we rarely do one task in isoĀ­laĀ­tion – we often need to focus on one thing amid disĀ­tracĀ­tions, and thereĀ­fore research comĀ­pletĀ­ed by parĀ­ticĀ­iĀ­pants at home may be more reflecĀ­tive of a real-world situation.

UltiĀ­mateĀ­ly, we are interĀ­estĀ­ed in how humans behave in real life. Real life is messy! It’s noisy! And it’s often chaotic.

If you find an effect that works in a quiet and clean lab what does that tell you about the real world? If you can find an effect that works in a messy and noisy sitĀ­uĀ­aĀ­tion, it’s far more likely to repliĀ­cate in other real-world sitĀ­uĀ­aĀ­tions. So, lean into your lack of control over the testing enviĀ­ronĀ­ment — it might even make your research more robust.

Going back to the lab? Use what you learned

In the lab, we default to conĀ­trolĀ­ling the enviĀ­ronĀ­ment in an attempt to harness attenĀ­tion. When we take research online, we can’t control the enviĀ­ronĀ­ment, and so we’ve learned to better harness attenĀ­tion and objecĀ­tiveĀ­ly detect poor task attention.

Now it’s time to take these approachĀ­es back to the lab if you must test onsite. Since data quality is driven by parĀ­ticĀ­iĀ­pant engageĀ­ment and attenĀ­tion, you can simply use the same approachĀ­es that we use online:

  1. Make your parĀ­ticĀ­iĀ­pant a research partner, not a cog
  2. Make your task interĀ­estĀ­ing and engaging
  3. PreĀ­regĀ­isĀ­traĀ­tion of objecĀ­tive meaĀ­sures of poor data quality and use this to exclude trials and participants

That way you don’t have to control the enviĀ­ronĀ­ment to measure a proxy for attenĀ­tion. You’ve learned how to harness and assess parĀ­ticĀ­iĀ­pant attenĀ­tion directĀ­ly — both online and in the lab.

Strong benĀ­eĀ­fits are becomĀ­ing more evident

As researchers, we were all looking forward to the time when we could go back onto camĀ­pusĀ­es and into labs safely. Yet, the illuĀ­sion of control in lab face-to-face testing is being shatĀ­tered, and the strong benĀ­eĀ­fits of online research are becomĀ­ing more evident.

Online research tools allow us to conduct research faster, at a larger scale, and with greater reach which in turn gives us greater conĀ­fiĀ­dence in our results, and it’s here to stay.

Not already online? Why not? Users overĀ­whelmĀ­ingĀ­ly report that it’s easier than they expectĀ­ed. We offer a best pracĀ­tice guide to online research as well as weekly onboardĀ­ing webiĀ­naĀ­rs so that researchers can hit the ground running. See you there!

Jo EverĀ­shed

Jo is the CEO and co-founder of CaulĀ­dron and Gorilla. Her mission is to provide behavĀ­iourĀ­al sciĀ­enĀ­tists with the tools needed to improve the scale and impact of the eviĀ­dence-based interĀ­venĀ­tions that benefit society.