Some researchers have resistĀed the move to online research over the last few years, but the COVID crisis has forced many to switch to online methods. However, one quesĀtion keeps coming up: āJo, is there a good way to monitor the parĀticĀiĀpant enviĀronĀment when testing remotely?ā
The fear of losing control of the testing enviĀronĀment when taking research online is real, so letās address it.
But first, letās look at the benĀeĀfits of online and lab research to get on the same page before we look into whatās posĀsiĀble in terms of enviĀronĀmenĀtal monĀiĀtorĀing ā and what I think may be a better approach.
BenĀeĀfits of behavĀioral online research: Speed, reach, scale
- Online data colĀlecĀtion can be comĀpletĀed at an incredĀiĀble speed. The tools for online research are now so good, that it can take only a few hours to create a study. Long gone are the days of painfulĀly coding both the parĀticĀiĀpant and server-side expeĀriĀence. Couple behavĀioral science softĀware with any number of parĀticĀiĀpant recruitĀment serĀvices and you can see the data come flying in.
- Go large scale and say goodbye to underĀpowĀered studies. As you no longer need to sit in the lab with each parĀticĀiĀpant, mulĀtiĀple parĀticĀiĀpants can comĀplete your experiment simulĀtaĀneĀousĀly, leading to much larger samples. With experiment softĀware, you can get data from thouĀsands of parĀticĀiĀpants in a day.
- Extend your reach and recruit the parĀticĀiĀpants you need. Do you need a more diverse sample? Or a really speĀcifĀic group of parĀticĀiĀpants? InteĀgrate your experĀiĀmenĀtaĀtion platĀform with a recruitĀment service like ProĀlifĀic or SONA and reach groups that you couldnāt have done in the lab.
BenĀeĀfits of behavĀioral lab research: Proxy for parĀticĀiĀpant attention
So with all these benĀeĀfits, why do we stay in the lab? Control!
As researchers, we like to feel like weāre in control in the lab. We want control over the enviĀronĀment because (1) we had it in the lab and (2) we use it as a proxy for parĀticĀiĀpant attenĀtion.
We may fear that remainĀing online removes this sense of control. It seems scary to have to trust that our parĀticĀiĀpants will pay attenĀtion to the task we give them ā espeĀcialĀly if weāre not there to keep things on track. Itās scary to think about all the reasons why we may need to exclude parĀticĀiĀpants and to come up with a list of pre-defined excluĀsion criteria.
But in reality, these are things we should be thinkĀing about anyway. Perhaps we donāt have full control in the lab after all ā perhaps the control is just an illuĀsion. You had control of the enviĀronĀment, but you never had control of their mind.
The illuĀsion of control: The mind is free
When a parĀticĀiĀpant comes into the lab we can interĀact with them and watch them comĀplete the task. We can make sure they are in a quiet, disĀtracĀtion-free room, and sit in a senĀsiĀble workspace.
Yet, we cannot control where their attenĀtion is focused. They may look like they are paying attenĀtion to the task, but perhaps they are dayĀdreamĀing or just not taking it seriĀousĀly, and you can often only see this in the data later on in the research process.
Online we can ask parĀticĀiĀpants to find a quiet space, but we can never be sure if they have done this. Again this is someĀthing that we wouldnāt necĀesĀsarĀiĀly spot until we look at the data.
EnviĀronĀmenĀtal MonĀiĀtorĀing is problematic
Of course, we could with the consent of our parĀticĀiĀpants, interĀleave task trials with short bursts of recordĀing the backĀground audio (with the audio zone) and video of the home enviĀronĀment (from the webcam).
However:
- Your parĀticĀiĀpants may not like this at all! They could rightly be worried about secuĀriĀty. And the very act of asking for this sets up an antagĀoĀnisĀtic relaĀtionĀship with your parĀticĀiĀpant. Iāve written before about the imporĀtance of making parĀticĀiĀpants a research partner and treatĀing them with respect.
- Your ethics comĀmitĀtee might not like this at all! Weāre now colĀlectĀing perĀsonĀalĀly idenĀtiĀfyĀing data and colĀlectĀing data that isnāt necĀesĀsarĀiĀly relĀeĀvant to the task. In terms of secuĀriĀty, a good rule of thumb is to collect the minimum data posĀsiĀble. This goes against that rule of thumb.
- You will now have to watch and listen to all these files (it canāt be autoĀmatĀed), and you might regret your choice espeĀcialĀly when there are better and more autoĀmatĀed ways to achieve the same results. Read on!
PilotĀing and pre-regĀisĀterĀing is the way to go
So, someĀthing can happen both in lab research and online research⦠and we want to deal with the issue. The best way to do this is through strong pilotĀing of your study and working out objecĀtive excluĀsion criĀteĀria based on data quality. From this, we can pre-regĀisĀter our criĀteĀria strengthĀenĀing the trust other sciĀenĀtists can have in our work.
You could pilot your study this way: Once youāve designed your parĀticĀiĀpant expeĀriĀence in the testing platĀform, do some user testing. Get 10 parĀticĀiĀpants to take part while you watch over zoom. Youāll get incredĀiĀble feedĀback about what is clear, and whatās conĀfusĀing and this will allow you to make your parĀticĀiĀpant expeĀriĀence better. I know we love quanĀtiĀtaĀtive research, but qualĀiĀtaĀtive research has its place, espeĀcialĀly when it comes to user testing.
Next, collect a small set of data remoteĀly, and use the perĀforĀmance data to idenĀtiĀfy objecĀtive quanĀtiĀtaĀtive excluĀsion criĀteĀria. Time spent on the instrucĀtions. Number of missed trials. Maximum and minimum response threshĀolds. This allows you to objecĀtiveĀly exclude trials and exclude parĀticĀiĀpants that are behavĀing difĀferĀentĀly and which you assume to be disĀtractĀed at that moment.
Finally, to ensure you arenāt cherry-picking the data, pre-regĀisĀter these objecĀtive criĀteĀria and then apply them rigorously.
Pre-regĀisĀterĀing eleĀments of our study is someĀthing that does give us some control over our research ā thinkĀing about these things ahead of data colĀlecĀtion and analyĀsis is incredĀiĀbly imporĀtant. More insight into mainĀtainĀing data quality when you canāt see your parĀticĀiĀpants can be found in Jenni Roddās BeOnĀline 2020 lecture.
Level up: Gamify tasks to maxĀiĀmize data quality
Iāve written before about how to harness parĀticĀiĀpant engageĀment and attenĀtion to maxĀiĀmize data quality when testing online. In a nutĀshell, you harness parĀticĀiĀpant attenĀtion by making your task interĀestĀing and engagĀing parĀticĀiĀpants in your research question.
Top tips include making your parĀticĀiĀpant a research partner and making your task fun. You can even conĀsidĀer gamĀiĀfiĀcaĀtion ā itās easier than you might think!
The pain of face-to-face testing can be over
Many types of behavĀioral science research involves working with one parĀticĀiĀpant at a time, and bringĀing them to the lab. Maybe you can book 2 parĀticĀiĀpants per day, so to get a sample of 100 parĀticĀiĀpants, that will be 50 days ā but thatās only if every person turns up. Add in weekĀends and no-shows, youāre looking at around 2 months of data collection.
Instead, imagine putting your study online and colĀlectĀing data from 500 parĀticĀiĀpants in one hour. Even if you had to exclude say 10% due to poor data quality, thatās still 450 parĀticĀiĀpants in one hour. The amount of time and stress saved is immense!
Many Ph.D. stuĀdents are funded using public funds, and so this time saving is also a cost-saving and allows PhDs to focus on better experiment design or on a task that will benefit their future research objectives.
The flexĀiĀbilĀiĀty embedĀded in online research also allows for a more repĀreĀsenĀtaĀtive sample. Often face-to-face lab research will be missing out on parĀticĀiĀpants who are unable to attend the lab during the working day. Going online allows people to comĀplete your study at a time that suits them, meaning you can get reach parĀticĀiĀpants that othĀerĀwise will not have been accountĀed for.
UnnatĀurĀal behavĀior in artiĀfiĀcial situations
From another perĀspecĀtive, maybe too much control over parĀticĀiĀpants is a bad thing ā we put parĀticĀiĀpants in an artiĀfiĀcial sitĀuĀaĀtion, one that may be very new to them, and then sit and watch them comĀplete a task. This may mean we are no longer getting a measure of ānaturalā human behavĀior, but how they respond in difĀferĀent circumstances.
A parĀticĀiĀpant comĀpletĀing a task online cannot have their behavĀior altered by our presĀence in the same way it could in the lab. In fact, in real life we rarely do one task in isoĀlaĀtion ā we often need to focus on one thing amid disĀtracĀtions, and thereĀfore research comĀpletĀed by parĀticĀiĀpants at home may be more reflecĀtive of a real-world situation.
UltiĀmateĀly, we are interĀestĀed in how humans behave in real life. Real life is messy! Itās noisy! And itās often chaotic.
If you find an effect that works in a quiet and clean lab what does that tell you about the real world? If you can find an effect that works in a messy and noisy sitĀuĀaĀtion, itās far more likely to repliĀcate in other real-world sitĀuĀaĀtions. So, lean into your lack of control over the testing enviĀronĀment ā it might even make your research more robust.
Going back to the lab? Use what you learned
In the lab, we default to conĀtrolĀling the enviĀronĀment in an attempt to harness attenĀtion. When we take research online, we canāt control the enviĀronĀment, and so weāve learned to better harness attenĀtion and objecĀtiveĀly detect poor task attention.
Now itās time to take these approachĀes back to the lab if you must test onsite. Since data quality is driven by parĀticĀiĀpant engageĀment and attenĀtion, you can simply use the same approachĀes that we use online:
- Make your parĀticĀiĀpant a research partner, not a cog
- Make your task interĀestĀing and engaging
- PreĀregĀisĀtraĀtion of objecĀtive meaĀsures of poor data quality and use this to exclude trials and participants
That way you donāt have to control the enviĀronĀment to measure a proxy for attenĀtion. Youāve learned how to harness and assess parĀticĀiĀpant attenĀtion directĀly ā both online and in the lab.
Strong benĀeĀfits are becomĀing more evident
As researchers, we were all looking forward to the time when we could go back onto camĀpusĀes and into labs safely. Yet, the illuĀsion of control in lab face-to-face testing is being shatĀtered, and the strong benĀeĀfits of online research are becomĀing more evident.
Online research tools allow us to conduct research faster, at a larger scale, and with greater reach which in turn gives us greater conĀfiĀdence in our results, and itās here to stay.
Not already online? Why not? Users overĀwhelmĀingĀly report that itās easier than they expectĀed. We offer a best pracĀtice guide to online research as well as weekly onboardĀing webiĀnaĀrs so that researchers can hit the ground running. See you there!