As the CEO of Gorilla, I speak to many researchers who are looking at switching to online methods. When I do, one question keeps coming up: “Jo, is there a good way to monitor the participant environment when testing remotely?”

The fear of losing control of the testing environment when taking research online is real, so let’s address it.

But first, let’s look at the benefits of online and lab research to get on the same page before we look into what’s possible in terms of environmental monitoring — and what I think may be a better approach.

Benefits of behavioral online research: Speed, reach, scale

1. Online data collection can be completed at an incredible speed. The tools for online research are now so good, that it can take only a few hours to create a study. Long gone are the days of painfully coding both the participant and server-side experience. Couple behavioral science software with any number of participant recruitment services and you can see the data come flying in.
2. Go large scale and say goodbye to underpowered studies. As you no longer need to sit in the lab with each participant, multiple participants can complete your experiment simultaneously, leading to much larger samples. With high-quality online experiment software, you can get high-quality data from thousands of participants in a day.
3. Extend your reach and recruit the participants you need. Do you need a more diverse sample? Or a really specific group of participants? Integrate your experimentation platform with a recruitment service like Cloud Research or Prolific and reach groups that would not have come to your lab in sufficient numbers.

Benefits of behavioral lab research: Proxy for participant attention

So with all these benefits, why do we stay in the lab? Control!

As researchers, we like to feel like we’re in control in the lab. We want control over the environment because (1) we had it in the lab and (2) we use it as a proxy for participant attention.

Many fear that taking research online removes this sense of control. It seems scary to have to trust that our participants will pay attention to the task we give them — especially if we’re not there to keep things on track. It’s scary to think about all the reasons why we may need to exclude participants and to come up with a list of pre-defined exclusion criteria.

But in reality, these are things we should be thinking about anyway. Perhaps we don’t have full control in the lab after all — perhaps the control is just an illusion. You had control of the environment, but you never had control of your participant’s mind.

The illusion of control: The mind is free

When a participant comes into the lab, we can interact with them and watch them complete the task. We can make sure they are in a quiet, distraction-free room, and sit in a sensible workspace.

Yet, we cannot control where their attention is focused. They may look like they are paying attention to the task, but perhaps they are daydreaming or just not taking it seriously, and you can often only see this in the data later on in the research process.

Online we can ask participants to find a quiet space, but we can never be sure if they have done this. Again this is something that we wouldn’t necessarily spot until we look at the data.

Environmental Monitoring is problematic

Of course, with the consent of our participants, we could interleave task trials with short bursts of recording the background audio (with the audio zone) and video of the home environment (from the webcam).

However:

1. Your participants may not like this at all! They could rightly be worried about security. And the very act of asking for this sets up an antagonistic relationship with your participant. I’ve written before about the importance of making participants a research partner and treating them with respect.
2. Your ethics committee might not like this at all! We’re now collecting personally identifying data and collecting data that isn’t necessarily relevant to the task. In terms of security, a good rule of thumb is to collect the minimum data possible. This goes against that rule of thumb.
3. You will now have to watch and listen to all these files, and you might regret your choice, especially when there are better and more automated ways to achieve the same results. Read on!

Piloting and pre-registering is the way to go

So, something can happen both in lab research and online research… and we want to deal with the issue. The best way to do this is through strong piloting of your study and working out objective exclusion criteria based on data quality. From this, we can pre-register our criteria strengthening the trust other scientists can have in our work.

You could pilot your study this way: Once you’ve designed your participant experience in the testing platform, do some user testing. Get 10 participants to take part while you watch over zoom. You’ll get incredible feedback about what is clear, and what’s confusing. This feedback will allow you to make your participant experience better. I know we love quantitative research, but qualitative research has its place, especially when it comes to user testing.

Next, collect a small set of data remotely, and use the performance data to identify objective quantitative exclusion criteria. Time spent on the instructions. Number of missed trials. Maximum and minimum response thresholds. This allows you to objectively exclude trials and exclude participants that are behaving differently and which you assume to be distracted at that moment.

Finally, to ensure you aren’t cherry-picking the data, pre-register these objective criteria and then apply them rigorously.

Pre-registering elements of our study is something that does give us some control over our research – thinking about these things ahead of data collection and analysis is incredibly important. More insight into maintaining data quality when you can’t see your participants can be found in Jenni Rodd’s BeOnline 2020 lecture.

Level up: Gamify tasks to maximize data quality

I’ve written before about how to harness participant engagement and attention to maximize data quality when testing online. In a nutshell, you harness participant attention by making your task interesting and engaging participants in your research question.

Top tips include making your participant a research partner and making your task fun. You can even consider gamification — it’s easier than you might think!

The pain of face-to-face testing can be over

Many types of behavioral science research involves working with one participant at a time, and bringing them to the lab. Maybe you can book 2 participants per day, so to get a sample of 100 participants, that will be 50 days – but that’s only if every person turns up. Add in weekends and no-shows, you’re looking at around 2 months of data collection.

Instead, imagine putting your study online and collecting data from 500 participants in one hour. Even if you had to exclude say 10% due to poor data quality, that’s still 450 participants in one hour. The amount of time, stress and salary saved is immense!

Many Ph.D. students are funded using precious public money. Wasting time and salary on unnecessary face-to-face data collection, robs the PhD students of time they could have spent focussing on better experiment design or on other tasks that will benefit their future research objectives.

The flexibility embedded in online research also allows for a more representative sample. Often face-to-face lab research will be missing out on participants who are unable to attend the lab during the working day. Going online allows people to complete your study at a time that suits them, meaning you can get reach more diverse participants.

Unnatural behavior in artificial situations

From another perspective, maybe too much control over participants is a bad thing. We put participants in an artificial situation, one that may be very new to them, and then sit and watch them complete a task. This may mean we are no longer getting a measure of “natural” human behaviour, but rather how they respond in an isolated and strange situation.

A participant completing a task online cannot have their behaviour altered by our presence in the same way it could in the lab. In fact, in real life we rarely do one task in isolation – we often need to focus on one thing amid distractions, and therefore research completed by participants at home may be more reflective of a real-world situation.

Ultimately, we are interested in how humans behave in real life. Real life is messy! It’s noisy! And it’s often chaotic.

If you find an effect that works in a quiet and clean lab what does that tell you about the real world? If you can find an effect that works in a messy and noisy situation, it’s far more likely to replicate in other real-world situations. So, lean into your lack of control over the testing environment — it might even make your research more robust.

Going back to the lab? Use what you learned

In the lab, we default to controlling the environment in an attempt to harness attention. When we take research online, we can’t control the environment, and so we’ve learned to better harness attention and objectively detect poor task attention.

Now — if you need to test onsite — it’s time to take these approaches back to the lab. Since data quality is driven by participant engagement and attention, you can simply use the same approaches that we use online:

1. Make your participant a research partner, not a cog
2. Make your task interesting and engaging
3. Preregistration of objective measures of poor data quality and use this to exclude trials and participants That way you don’t have to control the environment to measure a proxy for attention. You’ve learned how to harness and assess participant attention directly — both online and in the lab.

Strong benefits are becoming more evident

During the COVID-19 pandemic, we were all looking forward to the time when we could go back onto campuses and into labs safely. Yet, the illusion of control in lab face-to-face testing is being shattered, and the strong benefits of online research are becoming more evident.

Online research tools allow us to conduct research faster, at a larger scale, and with greater reach which in turn gives us greater confidence in our results, and it’s here to stay.

Not already online? Why not? Users overwhelmingly report that it’s easier than they expected. We offer a best practice guide to online research as well as weekly onboarding webinars so that researchers can hit the ground running. See you there!