replication crisis counteract
How Taking Research Online Can Help Counteract the Replication Crisis

The Replication Crisis is the great enemy of behavioural scientists everywhere. More and more, we are finding that previous studies are either incredibly challenging or impossible to reproduce.

But can the internet solve some of the challenges presented by this crisis? More on that in a little while. But first…

What is the replication crisis and how bad is it?

A project called Many Labs 2 found that only 50% of the studies they attempted to reproduce were successfully replicated.  Reasons as to why this is the case are strongly disputed from all corners of the behavioural science community.

On one side, there’s a group arguing that studies can’t be replicated because of things like ignorance, bias and incompetence, and so calling it a crisis is overblown. Others argue that this is a very real mess, because the experiments themselves are too underpowered. On top of that, the methodology may actually be flawed or impossible to replicate. Regardless of viewpoint, if 50% of our studies cannot be replicated, there is a lot of science that cannot be counted on or used to affect widespread change in areas like mental health and wellbeing.

But why repeat a study in the first place? A few good reasons include:

  • Increasing the participant sample – the initial findings might be promising but the participant sample is too small to make concrete generalisations. Repeating the study would allow us to check if findings actually replicate across different sections of humanity.
  • Increasing the integrity of the data – repeating experiments allows scientists to double check their results. It also allows for peer confirmation – data sets become increasingly valuable when studies can be reproduced by the wider scientific community. As you move to publish your work in journals read across the globe, the integrity of that data increases exponentially if the experiment can be repeated and then goes on to show the same trends and patterns.
  • Checking patterns and trends over extended periods of time – longitudinal studies require the monitoring of participants over months or years. Similarly studies conducted in different time periods may yield results relevant to their time, but maybe not in the present day. The only way to study phenomena across time periods is to be able to reproduce that experiment to the letter.
  • Fewer retractions – there are fewer ‘oopsies’ when experiments can be replicated. Whilst it is important that the scientific community is able to retract their findings when flaws or mistakes come to light, it’s always nice to avoid this where possible. Particularly as ‘scientific’ misinformation has been at the heart of many questionable science lessons.

The main point here is that if our data foundations are constantly shifting, behavioural scientists across the globe are unable to build on top of the discoveries of yesterday. So rather than making progress, we are constantly starting from scratch. Reproducibility issues mean that any findings remain philosophical; logically they make sense, but the empirical statistics to back it up remain elusive and so the question mark remains.

replication crisis diversity

How can the internet help?

However, there is a glimmer of hope. More than a glimmer, in fact.

The internet has been a game changer in so many ways. Online shopping, watching viral cat videos and the ability to share information are just a few activities the internet has made a million times easier. Another avenue it’s opened up is the ability to conduct behavioural experiments online from anywhere in the world.

Building your experiment online is not going to be a “one size fits all” solution. It also comes with it’s own set of challenges. Access to a reliable internet connection, participation and the technical challenge of creating online behavioural experiments are just a few of the challenges the internet has yet to iron out. However, it does alleviate some rather large challenges that are feeding into the replication crisis.

No more underpowered studies

By taking your research online, the whole world opens up. Historically, behavioural scientists running surveys for example, have had to collect data either via post, in person or over the phone. Post can get lost, people can fail to respond to the call and scheduling meetings with lots of people can be tricky. By creating surveys online, scientists have a fantastic workaround for some of these major issues.

Participants have the flexibility to complete the survey wherever they have access to the internet, there’s no need for any scheduling and fewer costs for things like postage. Internet access is becoming increasingly available for people. More access, means that more people can fill out your survey regardless of geographical location.

But it’s not just survey based experiments that stand to benefit. Researchers using behavioural methods never had the option to use postal or phone methods, but have also faced the difficulties associated with in-person testing. For instance, smaller samples to work with, and limited data for interpretation. With easy-to-use online behavioural research tools, researchers gain the many benefits of taking research online. By liberating their lab, they can access large diverse samples of participants and collect data at an extraordinary pace. 

By combining online behavioural research tools with a participant recruitment service you don’t even need to find your own participants. You can build the experiment, release it online and let your chosen platform find the participants. You’ll be eating lunch, sleeping, laughing at science memes and people out there in the great wide world will be giving you the data you need, in numbers that are scientifically robust!

Faster turnarounds

In addition to the above, the internet accelerates research. Data can be collected from hundreds, and even thousands of participants in a fraction of the time. There’s no limit on how many people complete the experiment, and nobody has to wait in line. More importantly, you can say goodbye to more tedious face-to-face lab testing and all the cash management problems that come with it!

More creative freedom and increased ecological validity

People are spending more and more of their time online. Things usually done in person like shopping, banking, socialising and learning have now moved into online spaces. By creating experimental versions of the same experiences, we can collect rich ecologically valid behavioural data to test your hypothesis. 

When it comes to human behaviour, it’s unrealistic to expect to be able to control everything around the testing environment. However, if you’re unrestricted by things like having venue space capacity and not having enough time in your calendar to test each person, you can focus instead on building a comprehensive experiment that will deliver that all-important data to test your hypotheses. Online experimentation gives us access to creative methodologies that we never thought would be possible, giving us access to key insights into life as we know it today.

Build experiments that are easy to reproduce

Building an experiment online means that you can coordinate with the scientific community all around the world. The code that powers your online experiment can be documented, replicated and peer reviewed, which increases the transparency of your published findings. But sharing code isn’t necessarily the best way to share your experiment.  Sharing code may be transparent for some, but code is a black box for many and the barrier to entry is just too high.  Instead, sharing your protocol – complete with stimuli – in a way that can be cloned and replicated at the click of a button is even more useful.”

More flexibility

The internet gives you options and adaptability in a time where all of our traditional experiment options are taking a large hiatus. The COVID-19 pandemic has hit researchers hard; online research may allow researchers to get back on track. If you’re in the middle of your degree, waiting another year for things to get back to normal isn’t always feasible.

Diversity of participants

Increasing your participant sample size is often a good thing. But quality matters too. It’s hard to make reliable inferences from WEIRD samples.  With samples that are representative of the population at large, we can be more confident that findings will generalise across other participants.

Moving your research online allows you to test a diverse range of people. For findings to be applicable to large swathes of the human population, participants need to come from a wide range of backgrounds and societies. Ideally, we’d love to travel around the world meeting people from all backgrounds and cultures. But it’s not a viable model for both financial and environmental reasons. But that doesn’t mean we can’t have access to diverse groups of people.

The next best method for getting that diversity is to take your experiment online and get it in front of as many different minds as possible. This will help make sure that the results you get are a valid reflection of the population.To conclude, building your experiments, and conducting behavioural research online is a very powerful option available to behavioural scientists across the world. It’ll give us an opportunity to really stand on the shoulders of giants, increase the integrity of work produced and contribute to scientists being able to make a difference in so many lives.

For more information on how to successfully take behavioural research online.  Download our 2021 University Guide to Best Practice in Online Behavioural Research.

Happy experimenting!

Jo Evershed

Jo is the CEO and co-founder of Cauldron and Gorilla. Her mission is to provide behavioural scientists with the tools needed to improve the scale and impact of the evidence-based interventions that benefit society.