replication crisis counteract

The Repli­ca­tion Crisis is the great enemy of behav­iour­al sci­en­tists every­where. More and more, we are finding that pre­vi­ous studies are either incred­i­bly chal­leng­ing or impos­si­ble to reproduce.

But can the inter­net solve some of the chal­lenges pre­sent­ed by this crisis? More on that in a little while. But first…

What is the repli­ca­tion crisis and how bad is it?

A project called Many Labs 2 found that only 50% of the studies they attempt­ed to repro­duce were suc­cess­ful­ly repli­cat­ed.  Reasons as to why this is the case are strong­ly dis­put­ed from all corners of the behav­iour­al science community.

On one side, there’s a group arguing that studies can’t be repli­cat­ed because of things like igno­rance, bias and incom­pe­tence, and so calling it a crisis is overblown. Others argue that this is a very real mess, because the exper­i­ments them­selves are too under­pow­ered. On top of that, the method­ol­o­gy may actu­al­ly be flawed or impos­si­ble to repli­cate. Regard­less of view­point, if 50% of our studies cannot be repli­cat­ed, there is a lot of science that cannot be counted on or used to affect wide­spread change in areas like mental health and wellbeing.

But why repeat a study in the first place? A few good reasons include:

  • Increas­ing the par­tic­i­pant sample — the initial find­ings might be promis­ing but the par­tic­i­pant sample is too small to make con­crete gen­er­al­i­sa­tions. Repeat­ing the study would allow us to check if find­ings actu­al­ly repli­cate across dif­fer­ent sec­tions of humanity.
  • Increas­ing the integri­ty of the data — repeat­ing exper­i­ments allows sci­en­tists to double check their results. It also allows for peer con­fir­ma­tion — data sets become increas­ing­ly valu­able when studies can be repro­duced by the wider sci­en­tif­ic com­mu­ni­ty. As you move to publish your work in jour­nals read across the globe, the integri­ty of that data increas­es expo­nen­tial­ly if the experiment can be repeat­ed and then goes on to show the same trends and patterns.
  • Check­ing pat­terns and trends over extend­ed periods of time — lon­gi­tu­di­nal studies require the mon­i­tor­ing of par­tic­i­pants over months or years. Sim­i­lar­ly studies con­duct­ed in dif­fer­ent time periods may yield results rel­e­vant to their time, but maybe not in the present day. The only way to study phe­nom­e­na across time periods is to be able to repro­duce that experiment to the letter.
  • Fewer retrac­tions — there are fewer ‘oopsies’ when exper­i­ments can be repli­cat­ed. Whilst it is impor­tant that the sci­en­tif­ic com­mu­ni­ty is able to retract their find­ings when flaws or mis­takes come to light, it’s always nice to avoid this where pos­si­ble. Par­tic­u­lar­ly as ‘sci­en­tif­ic’ mis­in­for­ma­tion has been at the heart of many ques­tion­able science lessons.

The main point here is that if our data foun­da­tions are con­stant­ly shift­ing, behav­iour­al sci­en­tists across the globe are unable to build on top of the dis­cov­er­ies of yes­ter­day. So rather than making progress, we are con­stant­ly start­ing from scratch. Repro­ducibil­i­ty issues mean that any find­ings remain philo­soph­i­cal; log­i­cal­ly they make sense, but the empir­i­cal sta­tis­tics to back it up remain elusive and so the ques­tion mark remains.

replication crisis diversity

How can the inter­net help?

However, there is a glimmer of hope. More than a glimmer, in fact.

The inter­net has been a game changer in so many ways. Online shop­ping, watch­ing viral cat videos and the ability to share infor­ma­tion are just a few activ­i­ties the inter­net has made a million times easier. Another avenue it’s opened up is the ability to conduct behav­iour­al exper­i­ments online from any­where in the world. 

Build­ing your experiment online is not going to be a “one size fits all” solu­tion. It also comes with it’s own set of chal­lenges. Access to a reli­able inter­net con­nec­tion, par­tic­i­pa­tion and the tech­ni­cal chal­lenge of cre­at­ing online behav­iour­al exper­i­ments are just a few of the chal­lenges the inter­net has yet to iron out. However, it does alle­vi­ate some rather large chal­lenges that are feeding into the repli­ca­tion crisis.

No more under­pow­ered studies

By taking your research online, the whole world opens up. His­tor­i­cal­ly, behav­iour­al sci­en­tists running surveys for example, have had to collect data either via post, in person or over the phone. Post can get lost, people can fail to respond to the call and sched­ul­ing meet­ings with lots of people can be tricky. By cre­at­ing surveys online, sci­en­tists have a fan­tas­tic workaround for some of these major issues.

Par­tic­i­pants have the flex­i­bil­i­ty to com­plete the survey wher­ev­er they have access to the inter­net, there’s no need for any sched­ul­ing and fewer costs for things like postage. Inter­net access is becom­ing increas­ing­ly avail­able for people. More access, means that more people can fill out your survey regard­less of geo­graph­i­cal location.

But it’s not just survey based exper­i­ments that stand to benefit. Researchers using behav­iour­al methods never had the option to use postal or phone methods, but have also faced the dif­fi­cul­ties asso­ci­at­ed with in-person testing. For instance, smaller samples to work with, and limited data for inter­pre­ta­tion. With easy-to-use online behav­iour­al research tools, researchers gain the many ben­e­fits of taking research online. By lib­er­at­ing their lab, they can access large diverse samples of par­tic­i­pants and collect data at an extra­or­di­nary pace. 

By com­bin­ing online behav­iour­al research tools with a par­tic­i­pant recruit­ment service you don’t even need to find your own par­tic­i­pants. You can build the experiment, release it online and let your chosen plat­form find the par­tic­i­pants. You’ll be eating lunch, sleep­ing, laugh­ing at science memes and people out there in the great wide world will be giving you the data you need, in numbers that are sci­en­tif­i­cal­ly robust!

Faster turn­arounds

In addi­tion to the above, the inter­net accel­er­ates research. Data can be col­lect­ed from hun­dreds, and even thou­sands of par­tic­i­pants in a frac­tion of the time. There’s no limit on how many people com­plete the experiment, and nobody has to wait in line. More impor­tant­ly, you can say goodbye to more tedious face-to-face lab testing and all the cash man­age­ment prob­lems that come with it!

More cre­ative freedom and increased eco­log­i­cal validity

People are spend­ing more and more of their time online. Things usually done in person like shop­ping, banking, social­is­ing and learn­ing have now moved into online spaces. By cre­at­ing exper­i­men­tal ver­sions of the same expe­ri­ences, we can collect rich eco­log­i­cal­ly valid behav­iour­al data to test your hypothesis. 

When it comes to human behav­iour, it’s unre­al­is­tic to expect to be able to control every­thing around the testing envi­ron­ment. However, if you’re unre­strict­ed by things like having venue space capac­i­ty and not having enough time in your cal­en­dar to test each person, you can focus instead on build­ing a com­pre­hen­sive experiment that will deliver that all-impor­tant data to test your hypothe­ses. Online exper­i­men­ta­tion gives us access to cre­ative method­olo­gies that we never thought would be pos­si­ble, giving us access to key insights into life as we know it today.

Build exper­i­ments that are easy to reproduce

Build­ing an experiment online means that you can coor­di­nate with the sci­en­tif­ic com­mu­ni­ty all around the world. The code that powers your online experiment can be doc­u­ment­ed, repli­cat­ed and peer reviewed, which increas­es the trans­paren­cy of your pub­lished find­ings. But sharing code isn’t nec­es­sar­i­ly the best way to share your experiment.  Sharing code may be trans­par­ent for some, but code is a black box for many and the barrier to entry is just too high.  Instead, sharing your pro­to­col — com­plete with stimuli — in a way that can be cloned and repli­cat­ed at the click of a button is even more useful.”

More flex­i­bil­i­ty

The inter­net gives you options and adapt­abil­i­ty in a time where all of our tra­di­tion­al experiment options are taking a large hiatus. The COVID-19 pan­dem­ic has hit researchers hard; online research may allow researchers to get back on track. If you’re in the middle of your degree, waiting another year for things to get back to normal isn’t always feasible. 

Diver­si­ty of participants

Increas­ing your par­tic­i­pant sample size is often a good thing. But quality matters too. It’s hard to make reli­able infer­ences from WEIRD samples.  With samples that are rep­re­sen­ta­tive of the pop­u­la­tion at large, we can be more con­fi­dent that find­ings will gen­er­alise across other participants.

Moving your research online allows you to test a diverse range of people. For find­ings to be applic­a­ble to large swathes of the human pop­u­la­tion, par­tic­i­pants need to come from a wide range of back­grounds and soci­eties. Ideally, we’d love to travel around the world meeting people from all back­grounds and cul­tures. But it’s not a viable model for both finan­cial and envi­ron­men­tal reasons. But that doesn’t mean we can’t have access to diverse groups of people.

The next best method for getting that diver­si­ty is to take your experiment online and get it in front of as many dif­fer­ent minds as pos­si­ble. This will help make sure that the results you get are a valid reflec­tion of the population.To con­clude, build­ing your exper­i­ments, and con­duct­ing behav­iour­al research online is a very pow­er­ful option avail­able to behav­iour­al sci­en­tists across the world. It’ll give us an oppor­tu­ni­ty to really stand on the shoul­ders of giants, increase the integri­ty of work pro­duced and con­tribute to sci­en­tists being able to make a dif­fer­ence in so many lives.

For more infor­ma­tion on how to suc­cess­ful­ly take behav­iour­al research online.  Down­load our 2021 Uni­ver­si­ty Guide to Best Prac­tice in Online Behav­iour­al Research.

Happy exper­i­ment­ing!

Jo Ever­shed

Jo is the CEO and co-founder of Caul­dron and Gorilla. Her mission is to provide behav­iour­al sci­en­tists with the tools needed to improve the scale and impact of the evi­dence-based inter­ven­tions that benefit society.