replication crisis counteract
How Tak­ing Research Online Can Help Coun­ter­act the Repli­ca­tion Crisis

The Repli­ca­tion Cri­sis is the great enemy of behav­iour­al sci­en­tists every­where. More and more, we are find­ing that pre­vi­ous stud­ies are either incred­i­bly chal­leng­ing or impos­si­ble to reproduce.

But can the inter­net solve some of the chal­lenges pre­sent­ed by this cri­sis? More on that in a lit­tle while. But first…

What is the repli­ca­tion cri­sis and how bad is it?

A project called Many Labs 2 found that only 50% of the stud­ies they attempt­ed to repro­duce were suc­cess­ful­ly repli­cat­ed.  Rea­sons as to why this is the case are strong­ly dis­put­ed from all cor­ners of the behav­iour­al sci­ence community.

On one side, there’s a group argu­ing that stud­ies can’t be repli­cat­ed because of things like igno­rance, bias and incom­pe­tence, and so call­ing it a cri­sis is overblown. Oth­ers argue that this is a very real mess, because the exper­i­ments them­selves are too under­pow­ered. On top of that, the method­ol­o­gy may actu­al­ly be flawed or impos­si­ble to repli­cate. Regard­less of view­point, if 50% of our stud­ies can­not be repli­cat­ed, there is a lot of sci­ence that can­not be count­ed on or used to affect wide­spread change in areas like men­tal health and wellbeing.

But why repeat a study in the first place? A few good rea­sons include:

  • Increas­ing the par­tic­i­pant sam­ple — the ini­tial find­ings might be promis­ing but the par­tic­i­pant sam­ple is too small to make con­crete gen­er­al­i­sa­tions. Repeat­ing the study would allow us to check if find­ings actu­al­ly repli­cate across dif­fer­ent sec­tions of humanity.
  • Increas­ing the integri­ty of the data — repeat­ing exper­i­ments allows sci­en­tists to dou­ble check their results. It also allows for peer con­fir­ma­tion — data sets become increas­ing­ly valu­able when stud­ies can be repro­duced by the wider sci­en­tif­ic com­mu­ni­ty. As you move to pub­lish your work in jour­nals read across the globe, the integri­ty of that data increas­es expo­nen­tial­ly if the exper­i­ment can be repeat­ed and then goes on to show the same trends and patterns.
  • Check­ing pat­terns and trends over extend­ed peri­ods of time — lon­gi­tu­di­nal stud­ies require the mon­i­tor­ing of par­tic­i­pants over months or years. Sim­i­lar­ly stud­ies con­duct­ed in dif­fer­ent time peri­ods may yield results rel­e­vant to their time, but maybe not in the present day. The only way to study phe­nom­e­na across time peri­ods is to be able to repro­duce that exper­i­ment to the letter.
  • Fewer retrac­tions — there are fewer ‘oop­sies’ when exper­i­ments can be repli­cat­ed. Whilst it is impor­tant that the sci­en­tif­ic com­mu­ni­ty is able to retract their find­ings when flaws or mis­takes come to light, it’s always nice to avoid this where pos­si­ble. Par­tic­u­lar­ly as ‘sci­en­tif­ic’ mis­in­for­ma­tion has been at the heart of many ques­tion­able sci­ence lessons.

The main point here is that if our data foun­da­tions are con­stant­ly shift­ing, behav­iour­al sci­en­tists across the globe are unable to build on top of the dis­cov­er­ies of yes­ter­day. So rather than mak­ing progress, we are con­stant­ly start­ing from scratch. Repro­ducibil­i­ty issues mean that any find­ings remain philo­soph­i­cal; log­i­cal­ly they make sense, but the empir­i­cal sta­tis­tics to back it up remain elu­sive and so the ques­tion mark remains.

replication crisis diversity

How can the inter­net help?

How­ev­er, there is a glim­mer of hope. More than a glim­mer, in fact.

The inter­net has been a game chang­er in so many ways. Online shop­ping, watch­ing viral cat videos and the abil­i­ty to share infor­ma­tion are just a few activ­i­ties the inter­net has made a mil­lion times eas­i­er. Anoth­er avenue it’s opened up is the abil­i­ty to con­duct behav­iour­al exper­i­ments online from any­where in the world. 

Build­ing your exper­i­ment online is not going to be a “one size fits all” solu­tion. It also comes with it’s own set of chal­lenges. Access to a reli­able inter­net con­nec­tion, par­tic­i­pa­tion and the tech­ni­cal chal­lenge of cre­at­ing online behav­iour­al exper­i­ments are just a few of the chal­lenges the inter­net has yet to iron out. How­ev­er, it does alle­vi­ate some rather large chal­lenges that are feed­ing into the repli­ca­tion crisis.

No more under­pow­ered studies

By tak­ing your research online, the whole world opens up. His­tor­i­cal­ly, behav­iour­al sci­en­tists run­ning sur­veys for exam­ple, have had to col­lect data either via post, in per­son or over the phone. Post can get lost, peo­ple can fail to respond to the call and sched­ul­ing meet­ings with lots of peo­ple can be tricky. By cre­at­ing sur­veys online, sci­en­tists have a fan­tas­tic workaround for some of these major issues.

Par­tic­i­pants have the flex­i­bil­i­ty to com­plete the sur­vey wher­ev­er they have access to the inter­net, there’s no need for any sched­ul­ing and fewer costs for things like postage. Inter­net access is becom­ing increas­ing­ly avail­able for peo­ple. More access, means that more peo­ple can fill out your sur­vey regard­less of geo­graph­i­cal location.

But it’s not just sur­vey based exper­i­ments that stand to ben­e­fit. Researchers using behav­iour­al meth­ods never had the option to use postal or phone meth­ods, but have also faced the dif­fi­cul­ties asso­ci­at­ed with in-per­son test­ing. For instance, small­er sam­ples to work with, and lim­it­ed data for inter­pre­ta­tion. With easy-to-use online behav­iour­al research tools, researchers gain the many ben­e­fits of tak­ing research online. By lib­er­at­ing their lab, they can access large diverse sam­ples of par­tic­i­pants and col­lect data at an extra­or­di­nary pace. 

By com­bin­ing online behav­iour­al research tools with a par­tic­i­pant recruit­ment ser­vice you don’t even need to find your own par­tic­i­pants. You can build the exper­i­ment, release it online and let your cho­sen plat­form find the par­tic­i­pants. You’ll be eat­ing lunch, sleep­ing, laugh­ing at sci­ence memes and peo­ple out there in the great wide world will be giv­ing you the data you need, in num­bers that are sci­en­tif­i­cal­ly robust!

Faster turn­arounds

In addi­tion to the above, the inter­net accel­er­ates research. Data can be col­lect­ed from hun­dreds, and even thou­sands of par­tic­i­pants in a frac­tion of the time. There’s no limit on how many peo­ple com­plete the exper­i­ment, and nobody has to wait in line. More impor­tant­ly, you can say good­bye to more tedious face-to-face lab test­ing and all the cash man­age­ment prob­lems that come with it!

More cre­ative free­dom and increased eco­log­i­cal validity

Peo­ple are spend­ing more and more of their time online. Things usu­al­ly done in per­son like shop­ping, bank­ing, social­is­ing and learn­ing have now moved into online spaces. By cre­at­ing exper­i­men­tal ver­sions of the same expe­ri­ences, we can col­lect rich eco­log­i­cal­ly valid behav­iour­al data to test your hypothesis. 

When it comes to human behav­iour, it’s unre­al­is­tic to expect to be able to con­trol every­thing around the test­ing envi­ron­ment. How­ev­er, if you’re unre­strict­ed by things like hav­ing venue space capac­i­ty and not hav­ing enough time in your cal­en­dar to test each per­son, you can focus instead on build­ing a com­pre­hen­sive exper­i­ment that will deliv­er that all-impor­tant data to test your hypothe­ses. Online exper­i­men­ta­tion gives us access to cre­ative method­olo­gies that we never thought would be pos­si­ble, giv­ing us access to key insights into life as we know it today.

Build exper­i­ments that are easy to reproduce

Build­ing an exper­i­ment online means that you can coor­di­nate with the sci­en­tif­ic com­mu­ni­ty all around the world. The code that pow­ers your online exper­i­ment can be doc­u­ment­ed, repli­cat­ed and peer reviewed, which increas­es the trans­paren­cy of your pub­lished find­ings. But shar­ing code isn’t nec­es­sar­i­ly the best way to share your exper­i­ment.  Shar­ing code may be trans­par­ent for some, but code is a black box for many and the bar­ri­er to entry is just too high.  Instead, shar­ing your pro­to­col — com­plete with stim­uli — in a way that can be cloned and repli­cat­ed at the click of a but­ton is even more useful.”

More flex­i­bil­i­ty

The inter­net gives you options and adapt­abil­i­ty in a time where all of our tra­di­tion­al exper­i­ment options are tak­ing a large hia­tus. The COVID-19 pan­dem­ic has hit researchers hard; online research may allow researchers to get back on track. If you’re in the mid­dle of your degree, wait­ing anoth­er year for things to get back to nor­mal isn’t always feasible. 

Diver­si­ty of participants

Increas­ing your par­tic­i­pant sam­ple size is often a good thing. But qual­i­ty mat­ters too. It’s hard to make reli­able infer­ences from WEIRD sam­ples.  With sam­ples that are rep­re­sen­ta­tive of the pop­u­la­tion at large, we can be more con­fi­dent that find­ings will gen­er­alise across other participants.

Mov­ing your research online allows you to test a diverse range of peo­ple. For find­ings to be applic­a­ble to large swathes of the human pop­u­la­tion, par­tic­i­pants need to come from a wide range of back­grounds and soci­eties. Ide­al­ly, we’d love to trav­el around the world meet­ing peo­ple from all back­grounds and cul­tures. But it’s not a viable model for both finan­cial and envi­ron­men­tal rea­sons. But that doesn’t mean we can’t have access to diverse groups of people.

The next best method for get­ting that diver­si­ty is to take your exper­i­ment online and get it in front of as many dif­fer­ent minds as pos­si­ble. This will help make sure that the results you get are a valid reflec­tion of the population.To con­clude, build­ing your exper­i­ments, and con­duct­ing behav­iour­al research online is a very pow­er­ful option avail­able to behav­iour­al sci­en­tists across the world. It’ll give us an oppor­tu­ni­ty to real­ly stand on the shoul­ders of giants, increase the integri­ty of work pro­duced and con­tribute to sci­en­tists being able to make a dif­fer­ence in so many lives.

For more infor­ma­tion on how to suc­cess­ful­ly take behav­iour­al research online.  Down­load our 2021 Uni­ver­si­ty Guide to Best Prac­tice in Online Behav­iour­al Research.

Happy exper­i­ment­ing!

Jo Ever­shed

Jo is the CEO and co-founder of Caul­dron and Gorilla. Her mis­sion is to pro­vide behav­iour­al sci­en­tists with the tools need­ed to improve the scale and impact of the evi­dence-based inter­ven­tions that ben­e­fit society.