In this guest blog we hear from Dr James Bartlett about the impor­tance of researchers sharing their research mate­ri­als online. Thanks James!

When we teach stu­dents about writing an effec­tive method section, we nor­mal­ly empha­sise pro­vid­ing enough detail to allow another researcher to repli­cate the study. In reality, it can end up resem­bling a botched restora­tion of a 16th century artwork. You’re aiming for the orig­i­nal image, but in the end, you can only see the resem­blance if you squint and tilt your head to one side.

Left image: Ecce Homo by Elias Garcia Mar­tinez. Right image: Restora­tion attempt by Cecilia Giménez.

 

As rig­or­ous sci­en­tists, our desire is to run an exact repli­ca­tion, but frus­trat­ing­ly the details are rarely spe­cif­ic enough to allow it. This is a problem, as when researchers try to repli­cate the study without the orig­i­nal stimuli, incon­sis­ten­cies in the results can be attrib­uted to dif­fer­ences in the mate­ri­als or setting (Klein et al., 2018). In an ideal world, the exact mate­ri­als would accom­pa­ny the article to provide an inde­pen­dent researcher all they need to recre­ate the study. However, as we will see, sharing mate­ri­als is rare. After hope­ful­ly con­vinc­ing you why this should change, this article will provide a brief overview of options and con­sid­er­a­tions for sharing your materials.

 

Sharing is rare but rewarding

Researchers seldom share the mate­ri­als used in their experiment, with studies audit­ing the preva­lence of open mate­ri­als ranging from 1% (Adewumi et al., 2021) to 14% of arti­cles (Hard­wicke et al., 2021). Even looking for a lower bar of trans­paren­cy, just 38% of arti­cles report­ed where they sourced the images in their cog­ni­tive tasks (Pen­ning­ton et al., 2021). These studies show dis­ap­point­ing­ly low rates of trans­paren­cy in mate­ri­als across areas of psy­chol­o­gy. As dif­fer­ent areas of science have worked to become more open, being trans­par­ent about how you con­duct­ed your study is one of the key tenets (Munafó et al., 2017). Despite this, in a world where the quan­ti­ty not quality of research is valued (Allen & Mehler, 2019), going out of your way to doc­u­ment and share the mate­ri­als is typ­i­cal­ly not high on the agenda. What is the incen­tive? One ini­tia­tive is jour­nals pro­vid­ing badges to recog­nise arti­cles that have shared outputs like the mate­ri­als. In the journal Psy­cho­log­i­cal Science, offer­ing badges for open mate­ri­als increased rates from 13% to 30%, while control jour­nals not offer­ing these badges remained stable (Kidwell et al., 2016). More jour­nals have offered these badges (Grahe, 2018), but uptake is still the excep­tion, not the norm. Perhaps more entic­ing is arti­cles pro­vid­ing open mate­ri­als tend to receive a higher number of cita­tions (McK­ier­nan et al., 2016). These ini­tia­tives show that beyond the warm fuzzy feeling of being helpful, researchers can be recog­nised for sharing their mate­ri­als through a badge or the prospect of more cita­tions. Effec­tive­ly sharing and doc­u­ment­ing mate­ri­als takes extra time and can present a barrier for researchers as they are per­ceived to not be valued in the race for funding and jobs (Zečević et al., 2021). However, sharing mate­ri­als is often one of the easiest open science prac­tices to adopt as it does not have to be in the ethics process like open data. This means if you want to try ways of making your research more open and trans­par­ent, sharing your mate­ri­als is one of the easiest start­ing points.

 

Gorilla Open Materials

In my teach­ing and research, I use Gorilla to create exper­i­ments as I work for an online uni­ver­si­ty. One of their fea­tures is an Open Mate­ri­als func­tion. This allows you to share and doc­u­ment any task, ques­tion­naire, code, or even whole experiment. For example, in my PhD I used some­thing called the visual probe task with images of smoking/cigarettes. I added it to the Open Mate­ri­als page:

 

 

Gorilla Open Mate­ri­als over­comes many of the bar­ri­ers to sharing mate­ri­als. All you have to do is add a task you already made to an Open Mate­ri­als page. You provide a descrip­tion of the task and what it does. Further below in the screen­shot, you can add a task diagram to provide a visual overview. For those sweet cita­tion counts, you can add your pre­ferred cita­tion in the top right. The most impor­tant aspect is the ease of reuse. There are preview and clone buttons with each task. Pre­view­ing the task allows anyone to expe­ri­ence it exactly as the par­tic­i­pant did in your browser. This means anyone can see exactly how it worked, without down­load­ing any soft­ware or signing up for a Gorilla account. If a researcher wants to inspect, reuse or adapt it in their own work, they can clone the task into their own project. If you use Gorilla for your research, this presents an extreme­ly user-friend­ly way of facil­i­tat­ing repli­ca­tions of your work.

Note from the Gorilla team: We’ve heard that some researchers include their Gorilla Open Mate­ri­als page within pre-reg­is­tra­tions on the Open Science Frame­work which then also pro­vides a DOI for the resources.  In excit­ing news — Gorilla Open Mate­ri­als 2.0 will have the ability to add DOIs! Keep your eyes peeled for more infor­ma­tion on this…

 

Sharing task scripts on the OSF

If you do not use Gorilla, you can share your task scripts on a repos­i­to­ry to allow another researcher to reuse it. For example, if you have created an experiment using E‑Prime or Matlab, you can share the script and stimuli used to program the task. If these are all you have, then sharing is better than not sharing, but these are both exam­ples of pro­pri­ety soft­ware. This means someone would not be able to use the mate­ri­als until they bought a licence. If you share scripts from open-source alter­na­tives, anyone in the world can reuse them. For example, I have shared the scripts from OpenS­esame and PsychoPy on the Open Science Frame­work:

For example, this screen­shot shows a zip file con­tain­ing all the PsychoPy files needed to run the visual probe task. The only addi­tion I had to make was writing a README file which describes how it can be reused. It describes the trial struc­ture, how to replace the stimuli, and how to use the task without the eye-track­ing code. I know at least three people have ben­e­fit­ed from sharing these scripts on the OSF, as they have emailed me to ask ques­tions.  With OSF metric track­ing, I can also see this task has been down­loaded 220 times at the time of writing. This shows how a small time com­mit­ment can benefit other researchers. In addi­tion to pure com­mu­nal­ism, the OSF project has a DOI which means other researchers can cite the project when they reuse it. This shows you can also benefit from sharing your mate­ri­als as it con­tributes to your impact.

 

Sharing ques­tion­naires

Not all research is behav­iour­al, there­fore con­sid­er­a­tion of ques­tion­naire access is impor­tant. When a team of researchers val­i­dates and pub­lish­es a ques­tion­naire, the items and scoring instruc­tions can often be missing. Some­times this will be due to con­cerns over the journal retain­ing copy­right, while others could create pro­pri­etary ques­tion­naires and require payment for reuse (Hays et al., 2018). If you val­i­date a ques­tion­naire and want to share the mate­ri­als for maximum reuse, you can release it under a Cre­ative Commons licence. This means you can openly share the ques­tion­naire while still retain­ing rights as the author (Hays et al., 2018). Before you submit the val­i­da­tion article, you could add the ques­tion­naire to Gorilla Open Mate­ri­als, the OSF, or Figshare. The OSF and Figshare allow you to create a DOI, select a licence to state terms of reuse, and if required, set an embargo period until there is public access. This means you can cite the ques­tion­naire in your article and retain rights as the author and encour­age free use as long as you are cited as the original.

Note from the Gorilla Team: Within Gorilla Open Mate­ri­als there are three licences to choose from:

  • Cre­ative Commons Attribution
  • Cre­ative Commons Attri­bu­tion Non-Commercial
  • Gorilla Open Mate­ri­als Attri­bu­tion Non-Com­mer­cial Research Only

The third licence is only avail­able in Gorilla. It’s for sit­u­a­tions where you may have a measure (such as an IQ test) and you neither want it to be used com­mer­cial­ly or in the public sector (i.e., schools and hos­pi­tals). It only allows others to use it for non-com­mer­cial research purposes.

 

Sharing a video of the mate­ri­als and setting

Sharing the mate­ri­als is impor­tant, but an addi­tion­al concern in repli­ca­tion studies are dif­fer­ences in the set­tings and pro­ce­dure (Klein et al., 2018). One ini­tia­tive some researchers have started is sharing a record­ing doc­u­ment­ing the data col­lec­tion pro­ce­dure. For example, Wagen­mak­ers et al. (2016) shared the mate­ri­als from their repli­ca­tion attempt and a record­ing of the pro­ce­dure. This makes the method crystal clear: making it easier to reuse the mate­ri­als and reduces the oppor­tu­ni­ty for dif­fer­ences in pro­ce­dure to affect the meaning of fail­ures to replicate.

 

Con­clu­sion

The move­ment towards greater trans­paren­cy in research means sharing mate­ri­als has never been easier with tools like Gorilla Open Mate­ri­als, the OSF, and Figshare. However, esti­mates in psy­chol­o­gy of how often researchers share their methods are dis­ap­point­ing­ly low given how it is one of the easiest parts of the research process to open up. Although it takes extra time, you can be reward­ed with the warm fuzzy feeling of helping other researchers, and if that does not moti­vate you, the poten­tial for a greater number of cita­tions and badges to reward your effort might help. There are dif­fer­ent options avail­able for sharing your mate­ri­als, from Gorilla Open Mate­ri­als, to sharing your tasks and ques­tion­naires on the OSF. Just remem­ber to make it reusable for other researchers by doc­u­ment­ing how it works through a README file. Hope­ful­ly, you will see the rewards of greater trans­paren­cy from researchers being able to reuse and credit your mate­ri­als as an addi­tion­al output, to when you need to repli­cate a study and you find the authors help­ful­ly posted a video demon­strat­ing their procedure.

 

Dr James Bartlett

Dr James Bartlett (@JamesEBartlett), Senior Lec­tur­er, School of Psy­chol­o­gy and Social Science, Arden Uni­ver­si­ty, UK.