Pro­gram­ming Online Behav­ioral Exper­i­ments: How to Decide Where to Deploy Your Time for Best Effect

Under­stand­ing pro­gram­ming is one of the great keys to the future across so many spheres of life. In the field of behav­iour­al sci­ence, this is also true; there’s no doubt that a work­ing knowl­edge of pro­gram­ming might help you build bet­ter exper­i­ments. Learn­ing to code from scratch is an extreme­ly pow­er­ful skill, and it’s one that peo­ple in the sci­en­tif­ic com­mu­ni­ty spend years perfecting.

But how do you fig­ure out the right moment to invest that time into deep­en­ing your under­stand­ing of coding?

The Wear­ing of Many Hats

Before we answer the above ques­tion, let’s estab­lish a bit of con­text around the life of a behav­iour­al scientist.

We all know and appre­ci­ate that you wear many dif­fer­ent hats. As you con­duct your research, there is a whole port­fo­lio of skills you need to mas­ter at var­i­ous dif­fer­ent stages of your jour­ney. Under the umbrel­la of ‘behav­iour­al sci­en­tist’ you will need to cul­ti­vate the fol­low­ing skills to a level of proficiency:

  • Pitch­ing — Pitch­ing for grants and fund­ing requires the abil­i­ty to put for­ward a com­pelling argument
  • Tech­ni­cal Skill — Build­ing the exper­i­ment and craft­ing the par­tic­i­pant experience
  • Project Man­age­ment (Data Gath­er­ing and Man­age­ment) — Organ­is­ing and col­lect­ing the data and man­ag­ing that process
  • Under­stand­ing Sta­tis­tics — Becom­ing a sta­tis­ti­cian is a huge part of the science
  • Writ­ten Com­mu­ni­ca­tion — Writ­ing every­thing up into a paper
  • Dis­sem­i­nate your Find­ings — Share your find­ings through pub­lic speak­ing, lec­tures, and con­fer­ences to ensure your audi­ence con­nects with your work

Putting all things in con­text, behav­iour­al sci­en­tists have a huge amount to learn in a short amount of time. It’s actu­al­ly a tri­umph in and of itself that behav­iour­al sci­en­tists con­tin­u­al­ly make one-per­son teams out of them­selves in the pur­suit of knowledge.

We see you.

The point here is that there are so many things vying for your time, and the lim­it­ed resources at your dis­pos­al. At every stage, you’ll need to ask your­self, what should be the focus? Doing every­thing at once is unre­al­is­tic in a lot of cases, so where and when can we cre­ate workarounds that relieve the pres­sure sci­en­tists find them­selves under to be all things to all research?

More impor­tant­ly, when is the right time to push for learn­ing a lit­tle or a lot more about coding?

Dif­fer­ent Strokes for Dif­fer­ent Folks

The real­i­ty is that there is no one size fits all. If you’re pio­neer­ing a new method­ol­o­gy, it’s entire­ly pos­si­ble that the task will need cus­tom code. You’ll then need to learn the most appro­pri­ate lan­guage for the envi­ron­ment you want to test in. Mat­lab or Python if you’re in the lab. Or JavaScript if you want to test online.

There’s no deny­ing that being pro­fi­cient at soft­ware devel­op­ment is going to present some added ben­e­fits for you as you engi­neer your exper­i­ment. The invest­ment of time becomes worth it; you’re build­ing some­thing new that doesn’t yet exist or exists part­ly but not in the way you need it to.

But this route does have a ‘dark side’. That dark side comes in the form of count­less error mes­sages and a deep seat­ed frus­tra­tion that makes you want to throw your com­put­er out the win­dow and eat your body weight in snacks to con­sole yourself.

No judge­ment about the snacks. We understand.

Alter­na­tive­ly, you write the code and it is a thing of beau­ty. You mar­vel at your cre­ation. Then you go to bed and the pro­gramme you’ve coded it for gets a crit­i­cal update and the task you’ve built no longer works. Now, you have to re-do that code. Or the library you’re using gets an update and now you have to rework that sec­tion. That’s a whole new frus­tra­tion level right there.

For many exper­i­ments, this kind of stress just isn’t nec­es­sary. There are tools out there that have been designed to give you back your time, time that can be used more effec­tive­ly. They’ll give you stream­lined guid­ance on how to build an effec­tive task online in a frac­tion of the time it would take you to code from scratch your­self. I’ve chat­ted with sci­en­tists that are still strug­gling with their tasks after 3 months of steady progress, and we’ve repli­cat­ed their task in a task builder in under 30 minutes.

In such instances where time and resources are scarce, a well round­ed tool frees you up to focus more on the sci­ence. Those tools are known as task builders and exper­i­ment builders.

So How Do You Choose A Good Task Builder?

So if you decide to choose to use a task builder, how do you pick a good one?

Well think about your exper­i­ment — what dif­fer­ent facets will it have? If you’re going to be doing a ques­tion­naire exclu­sive­ly, for exam­ple, then a tool like Qualtrics is going to be your best bet. But if your task needs accu­rate reac­tion times, and a trial struc­ture, are there tools that can get you there with less cod­ing or no cod­ing at all?

For some, just using a task builder for a sim­ple exper­i­ment may actu­al­ly remove the cod­ing ele­ment alto­geth­er. If you’re try­ing to get to grips with all the other things, that’s going to be super handy. This video demon­strates how to build a cat­e­go­ry nam­ing task — and it only takes three minutes:


By load­ing the video, you agree to YouTube’s pri­va­cy pol­i­cy.
Learn more

Load video

How­ev­er, if you’re look­ing for a halfway house, where you’re free to write your own code in JavaScript, but don’t want to host and man­age your own serv­er, then you have other options, too. There are a vari­ety to choose from, which is great. But it’s impor­tant to ensure that the tool you choose enrich­es your research prac­tice, instead of com­pro­mis­ing it by allow­ing you to focus on the sci­ence and not the programming.

The way to check that a tool is right for you is to think about how much flex­i­bil­i­ty the inter­face gives you:

  • How cre­ative can you be with the build­ing blocks given?
  • Are you able to aug­ment the tool’s func­tion­al­i­ty so it can serve your unique needs if the default tool is not an exact fit?
  • Does it offer to host bare bones code, for when you absolute­ly have to write your tasks your­self, but don’t want to man­age a whole server?
  • Is it easy to get start­ed on this tool when you’re a begin­ner and pow­er­ful enough to power exper­i­ments for professors?
  • Can you build the whole par­tic­i­pant expe­ri­ence under one roof?

It’s like the dif­fer­ence between Duplo and Lego. With one you can build a rudi­men­ta­ry house. With the other, you can cre­ate a mini ver­sion of Hog­warts School of Witch­craft and Wiz­ardry. Com­plete with spires. And the Whomp­ing Wil­low. Or the Star­ship Enter­prise. The point is to choose a tool where the exper­i­ment world becomes your oys­ter. So make sure your tool can deliv­er that kind of func­tion­al­i­ty for you.

A Word of Cau­tion With This Tale…

No tool is per­fect; they all have their lim­i­ta­tions. If an inter­face isn’t able to deliv­er what you’re look­ing for despite hav­ing con­sid­er­able cre­ative free­dom, maybe your task can be tweaked a bit? Some­times, a bit of smoke and mir­rors is enough to get you what you need sci­en­tif­i­cal­ly, with­out adding tech­ni­cal complexity.

For exam­ple, sci­en­tists have found that it can be hard to get browsers to trig­ger audi­to­ry and visu­al stim­uli togeth­er with the nec­es­sary degree of pre­ci­sion. How­ev­er, a great design workaround is to use video (there­by com­bin­ing the audio and visu­al stim­uli into one file) as video is wide­ly sup­port­ed in HTML5. Alter­na­tive­ly, a dif­fer­ent cod­ing approach might give you the flex­i­bil­i­ty to tight­en up the syn­chro­nised onset of audi­to­ry and visu­al stimuli.

With the lat­ter option, you’d take on the bur­den of imple­ment­ing, test­ing and val­i­dat­ing your code. Great for all the cod­ing genius­es with easy access to tim­ing val­i­da­tion equip­ment. If that lat­ter option seems like a dis­tant dream though, then the moral of the story here is to choose a tool that gives you plen­ty of flex­i­bil­i­ty — and to find cun­ning work-arounds that give you bet­ter qual­i­ty data with no addi­tion­al tech­ni­cal complexity.

A Quick Word on the Impor­tance of a Sup­port Team

If you’ve gone back­wards and for­wards try­ing to fig­ure it out and nothing’s work­ing, make sure you make use of a tool that allows you to speak to some­one who knows the inter­face inside and out — i.e. a tool backed by an effi­cient sup­port team.

In some cases, the col­lab­o­ra­tion could lead to the build­ing of a cus­tom fea­ture for your exper­i­ment, which is always a huge­ly excit­ing endeavour.

Be Smart About Your Pro­gram­ming Needs

Over­all, when it comes to pro­gram­ming, it’s impor­tant to be real­is­tic about your time and how and where it should be spent. As a behav­iour­al sci­en­tist, you don’t have a choice about work­ing hard. It’s a given in this field. But you can also work smart, and con­duct your research in a way that makes the most effi­cient use of your time. Open-source research soft­ware, tools and task builders, when cor­rect­ly uti­lized, do the invalu­able job of lift­ing some of the bur­den and stress off your shoulders.

If you want to find out more about best prac­tice when it comes to build­ing exper­i­ments online, check out our insights here.

Happy Exper­i­ment­ing!

Jo Ever­shed

Jo is the CEO and co-founder of Caul­dron and Gorilla. Her mis­sion is to pro­vide behav­iour­al sci­en­tists with the tools need­ed to improve the scale and impact of the evi­dence-based inter­ven­tions that ben­e­fit society.