Gorilla Par­tic­i­pants, their devices and where they come from

Do you know your par­tic­i­pants? Each time a par­tic­i­pant launch­es a study on Gorilla, we col­lect some basic infor­ma­tion about the equip­ment they are using and their loca­tion. We have col­lat­ed and analysed sum­ma­ry sta­tis­tics on around 200,000 par­tic­i­pants whose data was includ­ed in a data down­load after tak­ing part in a study on the platform.

If you run stud­ies online, or are plan­ning to in the near future, this arti­cle will tell you — through descrip­tive data — about the equip­ment that your par­tic­i­pants are like­ly to be using, and hint at the demo­graph­ics you may have access to.

Devices

The vast major­i­ty of par­tic­i­pants use a com­put­er (Desk­top or Lap­top). Note this does not add to 100% due to some devices not enabling log­ging on their browsers.

Par­tic­i­pants accessed Gorilla and com­plet­ed exper­i­ments using over 1100 dif­fer­ent devices, rang­ing from Desk­top Com­put­ers to touch-screen mp3 play­ers — and even 11 Xbox users. This shows the strength of Gorilla as a flex­i­ble plat­form, as researchers do not have to adapt their tasks to work on a mas­sive range of devices — the task builder sorts this out for you.

Smart­phones account­ed for 20% of our users. The most pop­u­lar smart­phone devices for par­tic­i­pants were the iPhone, fol­lowed by Sam­sung range and then by Huawei devices. We can com­pare this to infor­ma­tion from StatsCounter which col­lects user data from over 10 bil­lion page views every month. Rel­a­tive to this broad­er mar­ket Gorilla Par­tic­i­pants are much more like­ly to be using an Apple phone (54% vs 22%), this is like­ly rep­re­sen­ta­tive of the con­sumer mar­kets par­tic­i­pants are locat­ed in (more on this below).

The rel­a­tive­ly small num­ber of tablet users was dom­i­nat­ed by iPads, fol­lowed by Sam­sung users. The oth­ers amount­ed to an almost neg­li­gi­ble num­ber. This is like­ly a biased sam­ple, as it is so small.

The dom­i­nance of desktop/laptop may reflect the desire of researchers con­cerned with pre­ci­sion tim­ing and con­sis­ten­cy across par­tic­i­pants using the Require­ments fea­ture in Gorilla to limit par­tic­i­pants to Com­put­ers (Desk­tops and Lap­tops). A screen­shot of this can be seen below.

Gorilla.sc has built in tools to limit par­tic­i­pants by brows­er or device. This will help con­trol your data quality.

The mas­sive vari­ety we see in user’s devices real­ly out­lines the impor­tance of (1) using the brows­er test­ing tools to check the appear­ance on a range of devices, or (2) apply­ing a require­ment in order to limit your par­tic­i­pants to a spe­cif­ic device or brows­er.

 

Browsers

Note this does not add to 100% due to some devices not enabling log­ging on their browsers.

We can see a het­ero­gene­ity of browsers here. Chrome, by far, is the most com­mon — which is in line with glob­al­ly report­ed trends on usage (64% used Chrome) from StatsCounter.

We now see that more par­tic­i­pants are access­ing exper­i­ments using the Face­book app brows­er than they are access­ing with inter­net explor­er. This is like­ly a reflec­tion that researchers com­mon­ly adver­tise their research on Face­book groups — par­tic­i­pants are high­ly like­ly to click on those links on the mobile app (which uses this brows­er). The per­cent­age of users using the Face­book brows­er (4%) is much high­er than the broad­er user stats linked to above.

This also out­lines the decline of Inter­net Explor­er which is at 2% of our sam­ples, with Edge being ahead. Also, note that more par­tic­i­pants are using the mobile ver­sion of Safari than the local ver­sion. WebKit is the mobile Safari brows­er but used on other devices (e.g. Kindle’s use WebKit).

 

Oper­at­ing Systems

Pre­dictably, Win­dows remains the dom­i­nant OS in use. Com­par­ing this again to stats from Stat­Counter, we can see that, over­all, our par­tic­i­pants are more like­ly to use some form of Laptop/Desktop over a mobile device than the aver­age world user, and are more like­ly to use macOS than the aver­age world user. This prob­a­bly reflects the require­ment of being an online par­tic­i­pant (eas­i­er on a per­son­al com­put­er), and the west­ern device mar­ket (more Macs).

 

Recruit­ment Platforms

There are a vari­ety of ways that you can recruit par­tic­i­pants to take part in a study on Gorilla. The most straight­for­ward one is a sim­ple link that can be shared how­ev­er you please (online, on posters, via email). Many researchers are moti­vat­ed to take their research online in order to use a recruit­ment ser­vice (e.g. Pro­lif­ic, mTurk, Qualtrics Pan­els, SONA, etc) to source participants.

You can see from the graph below, that using recruit­ment ser­vices accounts for 53% of par­tic­i­pants, the other 43% come from sim­ple links, where­as 3% are under a pilot recruit­ment pol­i­cy. This pol­i­cy is often used to pilot tasks before send­ing them out into the wilderness.

About 1% of par­tic­i­pants are using the more niche require­ments poli­cies. For instance, super­vised (which cre­ate a unique log in code for each par­tic­i­pant) and is often used in classrooms.

Dis­tri­b­u­tion of users recruit­ing via mTurk vs Prolific

MTurk and Pro­lif­ic are the most com­mon recruit­ment plat­forms in use. They both have pre-qual­i­fied participants/workers who are paid to take part in research. With­in the recruit­ment plat­form, these par­tic­i­pants fol­low a link to a study host­ed on Gorilla. Gorilla then hosts the exper­i­ments and cap­tures the data. A com­ple­tion ver­i­fi­ca­tion process then allows the par­tic­i­pant to col­lect their reimbursement.

Com­par­ing the two pop­u­lar recruit­ment plat­forms we can see that the geo­graph­ic dis­tri­b­u­tion of par­tic­i­pants recruit­ed onto Gorilla is dif­fer­ent. The major­i­ty of Pro­lif­ic par­tic­i­pants are with­in Europe, with almost a third in Amer­i­ca, and a much small­er num­ber in Aus­tralia, Africa, and Asia. MTurk par­tic­i­pants are over­whelm­ing­ly like­ly to be from Amer­i­ca, with a much high­er num­ber locat­ed in Africa or Asia rel­a­tive to Prolific.

This data sig­ni­fies the impor­tance of choos­ing your recruit­ment plat­form care­ful­ly, depend­ing on who you wish to be rep­re­sent­ed in your research. Both mTurk and Pro­lif­ic allow you to spec­i­fy your tar­get loca­tion, how­ev­er, with­in each plat­form this is like­ly to effect your uptake rate and poten­tial­ly your total sam­ple size.

Loca­tions

Con­ti­nent

The par­tic­i­pants recruit­ed on Gorilla are main­ly based in Europe, fol­lowed by Amer­i­ca, then Asia, and then Australia.

Time­zone

This is also reflect­ed in the users by city timezone:

It’s worth not­ing that the ‘cities’ record­ed in the time­zone brows­er data are like­ly to be course — e.g. ‘Lon­don’ rep­re­sents the time­zone that cov­ers the entire­ty of the Unit­ed King­dom as well Por­tu­gal and the west coast of Africa. In the Unit­ed States, Chica­go, New York, and Los Ange­les are like­ly to rep­re­sent dif­fer­ent times and cover large areas in the U.S.

This geo­graph­ic dis­tri­b­u­tion prob­a­bly goes part of the way to explain­ing why our user’s devices and brands are so depart­ed from the broad inter­net user pop­u­la­tion. For instance, when we look at UK inter­net users, the most com­mon mobile device is an Apple phone.

In order to access any given pop­u­la­tion, The Gorilla Exper­i­ment Builder allows you to define which loca­tions you want to restrict your users to.

Screen Size

The 99th per­centile of Device screen dimen­sions. Kerned Den­si­ty Plots & his­tograms with bin widths of 30 pix­els shown for each axis.

Through the use of our log­ging tools, Gorilla is able to detect the screen size users are using. Of inter­est to researchers is the vari­ance of screen dimen­sions we record­ed. These ranged from 320×205 pix­els to 2800×5982 pix­els. The graph above had to be fil­tered to the 99th quan­tile to get rid of these extremes — oth­er­wise it was very dif­fi­cult to interpret.

The clus­ter­ing of points along diag­o­nal lines indi­cates the com­mon aspect ratio for mon­i­tors, the most com­mon by far (indi­cat­ed by the lower blue line above) is the 16:9 aspect ratio.

The orange/mobile points appear to be in two diag­o­nal clus­ters, this rep­re­sents par­tic­i­pants with land­scape vs por­trait orientations.

The width of screens is rel­a­tive­ly dis­tinct for each device type, where­as the height seems to be more close­ly grouped. This like­ly rep­re­sents the mix between por­trait and land­scape use for mobiles and tablets — which is less com­mon than in com­put­er users.

This vari­ance out­lines the impor­tance of think­ing about par­tic­i­pant screen dimen­sions in your online research — espe­cial­ly if you are run­ning tasks that would nor­mal­ly be done on a spec­i­fied mon­i­tor in a lab set­ting. Gorilla allows this by restrict­ing by device type — Com­put­ers will be much bet­ter for these types of research.

Brows­er Win­dow Coverage

The brows­er win­dow refers to the space in which the brows­er has to dis­play con­tent. This can vary a lot, as users may not put their brows­er in full screen, or max­imise the win­dow. To try and cap­ture this vari­ance we cal­cu­lat­ed the per­cent­age of each participant’s screen that was cov­ered by the brows­er window.

Ker­nel Den­si­ty Esti­ma­tion for each devices’ view­port cov­er­age on the screen. Binned with a width of 2%. The lines above the X axis shows indi­vid­ual participant’s points for reference.

Rather reas­sur­ing­ly the cov­er­age was, on aver­age, rel­a­tive­ly high. Par­tic­i­pants had a mean of 81% cov­er­age, with a stan­dard devi­a­tion of 11%. This means that most par­tic­i­pants have a large amount of the screen cov­ered by the Gorilla experiment.

Com­put­ers have much more vari­able cov­er­age, as you can see by the tail in the graph above. The stan­dard devi­a­tion of com­put­ers was 11% cov­er­age, com­pared with 6% for mobile and tablets. Although, this is obvi­ous­ly because these devices have a lim­it­ed abil­i­ty to shrink win­dows with­in them.

 

A note about graphs

If you’re into that kinda thing, the graphs were con­struct­ed in Python using:

  • Seaborn
  • Mat­plotlib
  • geopan­das (for the map — Chloro­pleth if you’re being technical)
  • The screen size plot was made by adap­tion from this code: https://stackoverflow.com/a/55165689

Writ­ten by Alex Anwyl-Irvine

Alex is a part-time Devel­op­er at Gorilla and a PhD stu­dent at the MRC Cog­ni­tion and Brain Sci­ence in Cambridge.

Alex helps build con­cepts for new fea­tures, runs sci­en­tif­ic val­i­da­tion on the plat­form, writes up results and helps exper­i­menters cre­ate some of the more com­plex online experiments.

Alex’s PhD — super­vised by Dr Dun­can Astle — is on the devel­op­ment of resilience in chil­dren, and how this can be mod­elled through brain scans and big-data methodologies.