Amazon Mechanical Turk


What is AMT / MTurk


Amazon Mechanical Turk (AMT / MTurk) allows individuals around the world to take part in online tasks in return for payment. The basic way it works is:

  1. A requester lists their task on AMT, in the form of a URL
  2. An AMT Worker (Turker) decides to take part, and clicks on the URL
  3. Once the workers finishes the task, the task should provide them with some form of completion code
  4. The workers enters this completion code into AMT to say that they've finished the task
  5. The requester reviews completion codes in AMT. If they are valid, they verify them (and the workers gets paid). If they are invalid, they reject them (and the workers doesn't get paid).

You can sign up for an account here.

Geographic Availability


At the time of writing (May 2017) AMT is available to requesters in the following countries.

  • United States
  • Australia
  • Austria
  • Belgium
  • Botswana
  • Canada
  • Denmark
  • Estonia
  • Finland
  • France
  • Germany
  • Hong Kong
  • Iceland
  • Ireland
  • Lithuania
  • Luxembourg
  • Malta
  • Netherlands
  • New Zealand
  • Norway
  • Poland
  • Portugal
  • Singapore
  • Slovenia
  • Sweden
  • Switzerland
  • United Kingdom

Source

Turker Guidance


It's worth reading this document written by Turkers to help researchers work with them effectively.

This line is particularly important: 'Rejections leave workers with a mark counting against them on their 'permanent record' at MTurk that may take them below a qualification threshold necessary for certain other HITs.'

Wherever possible, have workers return hits rather than rejecting them.

Set Up


Simple Set Up

The simple way of listing your experiment on AMT is:

  1. Create your AMT project as a Survey Link.
  2. Set your recruitment policy to Amazon Mechanical Turk.
  3. Approve all HITs where the completion code matches a completion code in your Gorilla metrics.
  4. For any duplicates accept only the first HIT.

While this approach is suitable for a small pilot studies, there are two drawbacks:

  • As the workerID is not automatically captured, you have no way of joining your Gorilla metrics to reports that you download from AMT.
  • If a participant closes thir browser (or just the tab they are running Gorilla in), and then click the Gorilla link in the HIT again, they will start again from the beginning - their progress will not be saved.

Advanced Set Up

In order to mitigate these two problems we need to capture the AMT worker ID, so that Gorilla metrics line up with AMT reports.

AMT gives you full access to the HTML on the HIT listing page, and as a result, doesn't offer a way to automatically append the worker ID to the URL. Therefore, we need to do this ourselves, with a bit of JavaScript.

In Gorilla:

  1. Make a note of your Unique URL (e.g. https://research.sc/participant/login/dynamic/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX)

In AMT:

  1. Create your AMT project as a Survey Link
  2. Fill out the fields on the first Enter Properties page as normal

On the Design Layout page, click Source to see the HTML source for the page.

Find the line containing the URL for your survey - by default it is:

<a class="dont-break-out" href="http://[example.com/survey345.html]" target="_blank">http://example.com/survey345.html</a>

Replace it with:

<a id="gorilla-link" target="_blank" href="https://research.sc/participant/login/dynamic/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX">Survey Link</a>

Next, scroll down to the bottom where you should find some JavaScript:

$(document).ready(function() {
    // Instructions expand/collapse
    var content = $('#instructionBody');
    var trigger = $('#collapseTrigger');
    content.hide();
    $('.collapse-text').text('(Click to expand)');
    trigger.click(function(){
      content.toggle();
      var isVisible = content.is(':visible');
      if(isVisible){
        $('.collapse-text').text('(Click to collapse)');
      }else{
        $('.collapse-text').text('(Click to expand)');
      }
    });
    // end expand/collapse
  });

Change it to:

function appendWorkerIDToURL(linkElem, workerIDName) {
    var url = window.location.href;
    var name = 'workerId';
    var regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)"),
        results = regex.exec(url);
    if(results && results[2]) {
        var workerID = decodeURIComponent(results[2].replace(/\+/g, " "));
        var linkURL = $(linkElem).attr('href');
        var appendChar = linkURL.indexOf('?') >= 0 ? '&' : '?';
        $(linkElem).attr('href', linkURL + appendChar + workerIDName + '=' + workerID);
    }
}

  $(document).ready(function() {
    // Instructions expand/collapse
    var content = $('#instructionBody');
    var trigger = $('#collapseTrigger');
    content.hide();
    $('.collapse-text').text('(Click to expand)');
    trigger.click(function(){
      content.toggle();
      var isVisible = content.is(':visible');
      if(isVisible){
        $('.collapse-text').text('(Click to collapse)');
      }else{
        $('.collapse-text').text('(Click to expand)');
      }
    });
    // end expand/collapse
    appendWorkerIDToURL('#gorilla-link', 'external_id');
  });

This creates a function which looks at the current URL, retrieves the workerId and appends it to our survey link. We then call that function from within $(document).ready(), passing in our survey link that we created earlier.

You can now reliably verify your AMT HITs as follows:

  1. Download your experiment metrics
  2. For each participant, note the Public ID and Completion Code
  3. Open your AMT HITs
  4. Approve HITs where the workerId and completion code in AMT match the Public ID and Completion Code in Gorilla

Screening Participants


If your experimental design contains conditions where you wish to reject or screen out a participant, direct them to a Reject Node.

This, unlike a finish node, will not show them a completion code.

You may also want to include a simple questionnaire before the secondary reject node which tells the participant to return to AMT and return the HIT. This gives the worker an opportunity to protect their reputation.

Preventing Repeat Participants


By default, AMT will automatically prevent the same worker doing the same HIT multiple times.

However, some researchers deploy their experiments in batches of 9 HITs to avoid paying AMT a higher rate of commission. When researchers do this, AMT doesn't prevent the same worker doing multiple HITs because AMT has no way of knowing that the batches are related.

Nevertheless, on the Gorilla side, it is just one experiment. If you have followed the Proper Set Up instruction above, Gorilla will detect that the same worker is accessing the experiment again and resumes them where they left off. If they have reached the finish node, with the completion code, it will resume there.

From the participant's perspective, they would click on the HIT, press start and immediately be given a completion code.

If this happens, you want these participants to return the HIT, so that you don't have to reject it. To encourage this, make it clear in the task listing on AMT that if they get the experience above, they should return the HIT. It would be ideal to make this section of the instructions salient to the worker.

BEWARE: If you use ALL CAPS or RED workers may get put off your task completely!

An alternative approach, which is more reliable, is to issue a Qualification to each participant that completes one of your batches. You can then set this qualification as an exclusion criterion. This ensures that subjects who have already participated would not see the HITs again.

Novice Participants


Novice Turkers, without a reputation, may not have access to tasks that require a good reputation.

Novice Turkers are particularly valuable to researchers. To attract them, put out a HIT with a 0 cent payment and an appropriate bonus.

Longitudinal Studies


It is possible to run longitudinal studies in AMT.

  1. Set up a fresh AMT account
  2. Create a qualification task
  3. Give all participants that complete your qualification task the relevant qualification
    • For instance - Qualification Exp123_T1 - might remind me that these participants have completed the first test battery for Experiment 123
  4. Launch the secondary task making the qualification above mandatory
    • You can message all your workers for a nominal fee (1 cent + commission). This is why you want to use a fresh account.

If you want more details, don't hesitate to get in touch.