Mechanical Turk

Two ways to use Mechanical Turk

  1. For simple task designs use the web-based requester interface.
  2. For more complex designs use the command-line tools for External Questions.

1. Web-based requester interface

You should use the [https://requester.mturk.com/mturk/ web-based requester interface] (for the running actual subjects) or the web-based [https://requestersandbox.mturk.com/mturk/ requester sandbox] (for testing HITs before running actual subjects) if your experiment falls in one of the following categories:

Lists are uploaded as CSV files and results are downloaded as CSV files. A template for a HIT uses the [http://en.wikipedia.org/wiki/Apache_Velocity Velocity] templating language (or possibly just a subset?) to fill in any variables coming from your CSV files. Each row represents one HIT in your HIT group.

While your HITs are running, the requester site displays a progress bar that shows how many of the HITs in your group have completed.

2. External Questions

With External Questions, you host the experiment on an external server, but the results are [http://en.wikipedia.org/wiki/POST_%28HTTP%29 POSTed] to the the Mechanical Turk (or Mechanical Turk Sandbox) site.

You are responsible for the creation of any HTML templates your HIT needs, any backend (e.g. database) to fill the templates, CGI script(s) to present the templates, etc. All Amazon does is pass certain variables (discussed later on this page) to your webserver, display the page you specify in an iframe, and accept the POSTed results.

To upload an External Question, you need 3 files

as well as Amazon's [http://aws.amazon.com/developertools/694?_encoding=UTF8&jiveRedirect=1 command line tools] (written in Java, so you also need a JVM on your computer). For further documentation see http://mturk.s3.amazonaws.com/CLT_Tutorial/UserGuide.html

The results that your script POSTs to Mechanical Turk also are retrieved (as CSV files) using the command line tools when using External Questions. There is no progress bar on the requester site for External Question HITs.

Amazon gives your script:

2.1. Creating Balanced Lists

Amazon creates a HIT for each trial, and creates as many assignments as you tell it to of each HIT.

We want to:

Problems:

Solution: 

 If worker seen before:
    fetch items for trial based on list from past trials and display items
 Else:
    ??? Somehow assign them to one of the lists, attempting to get an equal number of workers on each list for each item.

One way to improve the number of HITs each worker completes in a group is to offer a bonus. Pay a low amount for each HIT, but state in the instructions that you will pay a bonus for workers that complete certain amounts of HITs, e.g. On an experiment with 16 HITs (possibly with multiple trials within a given HIT) pay $0.10 per HIT, but at 5 HITs pay a $0.45 bonus, at 10 HITs pay a $1.50 bonus, and at 16 trials pay a $3.90 bonus, where each worker gets paid for exactly one bonus level (the highest bonus level they completed). An example graphic that went with just such an experiment is [http://www.hlp.rochester.edu/mturk/compensation_crmrecall1.png at this link]. It helps to make a visual aid like this to make it clear to the workers that it's worth their time to do as many HITs in the group as they can.

Helpful Code

1. Geographic Info

Via Neal Snider from Robert Munro (w/ minor changes by Andrew Watts for formatting and to make it valid HTML):

If you place it in the design-view of your template, and it will use the IP address and browser settings of each Turker to populate fields with some useful demographics like 'City', 'Region', 'Country', and 'User Display Language'. 

<p>
  <input type="hidden" name="userDisplayLanguage" />
  <input type="hidden" name="browserInfo" />
  <input type="hidden" name="ipAddress" />
  <input type="hidden" name="country" />
  <input type="hidden" name="city" />
  <input type="hidden" name="region" />
</p>

<script type="text/javascript" src="http://gd.geobytes.com/gd?after=-1variables=GeobytesCountry,GeobytesCity,GeobytesRegion,GeobytesIpAddress">
</script>

<script type="text/javascript">
<!--
function getUserInfo() {
   var userDisplayLanguage = navigator.language ? navigator.language :
navigator.userDisplayLanguage;
   var browserInfo = navigator.userAgent;
   var ipAddress = sGeobytesIpAddress;
   var country = sGeobytesCountry;
   var city = sGeobytesCity;
   var region = sGeobytesRegion;

   document.mturk_form.userDisplayLanguage.value = userDisplayLanguage;
   document.mturk_form.browserInfo.value = browserInfo;
   document.mturk_form.ipAddress.value = ipAddress;
   document.mturk_form.country.value = country;
   document.mturk_form.city.value = city;
   document.mturk_form.region.value = region;
}

getUserInfo();

// -->
</script>

Tutorials

Papers

[attachment:KapelnerChandler-PreventingSatisficingInOnlineSurveys.pdf Preventing Satisficing In Online Surveys]

MoinMoin Appliance - Powered by TurnKey Linux