Mechanical Turk

1. External Questions

You need 3 files

• input
• properties
• question

blah blah. I should say much more. FIXME.

2. Creating Balanced Lists

Amazon gives you:

• HitId - identifier for a given HIT (aka "trial" to us)

• WorkerId - unique identifier of the user doing the HIT

• AssignmentId - unique identifier for the assignment. HitId + some sort of hash

• any user created annotation - I use TrialId

Amazon creates a HIT for each trial, and creates as many assignments as you tell it to of each HIT.

We want to:

• show each worker items from only one list
• use each list the same number of times
• use each item from each list the same number of times

Problems:

• Workers can start with any HIT (trial) in a given assignment
• Workers can return HITs at any time, making them available to a new worker, but given the information Amazon gives you, there's no way for you to know when this happens, so you can automatically start the new worker where the old one left off

Solution:

 If worker seen before:
    fetch items for trial based on list from past trials and display items
 Else:
    ???

• Assigning new workers by count of workers modulo number of lists doesn't work, as workers can return HITs at any point and throw off the list count.
• Assigning new workers by count of assignments of the current HIT (mod number of lists) doesn't work because workers can start with any HIT, so you could be assigning them to a list that's already taken by someone doing another HIT currently.
• Either way, in the pathological case you end up with some lists being overassigned and others underassigned. Based on the last experiment, many people only do one or two trials, most do less than five, so it's very easy for the pathological situation to happen.

3. Helpful Code

3.1. Geographic Info

Via Neal Snider from Robert Munro (w/ minor changes by Andrew Watts for formatting and to make it valid HTML):

If you place it in the design-view of your template, and it will use the IP address and browser settings of each Turker to populate fields with some useful demographics like 'City', 'Region', 'Country', and 'User Display Language'.

<p>
  <input type="hidden" name="userDisplayLanguage" />
  <input type="hidden" name="browserInfo" />
  <input type="hidden" name="ipAddress" />
  <input type="hidden" name="country" />
  <input type="hidden" name="city" />
  <input type="hidden" name="region" />
</p>

<script type="text/javascript" src="http://gd.geobytes.com/gd?after=-1variables=GeobytesCountry,GeobytesCity,GeobytesRegion,GeobytesIpAddress">
</script>

<script type="text/javascript">
<!--
function getUserInfo() {
   var userDisplayLanguage = navigator.language ? navigator.language :
navigator.userDisplayLanguage;
   var browserInfo = navigator.userAgent;
   var ipAddress = sGeobytesIpAddress;
   var country = sGeobytesCountry;
   var city = sGeobytesCity;
   var region = sGeobytesRegion;

   document.mturk_form.userDisplayLanguage.value = userDisplayLanguage;
   document.mturk_form.browserInfo.value = browserInfo;
   document.mturk_form.ipAddress.value = ipAddress;
   document.mturk_form.country.value = country;
   document.mturk_form.city.value = city;
   document.mturk_form.region.value = region;
}

getUserInfo();

// -->
</script>

4. Tutorials

• [http://www.itworld.com/internet/76659/experimenting-mechanical-turk-5-how-tos 5 Mechanical Turk Howtos]

  • n.b. Amazon removed the "send an email message" feature that the article says to use to invite good workers to do followups. Workers can email you, but you can't email them anymore.
  • Also, don't "ban" workers who have already done a study. It hurts their ability to do future HITs for anyone. The use of credentials they mention is difficult to do, and except for a few built-in ones (e.g. % approval rating, location), they can only be manipulated with and can only be used with experiments created with the command line tools.