Mechanical Turk

1. Creating Balanced Lists

Amazon gives you:

• HitId - identifier for a given HIT (aka "trial" to us)

• WorkerId - unique identifier of the user doing the HIT

• AssignmentId - unique identifier for the assignment. HitId + some sort of hash

• any user created annotation - I use TrialId

Amazon creates a HIT for each trial, and creates as many assignments as you tell it to of each HIT.

We want to:

• show each worker items from only one list
• use each list the same number of times
• use each item from each list the same number of times

Problems:

• Workers can start with any HIT (trial) in a given assignment
• Workers can return HITs at any time, making them available to a new worker, but given the information Amazon gives you, there's no way for you to know when this happens, so you can automatically start the new worker where the old one left off

Solution:

• ???

2. Helpful Code

2.1. Geographic Info

Via Neal Snider from Robert Munro (w/ minor changes by Andrew Watts for formatting and to make it valid HTML):

If you place it in the design-view of your template, and it will use the IP address and browser settings of each Turker to populate fields with some useful demographics like 'City', 'Region', 'Country', and 'User Display Language'.

<p>
  <input type="hidden" name="userDisplayLanguage" />
  <input type="hidden" name="browserInfo" />
  <input type="hidden" name="ipAddress" />
  <input type="hidden" name="country" />
  <input type="hidden" name="city" />
  <input type="hidden" name="region" />
</p>

<script type="text/javascript" src="http://gd.geobytes.com/gd?after=-1variables=GeobytesCountry,GeobytesCity,GeobytesRegion,GeobytesIpAddress">
</script>

<script type="text/javascript">
<!--
function getUserInfo() {
   var userDisplayLanguage = navigator.language ? navigator.language :
navigator.userDisplayLanguage;
   var browserInfo = navigator.userAgent;
   var ipAddress = sGeobytesIpAddress;
   var country = sGeobytesCountry;
   var city = sGeobytesCity;
   var region = sGeobytesRegion;

   document.mturk_form.userDisplayLanguage.value = userDisplayLanguage;
   document.mturk_form.browserInfo.value = browserInfo;
   document.mturk_form.ipAddress.value = ipAddress;
   document.mturk_form.country.value = country;
   document.mturk_form.city.value = city;
   document.mturk_form.region.value = region;
}

getUserInfo();

// -->
</script>