Mechanical Turk
1. Creating Balanced Lists
Amazon gives you:
HitId - identifier for a given HIT (aka "trial" to us)
WorkerId - unique identifier of the user doing the HIT
AssignmentId - unique identifier for the assignment. HitId + some sort of hash
any user created annotation - I use TrialId
Amazon creates a HIT for each trial, and creates as many assignments as you tell it to of each HIT.
We want to:
- show each worker items from only one list
- use each list the same number of times
- use each item from each list the same number of times
Problems:
- Workers can start with any HIT (trial) in a given assignment
- Workers can return HITs at any time, making them available to a new worker, but given the information Amazon gives you, there's no way for you to know when this happens, so you can automatically start the new worker where the old one left off
Solution:
- ???
2. Helpful Code
2.1. Geographic Info
Via Neal Snider from Robert Munro:
If you place it in the design-view of your template, and it will use the IP address and browser settings of each Turker to populate fields with some useful demographics like 'City', 'Region', 'Country', and 'User Display Language'.
<p><input type="hidden" name="userDisplayLanguage" /> <input type="hidden" name="browserInfo" /> <input type="hidden" name="ipAddress" /> <input type="hidden" name="country" /> <input type="hidden" name="city" /> <input type="hidden" name="region" /></p> <script language="Javascript" src="http://gd.geobytes.com/gd?after=-1&variables=GeobytesCountry,GeobytesCity,GeobytesRegion,GeobytesIpAddress"> </script> <script language="Javascript"> <!-- function getUserInfo() { var userDisplayLanguage = navigator.language ? navigator.language : navigator.userDisplayLanguage; var browserInfo = navigator.userAgent; var ipAddress = sGeobytesIpAddress; var country = sGeobytesCountry; var city = sGeobytesCity; var region = sGeobytesRegion; document.mturk_form.userDisplayLanguage.value = userDisplayLanguage; document.mturk_form.browserInfo.value = browserInfo; document.mturk_form.ipAddress.value = ipAddress; document.mturk_form.country.value = country; document.mturk_form.city.value = city; document.mturk_form.region.value = region; } getUserInfo(); // --> </script>