Mechanical Turk
1. External Questions
You need 3 files
- input
- properties
- question
blah blah. I should say much more. FIXME.
2. Creating Balanced Lists
Amazon gives you:
HitId - identifier for a given HIT (aka "trial" to us)
WorkerId - unique identifier of the user doing the HIT
AssignmentId - unique identifier for the assignment. HitId + some sort of hash
any user created annotation - I use TrialId
Amazon creates a HIT for each trial, and creates as many assignments as you tell it to of each HIT.
We want to:
- show each worker items from only one list
- use each list the same number of times
- use each item from each list the same number of times
Problems:
- Workers can start with any HIT (trial) in a given assignment
- Workers can return HITs at any time, making them available to a new worker, but given the information Amazon gives you, there's no way for you to know when this happens, so you can automatically start the new worker where the old one left off
Solution:
If worker seen before: fetch items for trial based on list from past trials and display items Else: ???
- Assigning new workers by count of workers modulo number of lists doesn't work, as workers can return HITs at any point and throw off the list count.
- Assigning new workers by count of assignments of the current HIT (mod number of lists) doesn't work because workers can start with any HIT, so you could be assigning them to a list that's already taken by someone doing another HIT currently.
- Either way, in the pathological case you end up with some lists being overassigned and others underassigned. Based on the last experiment, many people only do one or two trials, most do less than five, so it's very easy for the pathological situation to happen.
3. Helpful Code
3.1. Geographic Info
Via Neal Snider from Robert Munro (w/ minor changes by Andrew Watts for formatting and to make it valid HTML):
If you place it in the design-view of your template, and it will use the IP address and browser settings of each Turker to populate fields with some useful demographics like 'City', 'Region', 'Country', and 'User Display Language'.
<p> <input type="hidden" name="userDisplayLanguage" /> <input type="hidden" name="browserInfo" /> <input type="hidden" name="ipAddress" /> <input type="hidden" name="country" /> <input type="hidden" name="city" /> <input type="hidden" name="region" /> </p> <script type="text/javascript" src="http://gd.geobytes.com/gd?after=-1variables=GeobytesCountry,GeobytesCity,GeobytesRegion,GeobytesIpAddress"> </script> <script type="text/javascript"> <!-- function getUserInfo() { var userDisplayLanguage = navigator.language ? navigator.language : navigator.userDisplayLanguage; var browserInfo = navigator.userAgent; var ipAddress = sGeobytesIpAddress; var country = sGeobytesCountry; var city = sGeobytesCity; var region = sGeobytesRegion; document.mturk_form.userDisplayLanguage.value = userDisplayLanguage; document.mturk_form.browserInfo.value = browserInfo; document.mturk_form.ipAddress.value = ipAddress; document.mturk_form.country.value = country; document.mturk_form.city.value = city; document.mturk_form.region.value = region; } getUserInfo(); // --> </script>
4. Tutorials
[http://www.itworld.com/internet/76659/experimenting-mechanical-turk-5-how-tos 5 Mechanical Turk Howtos]
- n.b. Amazon removed the "send an email message" feature that the article says to use to invite good workers to do followups. Workers can email you, but you can't email them anymore.