Amazon Mechanical Turk

Relevant Papers / Blog Posts

Conducting behavioral research on Amazon’s Mechanical Turk by Winter Mason & Siddharth Suri should be a standard reading for anyone conducting mturk experiments in the lab.

Mechanical Turk Workers Are Not Anonymous - make sure you anonymize workerIds in anything published; they can often be trivially linked back to an individual. Our protocols and consent forms also state that we will anonymize anyway.

Example MTurk Experiments

Standard Practices

Consent form blurb (always make sure it links to the current version of the consent form):

<div id="consent">
    By accepting this HIT, you confirm that you have read and understood the
    <a target="_blank" href="http://www.hlp.rochester.edu/consent/English_2013-05-13.pdf">consent form</a>,
    that you are willing to participate in this experiment, and that you
    agree that the data you provide by participating can be used in
    scientific publications (no identifying information will be used).
    Sometimes it is necessary to share the data elicited from you
    &mdash; including sound files &mdash; with other researchers for scientific
    purposes (for replication purposes). That is the only reason for
    which we will share data and we will only share data with other
    researchers and only if it is for non-commercial use. Identifying
    information will <span style="font-weight:bold;">never</span> be shared (your MTurk ID will be replaced
    with an arbitrary alphanumeric code).
</div>

Always include the following text near where you link to the consent form:

<p>Sometimes it can happen that technical difficulties cause experimental scripts to freeze so that you will not be able to submit a HIT. We are trying our best to avoid these problems. Should they nevertheless occur, we urge you to contact us.</p>

Also include this blurb (including the link to the blog):

<p>If you are interested in hearing how the experiments you are participating in help us to understand the human brain, feel free to subscribe to our <a href="http://hlplab.wordpress.com/">lab blog</a> on which we announce new findings. Note that typically about one year passes before an experiment is published.</p>

If you are recording audio from the workers include this blurb:

<p>You must be a native speaker of American English. If you have not spent almost all of your time until the age of 10 in the United States, you cannot participate. If you have a non-American accent we will not be able to pay you. For example, British or Indian English is not permissible for this experiment.</p>

Please use this example as a template for your own instructions page.

Remember also that we may only recruit US workers. This is a University of Rochester policy. By default Amazon lets you recruit from all regions, which in practice means you will get a lot of Indians (2nd largest worker area after the US).

Always include this qualification in your properties file if you do external HITs (changing all of the 2s to the appropriate number based on how many other qualifications you have):

# this is a built-in qualification -- user must be in the United States
qualification.2:00000000000000000071
qualification.comparator.2:equalto
qualification.locale.2:US
qualification.private.2:true

If you use the web based interface, when you click "Design", then go to the first tab "Enter Properties" under "Additional Qualifications" be sure to add Location is UNITED STATES.

Tips

Mixed Content

In recent versions of Chrome (since 21), Firefox (since 23), and IE (since 9), if you mix http and https, the browser blocks all "active" http content. The MTurk pages are all https, and iframes (which external HITs are run in) count as "active" content. (JavaScript is also "active" content.)

In those browsers, if you click on the preview of a HIT with mixed content (either as a worker or in the Manage tab on the requester page) you just get a blank page and a message in the JavaScript console something like this:

[blocked] The page at https://requester.mturk.com/mturk/manageHITs? ran insecure content from http://www.hlp.rochester.edu/mturk/experiments/interactive_communication_1?…ignmentId=ASSIGNMENT_ID_NOT_AVAILABLE&hitId=2NKNJQ2172Z9EQJAQ9CC0EBVLXNYQL.

This is a fixable problem.

Audio

If you intend to play audio, keep in mind that not all browsers support the same formats within the <audio> tag. Make sure to provide multiple <source> tags within your <audio> block, in the order you'd prefer the browser choose them, e.g.

<audio>
    <source src="foo.m4a" type="audio/mp4" />
    <source src="foo.ogg" type="audio/ogg" />
    <source src="foo.mp3" type="audio/mpeg" />
    <source src="foo.wav" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

Use as many or few as you need to get full coverage, or at least coverage for all the browsers you care about.

Here is a table of current levels of support in the major browsers:

m4a

mp3

ogg

wav

IE9

no

yes

no

no

Opera 11

no

no

yes

yes

Firefox 13

no

no

yes

yes

Chrome 20

yes

yes

yes

yes

Safari 5.1

yes

yes

no

yes

To convert a wav to mp3:

lame --resample 22.05 -b 64 foo.wav foo.mp3

To convert a wav to m4a:

ffmpeg -y -i foo.wav -acodec libfaac -ab 64k foo.m4a

To convert a wav to ogg:

ffmpeg -i foo.wav -acodec libvorbis -ab 64k foo.ogg

Things to monitor


There are 1 attachment(s) stored for this page.

MTurkExperiments (last edited 2014-05-20 16:06:16 by AndrewWatts)