Amazon Mechanical Turk

Relevant Papers / Blog Posts

Conducting behavioral research on Amazon’s Mechanical Turk by Winter Mason & Siddharth Suri should be a standard reading for anyone conducting mturk experiments in the lab.

Mechanical Turk Workers Are Not Anonymous - make sure you anonymize workerIds in anything published; they can often be trivially linked back to an individual. Our protocols and consent forms also state that we will anonymize anyway.

Example MTurk Experiments

• Artificial Language Learning

• Written recall

• Phonetic perception/adaptation

• Norming

• Syntactic priming

• Self paced reading (run with modified IBex)

• an example of spoken recording as soon as we have it

Standard Practices

Consent form blurb (always make sure it links to the current version of the consent form):

<div id="consent">
    By accepting this HIT, you confirm that you have read and understood the
    <a target="_blank" href="http://www.hlp.rochester.edu/consent/English_2013-05-13.pdf">consent form</a>,
    that you are willing to participate in this experiment, and that you
    agree that the data you provide by participating can be used in
    scientific publications (no identifying information will be used).
    Sometimes it is necessary to share the data elicited from you
    &mdash; including sound files &mdash; with other researchers for scientific
    purposes (for replication purposes). That is the only reason for
    which we will share data and we will only share data with other
    researchers and only if it is for non-commercial use. Identifying
    information will <span style="font-weight:bold;">never</span> be shared (your MTurk ID will be replaced
    with an arbitrary alphanumeric code).
</div>

Always include the following text near where you link to the consent form:

<p>Sometimes it can happen that technical difficulties cause experimental scripts to freeze so that you will not be able to submit a HIT. We are trying our best to avoid these problems. Should they nevertheless occur, we urge you to contact us.</p>

Also include this blurb (including the link to the blog):

<p>If you are interested in hearing how the experiments you are participating in help us to understand the human brain, feel free to subscribe to our <a href="http://hlplab.wordpress.com/">lab blog</a> on which we announce new findings. Note that typically about one year passes before an experiment is published.</p>

If you are recording audio from the workers include this blurb:

<p>You must be a native speaker of American English. If you have not spent almost all of your time until the age of 10 in the United States, you cannot participate. If you have a non-American accent we will not be able to pay you. For example, British or Indian English is not permissible for this experiment.</p>

Please use this example as a template for your own instructions page.

Remember also that we may only recruit US workers. This is a University of Rochester policy. By default Amazon lets you recruit from all regions, which in practice means you will get a lot of Indians (2nd largest worker area after the US).

Always include this qualification in your properties file if you do external HITs (changing all of the 2s to the appropriate number based on how many other qualifications you have):

# this is a built-in qualification -- user must be in the United States
qualification.2:00000000000000000071
qualification.comparator.2:equalto
qualification.locale.2:US
qualification.private.2:true

If you use the web based interface, when you click "Design", then go to the first tab "Enter Properties" under "Additional Qualifications" be sure to add Location is UNITED STATES.

Tips

Excluding workers who have done a similar HIT

http://mechanicalturk.typepad.com/blog/2014/07/new-qualification-comparators-add-greater-flexibility-to-qualifications-.html

Quick HOWTO:

1. you create a qualification
2. assign the qualification to all the workers who did the relevant HIT(s)
3. use the comparator

   1. For the External Question type use the new DoesNotExist comparator to exclude workers

   2. For the web interface, when choosing qualifications under "Advanced":
      1. add the qualification type
      • ii. choose "has not been granted"
      • iii. change the "Only Workers who qualify to do my HITs can preview my HITs." radio button to "Yes"

Excluding by state

http://mechanicalturk.typepad.com/blog/2015/01/us-state-qualifications.html

Mixed Content

In recent versions of Chrome (since 21), Firefox (since 23), and IE (since 9), if you mix http and https, the browser blocks all "active" http content. The MTurk pages are all https, and iframes (which external HITs are run in) count as "active" content. (JavaScript is also "active" content.)

In those browsers, if you click on the preview of a HIT with mixed content (either as a worker or in the Manage tab on the requester page) you just get a blank page and a message in the JavaScript console something like this:

[blocked] The page at https://requester.mturk.com/mturk/manageHITs? ran insecure content from http://www.hlp.rochester.edu/mturk/experiments/interactive_communication_1?…ignmentId=ASSIGNMENT_ID_NOT_AVAILABLE&hitId=2NKNJQ2172Z9EQJAQ9CC0EBVLXNYQL.

This is a fixable problem.

• Our webserver has a valid SSL certificate, so when we create HITs we need to use https in the URL in the Question file
• Make sure that all links in your HITs are either relative or use https as well.

• As detailed at the MDN page on fixing mixed content pages you should also make sure any scripts are protocol relative like:

  •     <script src="//example.com/script.js" type="text/javascript"></script>

    Note that there is neither an http or https in the src tag; browsers will automatically choose the right one based on how the page is served.

Audio

If you intend to play audio, keep in mind that not all browsers support the same formats within the <audio> tag. Make sure to provide multiple <source> tags within your <audio> block, in the order you'd prefer the browser choose them, e.g.

<audio>
    <source src="foo.m4a" type="audio/mp4" />
    <source src="foo.ogg" type="audio/ogg" />
    <source src="foo.mp3" type="audio/mpeg" />
    <source src="foo.wav" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

Use as many or few as you need to get full coverage, or at least coverage for all the browsers you care about.

Here is a table of current levels of support in the major browsers:

To convert a wav to mp3:

lame --resample 22.05 -b 64 foo.wav foo.mp3

To convert a wav to m4a:

ffmpeg -y -i foo.wav -acodec libfaac -ab 64k foo.m4a

To convert a wav to ogg:

ffmpeg -i foo.wav -acodec libvorbis -ab 64k foo.ogg

Things to monitor

• TurkOpticon - Workers give feedback on requesters at TurkOpticon. If you have a lot of bad reviews, people will not want to do your HITs. Make sure (within reason) that you are not making workers angry. If you run your own HITs, consider making an account to (politely) engage workers there.

There are 1 attachment(s) stored for this page.