Amazon Mechanical Turk

Relevant Papers / Blog Posts

Conducting behavioral research on Amazon’s Mechanical Turk by Winter Mason & Siddharth Suri should be a standard reading for anyone conducting mturk experiments in the lab.

Mechanical Turk Workers Are Not Anonymous - make sure you anonymize workerIds in anything published; they can often be trivially linked back to an individual. Our protocols and consent forms also state that we will anonymize anyway.

Example MTurk Experiments

Artificial Language Learning
Written recall
Phonetic perception/adaptation
Norming
Syntactic priming
Self paced reading (run with modified IBex)
an example of spoken recording as soon as we have it

Standard Practices

Consent form blurb (always make sure it links to the current version of the consent form):

<div id="consent">
    By accepting this HIT, you confirm that you have read and understood the
    <a target="_blank" href="http://www.hlp.rochester.edu/consent/English_2013-05-13.pdf">consent form</a>,
    that you are willing to participate in this experiment, and that you
    agree that the data you provide by participating can be used in
    scientific publications (no identifying information will be used).
    Sometimes it is necessary to share the data elicited from you
    &mdash; including sound files &mdash; with other researchers for scientific
    purposes (for replication purposes). That is the only reason for
    which we will share data and we will only share data with other
    researchers and only if it is for non-commercial use. Identifying
    information will <span style="font-weight:bold;">never</span> be shared (your MTurk ID will be replaced
    with an arbitrary alphanumeric code).
</div>

Always include the following text near where you link to the consent form:

<p>Sometimes it can happen that technical difficulties cause experimental scripts to freeze so that you will not be able to submit a HIT. We are trying our best to avoid these problems. Should they nevertheless occur, we urge you to contact us.</p>

Also include this blurb (including the link to the blog):

<p>If you are interested in hearing how the experiments you are participating in help us to understand the human brain, feel free to subscribe to our <a href="http://hlplab.wordpress.com/">lab blog</a> on which we announce new findings. Note that typically about one year passes before an experiment is published.</p>

If you are recording audio from the workers include this blurb:

<p>You must be a native speaker of American English. If you have not spent almost all of your time until the age of 10 in the United States, you cannot participate. If you have a non-American accent we will not be able to pay you. For example, British or Indian English is not permissible for this experiment.</p>

Please use this example as a template for your own instructions page.

Remember also that we may only recruit US workers. This is a University of Rochester policy. By default Amazon lets you recruit from all regions, which in practice means you will get a lot of Indians (2nd largest worker area after the US).

Always include this qualification in your properties file if you do external HITs (changing all of the 2s to the appropriate number based on how many other qualifications you have):

# this is a built-in qualification -- user must be in the United States
qualification.2:00000000000000000071
qualification.comparator.2:equalto
qualification.locale.2:US
qualification.private.2:true

If you use the web based interface, when you click "Design", then go to the first tab "Enter Properties" under "Additional Qualifications" be sure to add Location is UNITED STATES.

RSRB Survey

Any experiment run on MTurk after April 5th, 2013 should always include the survey below starting with this preamble:

<h1>Jaeger Laboratory Participant Questionnaire</h1>
<p>This is an optional questionnaire. Answering this questionnaire will not affect your participation in this study, nor will it affect your payment for participation in this study. You may answer some of these questions, all of these questions, or none of these questions. The questions are designed for one of three things:</p>
<ol>
  <li>Provide us with information that will better help us conduct the experiment and analyze the data from the experiment.</li>
<li>Provide us with information regarding participant demographics (age, race, sex, etc.) that is often requested by groups monitoring and/or funding our research. This information is requested by groups such as the National Science Foundation (NSF), the National Institute of Health (NIH), and the university’s Research Review Board (RSRB) to ensure that human research studies accurately sample the population base of the region. This also helps to make sure that certain population subsets (minorities, females, etc.) are no longer wittingly or unwittingly underrepresented in scientific research data.</li>
</ol>
<p>If after you complete and submit this questionnaire you later decide that you wish to amend or erase any information you supplied us, please let us know, as you are always free to make changes to this info.</p>

Always use this exact HTML (or at least exact form elements):

<ol>
  <li>Age
      <input type="number" min="0" max="100" step="1" name="rsrb.age"/>
  </li>
  <li>Sex
      <select name="rsrb.sex">
          <option value="NA" selected>N/A</option>
          <option value="Male">Male</option>
          <option value="Female">Female</option>
      </select>
  </li>
  <li>Ethnicity
      <select name="rsrb.ethnicity">
          <option value="NA" selected>N/A</option>
          <option value="Hisp">Hispanic or Latino</option>
          <option value="NonHisp">Not Hispanic or Latino</option>
      </select>
  </li>
  <li>Race (you may check more than one)
      <ul>
          <li><input type="checkbox" name="rsrb.race" id="race_amerind" value="amerind;"/><label for="race_amerind">American Indian / Alaska Native</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_black" value="black;"/><label for="race_black">Black or African-American</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_pacif" value="pacif;"/><label for="race_pacif">Native Hawaiian or Other Pacific Islander</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_asian" value="asian;"/><label for="race_asian">Asian</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_white" value="white;"/><label for="race_white">White</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_unknown" value="unknown;"/><label for="race_unknown">Unknown</label></li>
          <li><input type="checkbox" name="rsrb.race" id="race_other" value="other;"/><label for="race_other">Other</label></li>
      </ul>
      If Other:
      <input type="text" name="rsrb.raceother"/>
  </li>
</ol>
<input type="hidden" name="rsrb.protocol" value="RSRB00045955"/>

And make sure that the value of "rsrb.protocol" matches the protocol number your experiment is being run under.

Payment

Please pay workers in a timely manner. We are required by the RSRB to pay all workers anyway (just like we have to pay all subjects who come into the lab and are consented). Workers talk on various online forums, and we don't want to develop a bad reputation when it comes to payment.

Best practice would be to pay workers as soon as a HIT is Reviewable (i.e. all assignments have been completed). Run the commands to get the results and to approve work back to back.

If you don't want to manually approve work, set the autoapproval time to 24 hours. The default is two weeks. Letting it go this long annoys workers and has no benefit for us.

You can always block workers after the fact if they did unacceptable work.

Tips

Excluding workers who have done a similar HIT

http://mechanicalturk.typepad.com/blog/2014/07/new-qualification-comparators-add-greater-flexibility-to-qualifications-.html

Quick HOWTO:

you create a qualification
assign the qualification to all the workers who did the relevant HIT(s)
use the comparator
1. For the External Question type use the new DoesNotExist comparator to exclude workers
2. For the web interface, when choosing qualifications under "Advanced":
  1. add the qualification type
  - ii. choose "has not been granted"
  - iii. change the "Only Workers who qualify to do my HITs can preview my HITs." radio button to "Yes"

Excluding by state

http://mechanicalturk.typepad.com/blog/2015/01/us-state-qualifications.html

Mixed Content

In recent versions of Chrome (since 21), Firefox (since 23), and IE (since 9), if you mix http and https, the browser blocks all "active" http content. The MTurk pages are all https, and iframes (which external HITs are run in) count as "active" content. (JavaScript is also "active" content.)

In those browsers, if you click on the preview of a HIT with mixed content (either as a worker or in the Manage tab on the requester page) you just get a blank page and a message in the JavaScript console something like this:

[blocked] The page at https://requester.mturk.com/mturk/manageHITs? ran insecure content from http://www.hlp.rochester.edu/mturk/experiments/interactive_communication_1?…ignmentId=ASSIGNMENT_ID_NOT_AVAILABLE&hitId=2NKNJQ2172Z9EQJAQ9CC0EBVLXNYQL.

This is a fixable problem.

Our webserver has a valid SSL certificate, so when we create HITs we need to use https in the URL in the Question file
Make sure that all links in your HITs are either relative or use https as well.
As detailed at the MDN page on fixing mixed content pages you should also make sure any scripts are protocol relative like:
- ```
    <script src="//example.com/script.js" type="text/javascript"></script>
```
  Note that there is neither an http or https in the src tag; browsers will automatically choose the right one based on how the page is served.

Audio

If you intend to play audio, keep in mind that not all browsers support the same formats within the <audio> tag. Make sure to provide multiple <source> tags within your <audio> block, in the order you'd prefer the browser choose them, e.g.

<audio>
    <source src="foo.m4a" type="audio/mp4" />
    <source src="foo.ogg" type="audio/ogg" />
    <source src="foo.mp3" type="audio/mpeg" />
    <source src="foo.wav" type="audio/wav" />
    Your browser does not support the audio element.
</audio>

Use as many or few as you need to get full coverage, or at least coverage for all the browsers you care about.

Here is a table of current levels of support in the major browsers:

	m4a	mp3	ogg	wav
IE9	no	yes	no	no
Opera 11	no	no	yes	yes
Firefox 13	no	no	yes	yes
Chrome 20	yes	yes	yes	yes
Safari 5.1	yes	yes	no	yes

To convert a wav to mp3:

lame --resample 22.05 -b 64 foo.wav foo.mp3

To convert a wav to m4a:

ffmpeg -y -i foo.wav -acodec libfaac -ab 64k foo.m4a

To convert a wav to ogg:

ffmpeg -i foo.wav -acodec libvorbis -ab 64k foo.ogg

Things to monitor

TurkOpticon - Workers give feedback on requesters at TurkOpticon. If you have a lot of bad reviews, people will not want to do your HITs. Make sure (within reason) that you are not making workers angry. If you run your own HITs, consider making an account to (politely) engage workers there.

There are 1 attachment(s) stored for this page.