Guidelines for running experiments

Last updated: 3/25/14 by Andrew

Table of Contents

• Well in advance of starting an experiment

  • RSRB

  • EPRP

• Before designing stimuli

• Getting ready to run an experiment

  • Recruitment

  • Scheduling participants

• While the experiment is running

  • Deviations and violations

  • Consent Forms

  • updated: Ensuring that participants pay attention (01/26/11)

  • Participant payment

  • new: Bilingual participants (10/01/10)

  • Participant sheets

  • Where to store forms

• Coordinating with other experiments

• After the experiment is finished

  • After a Mechanical Turk experiment is finished

1. Well in advance of starting an experiment

1.1. RSRB

At least one month before running an experiment, the experimenter needs to check whether there is RSRB approval for the study they plan to conduct. In particular, is both the method and the language covered by valid RSRB approval? Will the approval still be valid throughout the entire expected duration of the study (which may take several months)? Will the experiment require consent forms in a language that we do not yet have available (e.g. we have approval to run studies on Chinese, but only on Chinese speakers on campus, all of whom speak English, so we do NOT have a consent form in Chinese yet)?

1.2. EPRP

If you have not gotten an EPRP (Ethical Principals in Research Program) number, please go to the EPRP website and take the test to get one before you run any subjects and sign consent forms. It is mandatory that any faculty, grad students, post-docs, or staff who are on the protocol as investigators or co-investigators have one. Undergrads need to get an EPRP number, because you are required to have one if you ever run subjects for us, even though you most likely won't be on the protocol and consent forms.

EPRP numbers do expire every few years. Be sure to know when yours does, and take steps to get it renewed in time that your ability to run experiments is not interrupted.

2. Before you start designing stimuli

Create a wiki-page for your experiment. It should contain detailed information about the design, the type of stimuli, the methods you plan to use, the number of items you will need, how you plan to analyze the data, etc. This page should only be readable to hlp lab members (see how to restrict access to wiki pages). If you do not know how to edit the wiki, meet with the lab manager to get instructions (it's easy). You can find an example for a good experiment summary wiki page here. This page should later also contain a link to the annotation guidelines for RAs -- even if you use a standard annotation scheme.

3. Getting ready to run an experiment

3.1. Recruitment

Recruitment for our lab can be done three ways:

1. Via email. Each semester the department provides an Excel spreadsheet to lab managers with contact information for students willing to participate in the subject pool. If you want access to the list, please ask the lab manager. You can only email the students on this list or subjects who have run in experiments in our lab who have indicated willingness to participate in more experiments. Subject lists cannot be shared between labs, with the exception of the department-wide list (which has special RSRB approval).
2. Via flyers. Any flyers must be approved by the RSRB and have their stamp on them. Flyers can only be posted in locations listed in our approved protocol. Currently, we have two approved flyers (for single-day and 5-day experiments) and permission to put flyers up on the campuses of UR, RIT, and MCC. If you can sign into the RSRB system and are on our protocol, you can download approved flyers from there, otherwise you should ask the lab manager for the approved flyer.

3. Via SONA. Experimenters can add their experiments to the BCS SONA system (currently Tanenhaus and Jaeger labs only) and have it approved by an administrator (i.e. lab manager). We cannot directly contact subjects this way, but if someone creates an account on our SONA they can search experiments and sign up for time slots.

People responding to flyers will email the lab email account hlpstudy@gmail.com. All the RAs in the lab should have access to this. Separate labels are to be made for each study, and the RA responsible for running subjects for that study should move emails that they respond to into that folder and out of the inbox. Responding to emails is primarily on a first come, first serve basis (studies that are more time sensitive may be prioritized for subjects).

Any method not listed is not allowed. If you want to recruit another way, ask the lab manager for advice on the possibility of and method for getting further methods approved.

NB: For subject recruited from off-campus locations (e.g. RIT) to come to our campus lab, please be clear that they will be reimbursed the $3 parking cost on top of the subject payment.

3.2. Scheduling participants

When scheduling participants, please check the lab calendar. No experiments should be run during lab meetings or other regular meetings which are all listed in the lab calendar. Doing so, distracts participants and leads to unreliable results.

It also is interruptive to the meetings being held in the lab.

4. While the experiment is running

4.1. Deviations and violations

Unfortunately, there are going to be deviations and violations of RSRB protocols every once in a while. Although we need to do all we can to avoid them, we will keep track of those that occur. It's still better to do that than not to do it. So, if you are in doubt about something, you should ask the lab manager and/or Florian. In short, please inform us about possible deviations or violations. The consequences for the lab, if you fail to report such deviations, are quite severe.

Below, you'll first learn about general considerations about consent forms, payment form (who got paid how much), and subject sheets (who you're running in what list). Then read on to see where all these forms should be stored and how should they be organized. It is very important that you follow these guidelines.

4.2. Consent forms

Anyone running experiments in the HLP/Jaeger lab paid for out of Florian's account must use a Cross-linguistic Studies on Human Communication or Linguistic Adaptation in a Variable Environment consent form.

Only consent forms with an RSRB stamp are valid. Even if a copy is translated literally from an approved consent form, it is only valid to use that form if it itself has been approved and hence has an RSRB stamp on it. Also, the first page of the consent form must be on University of Rochester letterhead.

We currently have consent forms in a variety of languages and for studies in several languages, as well as an Amazon Mechanical Turk consent form (English only) online at http://www.hlp.rochester.edu/consent/. Before you run an experiment, make sure our RSRB protocol covers that language and that you are using the consent form in the right language.

If you are on the protocol as a co-investigator, you can go to the RSRB website, sign in and go to the Documents tab on the protocol to download the current approved consent form. These are not on letterhead, so you'll have to print them on letterhead if you download them yourself. Andrew will also keep master copies of the current consent forms in the lab, and will copy them for you if you let him know you are running a study. He will also email the lab if the consent forms change so you always have the most up to date.

Consent forms expire once a year after the Continuing Review, at which point we get new versions with the next year's stamp. Make sure that you have the current version of the consent form. The expiration date is on the RSRB stamp on the form. The Amazon Mechanical Turk consent forms are stored on the webserver with the expiration date in the filename, i.e Languagename-YYYY-MM-DD.pdf

Each time you run a subject, you are required to get a consent form for them, even if they have run in experiments in the past, or even another day of the same experiment. (i.e. If you have a 5 day experiment, you need 5 consent forms for each subject.) If the consent form changes while you are in the middle of a study, you need to start using the new consent form immediately, not when you finish the current study. The RSRB also now requires that we give each subject a copy of the consent form. Previously they only required that we offer the subject a copy. You can give them an unsigned consent form; you don't have to photocopy the signed one. Most subjects don't actually want a copy of the consent form, so please stress that we are required to give it to them, but inform them that if they don't want it they can leave it on the table on their way out.

Do not leave forms with personally identifying information (e.g. consent forms and subject running lists) out in the lab! When you leave the lab, make sure all paperwork is in its folder in a filing drawer.

4.3. Ensuring that participants pay attention

IMPORTANT: Participants do not necessarily understand that they need to be completely focused on the task they are doing for your experiment. Please follow the steps outlined below to increase our chances to get meaningful results. These techniques have been employed by other labs successfully for many years:

1) Inform participants that they are participating in a science experiment. Their attention is appreciated and crucial to the success of the experiment.

2) ALWAYS explicitly remind subjects that they are not allowed to use their cell phones (for any reason) or open any other applications on the computer while they are doing the experiment. The best strategy is to have them leave all their personal belongings in the lab, not in the experimental booth.

3) Whenever possible, sit in the same room as the participant. This may not be possible in the booths. If so, inform participants that you will be in a neighboring room, if they need any help. Generally, we want participants to be aware that we are monitoring them. That's part of being in an experiment. So, it's perfectly fine, if not preferred, to look through the windows into the booth, just to make sure that participants are actually focusing on the task.

4.4. Paying Participants (Who has been paid what?)

Every time you pay a participant, you also need to have them sign a payment sheet. That is our receipt and it allows us to get reimbursed. It also is necessary in case we get audited (about every 2-3 years).

Each subject must sign a payment form and provide their home address, which cannot be a P.O. Box or campus box. If the subject is a student and willing, it would be helpful for accounting purposes to get their student ID number. In the past, we got SSNs, but that is no longer allowed. Payment forms are available from Andrew or Dana, as well as being available on this page: Subject_Payment_Form.xls. Payment forms must be returned to Meliora 360, so the department can be reimbursed and we can keep getting money to pay subjects.

Money for paying subjects comes from Chris Dambra or Jennifer Gillis in the BCS office in Meliora 360. You can only get money or return payment forms from 10am-12pm and 2pm-4pm Monday through Friday. Be sure to get the money before your subjects are scheduled.

Since the lab has begun recruiting participants from other college campuses, these participants may need a parking pass. Let them know before they arrive that they need to stop at the booth by the front entrance of the campus to get a guest parking pass for $3, and bring the receipt for a reimbursement (on top of the payment that they would be receiving). If they do not bring a receipt they do not get a reimbursement for the pass!

4.5. Bilingual participants

After a presentation by Judith Kroll (plenary talk at AMLaP 2010) on differences between bilingual and monolingual speakers and how bilinguals process their own (dominant native) language differently than monolinguals do, I decided that we should take this differences more seriously.

So, from now on please following the following guidelines:

• Elicit information on whether somebody is multilingual. If so, please elicit information on which languages the participant speaks natively (e.g. prolonged exposure of several years to the language before the age of 7 or many years of experience before the age of 20; having had a language in high school or college is not sufficient to be considered a native speaker).

• Record the following information as part of the sheets that will be use for data annotation later. It is not enough to record this information in separate files as you will be unlikely to bother to include it in your analyses:
  • Languages in the following format: "English", "English/Spanish", "Spanish/Russian/Maya"

  • LanguageExpertise in the following format: e.g. "since 7/since birth" for language 1 and 2.

• And, of course, as always: Only native speakers (whether mono- or multilingual) should be run in experiments.

• Use this information in your analyses. Test whether interactions of the Languages variable with your design factors add the model fit. If so, make sure to understand how mono- and multi-linguals differ from each other.

4.6. Participant sheets (Who is run in what list?)

Keep a sheet with information about subjects that run in your experiment. Write the date, the list number, the subject's name, which computer, and any relevant notes for each subject. For future reference, keep this list in the same folder as the consent forms.

4.7. Where to store all these forms

Please follow these guidelines as to where to store consent forms, payment forms, and subject sheets.

• Each (series of) experiment(s) should have a green folder in the filing cabinet in the main room to the left of Andrew's office door. That folder needs to be accurately labeled with the name of the experiment, the abbreviation code for the experiment if there is one, and the name of the main investigator (the student(s) who designed the study).
• Within each of these green folders, there should be separate beige folders for each experiment and for (a) consent forms, (b) payment forms, and (c) whatever other documents you use to keep track of how many subjects have been run in which list of the experiment. Each folder should be labeled with the experiment name, experiment code, experimenter's name, and the name(s) of the RAs running the subjects.
• Participant surveys should be stored in the folder in that cabinet (on the right side in the top drawer). Since they are the same for all experiments, they should all be stored in the same folder.

• Please do not keep copies of unused forms in the experiment folders. That leads to outdated forms being used. Please always use forms from the "Unused Forms" folder (on the right side in the top drawer). If you need to make more copies, only use the master copies in the "Master copies" folder (on the right side in the top drawer).

If this suggested routine causes problems, please inform Florian about them, so that we can adapt the procedure.

5. Coordinating with other experiments

We want to use the time we have with each subject as much as possible, so please coordinate with other experimenters to arrange running subjects in multiple experiments at the same time, especially if each experiment is under one hour. Even if the experiment is long and it doesn't make sense to combine, try to get the subject scheduled for another experiment they haven't done, if there are any they can do. (Coordinate with the other experimenter, of course.) Keep in mind though that some experiments shouldn't be run together and that some experiments shouldn't be done if particular experiments have been done in the past. For the time being, we have a spreadsheet on Google Docs with the experiments. The MailTo(hlpstudy AT gmail DOT com) account has viewing and editing access. Check it to see what all is running. Whoever runs subjects for an experiment has to keep this document up to date at least weekly. If you need access, contact Andrew to get it. Eventually we will have a subject scheduling system that keeps track of what experiments subjects have run and what experiments block other experiments.

6. After the experiment is completed

This means after all subjects have run, not after each subject...

FILL THIS IN

6.1. At the end of a Mechanical Turk experiment

• Make sure to submit the invoice from Amazon to Kathy for reimbursement as soon as you get it. Also CC Florian on that email, as Mechanical Turk charges his personal American Express card, and he needs to know the charge occurred.

• Copy your results file(s) to /p/hlp/experiments/mturk_results/<your username>/. If a directory with your username doesn't exist, try to make it, and failing that, ask Andrew to create it for you.