Guidelines for running experiments

Table of Contents

1. Well in advance of starting an experiment

Anchor(advance)

1.1. RSRB

Anchor(rsrb) At least one month before running an experiment, the experimenter needs to check whether there is RSRB approval for the study they plan to conduct. In particular, is both the method and the language covered by valid RSRB approval? Will the approval still be valid throughout the entire expected duration of the study (which may take several months)? Will the experiment require consent forms in a language that we do not yet have available (e.g. we have approval to run studies on Chinese, but only on Chinese speakers on campus, all of whom speak English, so we do NOT have a consent form in Chinese yet)?

1.2. EPRP

Anchor(eprp) If you have not gotten an EPRP (Ethical Principals in Research Program) number, please go to the [http://www.rochester.edu/ohsp/certification/eprp.html EPRP website] and take the test to get one before you run any subjects and sign consent forms. It is mandatory that any faculty, grad students, post-docs, or staff who are on the protocol as investigators or co-investigators have one. It is a good idea even for the undergrads to get an EPRP number though, because you are required to have one if you ever run subjects for us, even though you most likely won't be on the protocol and consent forms.

EPRP numbers do expire every few years. Be sure to know when yours does, and take steps to get it renewed in time that your ability to run experiments is not interrupted.

2. Before you start designing stimuli

Anchor(stimuli) Create a wiki-page for your experiment. It should contain detailed information about the design, the type of stimuli, the methods you plan to use, the number of items you will need, how you plan to analyze the data, etc. This page should only be readable to hlp lab members (see [wiki:HlpLab/WikiEditing how to restrict access to wiki pages]). If you do not know how to edit the wiki, meet with the lab manager to get instructions (it's easy). You can find an [wiki:HlpLab/Projects/ChineseClassifiers/LabLog example for a good experiment summary wiki page here]. This page should later also contain a link to the annotation guidelines for RAs -- even if you use a standard annotation scheme.

3. While the experiment is running

Anchor(during)

3.1. Recruitment

Anchor(recruitment) Recruitment for our lab can be done two ways:

  1. Via email. Each semester the department provides an Excel spreadsheet to lab managers with contact information for students willing to participate in the subject pool. If you want access to the list, please ask the lab manager. You can only email the students on this list or subjects who have run in experiments in our lab who have indicated willingness to participate in more experiments. Subject lists cannot be shared between labs, with the exception of the department-wide list (which has special RSRB approval).
  2. Via flyers. Any flyers must be approved by the RSRB and have their stamp on them. Flyers can only be posted in locations listed in our approved protocol. Currently, we have two approved flyers (for single-day and 5-day experiments) and permission to put flyers up on the campuses of UR, RIT, and MCC. If you can sign into the RSRB system and are on our protocol, you can download approved flyers from there, otherwise you should ask the lab manager for the approved flyer.

Any method not listed is not allowed. If you want to recruit another way, ask the lab manager for advice on the possibility of and method for getting further methods approved.

3.2. Deviations and violations

Anchor(deviations) Unfortunately, there are going to be deviations and violations of RSRB protocols every once in a while. Although we need to do all we can to avoid them, we will keep track of those that occur. It's still better to do that than not to do it. So, if you are in doubt about something, you should ask the lab manager and/or Florian. In short, please inform us about possible deviations or violations. The consequences for the lab, if you fail to report such deviations, are quite severe.

Below, you'll first learn about general considerations about consent forms, payment form (who got paid how much), and subject sheets (who you're running in what list). Then read on to see where all these forms should be stored and how should they be organized. It is very important that you follow these guidelines.

Anchor(consent) Anyone running experiments in the HLP/Jaeger lab paid for out of Florian's account must use a Cross-linguistic Studies on Human Communication consent form.

Only consent forms with an RSRB stamp are valid. Even if a copy is translated literally from an approved consent form, it is only valid to use that form if it itself has been approved and hence has an RSRB stamp on it. Also, the first page of the consent form must be on University of Rochester letterhead.

We currently have consent forms in a variety of languages and for studies in several languages, as well as an Amazon Mechanical Turk consent form (English only) online at [http://www.hlp.rochester.edu/consent/English_2011-05-13.pdf]. Before you run an experiment, make sure our RSRB protocol covers that language and that you are using the consent form in the right language.

If you are on the protocol as a co-investigator, you can go to the [https://rsrb01.urmc.rochester.edu/ RSRB website], sign in and go to the Documents tab on the protocol to download the current approved consent form. These are not on letterhead, so you'll have to print them on letterhead if you download them yourself. Andrew will also keep master copies of the current consent forms in the lab, and will copy them for you if you let him know you are running a study. He will also email the lab if the consent forms change so you always have the most up to date.

Consent forms expire once a year after the Continuing Review, at which point we get new versions with the next year's stamp. Make sure that you have the current version of the consent form. The expiration date is on the RSRB stamp on the form. The Amazon Mechanical Turk consent forms are stored on the webserver with the expiration date in the filename, i.e Languagename-YYYY-MM-DD.pdf

Each time you run a subject, you are required to get a consent form for them, even if they have run in experiments in the past, or even another day of the same experiment. (i.e. If you have a 5 day experiment, you need 5 consent forms for each subject.) If the consent form changes while you are in the middle of a study, you need to start using the new consent form immediately, not when you finish the current study. The RSRB also now requires that we give each subject a copy of the consent form. Previously they only required that we offer the subject a copy. You can give them an unsigned consent form; you don't have to photocopy the signed one. Most subjects don't actually want a copy of the consent form, so please stress that we are required to give it to them, but inform them that if they don't want it they can leave it on the table on their way out.

Do not leave forms with personally identifying information (e.g. consent forms and subject running lists) out in the lab! When you leave the lab, make sure all paperwork is in its folder in a filing drawer.

3.4. Paying subjects (Who has been paid what?)

Anchor(payment) Every time you pay a participant, you also need to have them sign a payment sheet. That is our receipt and it allows us to get reimbursed. It also is necessary in case we get audited (about every 2-3 years).

Each subject must sign a payment form and provide their home address, which cannot be a P.O. Box or campus box. If the subject is a student and willing, it would be helpful for accounting purposes to get their student ID number. In the past, we got SSNs, but that is no longer allowed. Payment forms are available from Andrew or Dana, as well as being available on this page: attachment:Subject_Payment_Form.xls. Payment forms must be returned to Meliora 360, so the department can be reimbursed and we can keep getting money to pay subjects.

Money for paying subjects comes from Chris Dambra or Jennifer Gillis in the BCS office in Meliora 360. You can only get money or return payment forms from 10am-12pm and 2pm-4pm Monday through Friday. Be sure to get the money before your subjects are scheduled.

3.5. Subject sheets (Who is run in what list?)

Anchor(sheets) Keep a sheet with information about subjects that run in your experiment. Write the date, the list number, the subject's name, which computer, and any relevant notes for each subject. For future reference, keep this list in the same folder as the consent forms.

3.6. Where to store all these forms

Anchor(storage) Please follow these guidelines as to where to store consent forms, payment forms, and subject sheets.

If this suggested routine causes problems, please inform Florian about them, so that we can adapt the procedure.

4. Coordinating with other experiments

Anchor(coordinating) ANDREW, PLEASE UPDATE

We want to use the time we have with each subject as much as possible, so please coordinate with other experimenters to arrange running subjects in multiple experiments at the same time, especially if each experiment is under one hour. Even if the experiment is long and it doesn't make sense to combine, try to get the subject scheduled for another experiment they haven't done, if there are any they can do. (Coordinate with the other experimenter, of course.) Keep in mind though that some experiments shouldn't be run together and that some experiments shouldn't be done if particular experiments have been done in the past. For the time being, we have a spreadsheet on Google Docs with the experiments. Check it to see what all is running. If you need access, contact Andrew or Katrina to get it. Eventually we will have a subject scheduling system that keeps track of what experiments subjects have run and what experiments block other experiments.

5. After the experiment is completed

Anchor(after) This means after all subjects have run, not after each subject...

FILL THIS IN

5.1. At the end of a Mechanical Turk experiment

Anchor(mturk) Make sure to submit the invoice from Amazon to Kathy for reimbursement as soon as you get it. Also CC Florian on that email, as Mechanical Turk charges his personal American Express card, and he needs to know the charge occurred.

MoinMoin Appliance - Powered by TurnKey Linux