Guidelines for running experiments

Last updated: 7/13/15 by Andrew and Olga

Table of Contents

• Well in advance of starting an experiment

  • RSRB

  • CITI

  • ROSS

• Before designing stimuli

• Getting ready to run an experiment

  • Recruitment

  • Scheduling participants

• While the experiment is running

  • Deviations and violations

  • Consent Forms

  • updated: Ensuring that participants pay attention (01/26/11)

  • RedCap Database

  • Participant payment

  • new: Bi- and multilingual participants (10/01/10)

  • Participant sheets

  • Treating identifying information adequately

  • Where to store forms

• Coordinating with other experiments

• After the experiment is finished

  • After a Mechanical Turk experiment is finished

1. Well in advance of starting an experiment

1.1. RSRB

At least one month before running an experiment, the experimenter needs to check whether there is RSRB approval for the study they plan to conduct. In particular, is both the method and the language covered by valid RSRB approval? Will the approval still be valid throughout the entire expected duration of the study (which may take several months)? Will the experiment require consent forms in a language that we do not yet have available (e.g. we have approval to run studies on Chinese, but only on Chinese speakers on campus, all of whom speak English, so we do NOT have a consent form in Chinese yet)?

1.2. CITI

If you have not gotten CITI certified at UR (you have to retake the test if you were certified at you previous institution, although overlapping modules should transfer), please go to the Initial CITI certification page and take the test to get certified before you run any subjects and sign consent forms. It is mandatory that any faculty, grad students, post-docs, or staff who are on the protocol as investigators or co-investigators have one. Undergrads need to get a CITI certified, because you are required to be if you ever run subjects for us or handle any subject data, even though you most likely won't be on the protocol and consent forms.

CITI certifications do expire every few years. Be sure to know when yours does, and take steps to get it renewed in time that your ability to run experiments is not interrupted.

Once you receive your certification by email, please forward the email to the lab manager(s) so we have your certification number and expiration date.

1.3. ROSS

Once you have your CITI certification, you need to go to the RSRB On-line Submission System (ROSS), click the “Registration” link and follow the instructions. Your “research role” is “Sub-Investigator” if you are a grad student or postdoc. We cannot add you to the IRB protocols until you have a ROSS account.

Once you receive your account info by email, please let the lab manager(s) know (but don't forward emails with passwords!!) so we can get you added on the next amendment.

2. Before you start designing stimuli

Create a wiki-page for your experiment. It should contain detailed information about the design, the type of stimuli, the methods you plan to use, the number of items you will need, how you plan to analyze the data, etc. This page should only be readable to hlp lab members (see how to restrict access to wiki pages). If you do not know how to edit the wiki, meet with the lab manager to get instructions (it's easy). You can find an example for a good experiment summary wiki page here. This page should later also contain a link to the annotation guidelines for RAs -- even if you use a standard annotation scheme.

3. Getting ready to run an experiment

3.1. Recruitment

Recruitment for our lab can be done three ways:

1. Via email. Each semester the department provides an Excel spreadsheet to lab managers with contact information for students willing to participate in the subject pool. If you want access to the list, please ask the lab manager. You can only email the students on this list or subjects who have run in experiments in our lab who have indicated willingness to participate in more experiments. Subject lists cannot be shared between labs, with the exception of the department-wide list (which has special RSRB approval).
2. Via flyers. Any flyers must be approved by the RSRB and have their stamp on them. Flyers can only be posted in locations listed in our approved protocol. Currently, we have two approved flyers (for single-day and 5-day experiments) and permission to put flyers up on the campuses of UR, RIT, and MCC. If you can sign into the RSRB system and are on our protocol, you can download approved flyers from there, otherwise you should ask the lab manager for the approved flyer.

3. Via SONA. Experimenters can add their experiments to the BCS SONA system (currently Tanenhaus and Jaeger labs only) and have it approved by an administrator (i.e. lab manager). We cannot directly contact subjects this way, but if someone creates an account on our SONA they can search experiments and sign up for time slots.

People responding to flyers will email the lab email account hlpstudy@gmail.com. All the RAs in the lab should have access to this. Separate labels are to be made for each study, and the RA responsible for running subjects for that study should move emails that they respond to into that folder and out of the inbox. Responding to emails is primarily on a first come, first serve basis (studies that are more time sensitive may be prioritized for subjects).

Any method not listed is not allowed. If you want to recruit another way, ask the lab manager for advice on the possibility of and method for getting further methods approved.

NB: For subject recruited from off-campus locations (e.g. RIT) to come to our campus lab, please be clear that they will be reimbursed the $3 parking cost on top of the subject payment.

3.2. Scheduling participants

Direct your RAs to schedule all the participants to sign up through SONA. They should not be doing the end scheduling via e-mail, only initial questions. When scheduling participants, please check the lab calendar. No experiments should be run during lab meetings or other regular meetings which are all listed in the lab calendar. Doing so, distracts participants and leads to unreliable results.

It also is interruptive to the meetings being held in the lab.

4. While the experiment is running

4.1. Deviations and violations

Unfortunately, there are going to be deviations and violations of RSRB protocols every once in a while. Although we need to do all we can to avoid them, we will keep track of those that occur. It's still better to do that than not to do it. So, if you are in doubt about something, you should ask the lab manager and/or Florian. In short, please inform us about possible deviations or violations. The consequences for the lab, if you fail to report such deviations, are quite severe.

Below, you'll first learn about general considerations about consent forms, payment form (who got paid how much), and subject sheets (who you're running in what list). Then read on to see where all these forms should be stored and how should they be organized. It is very important that you follow these guidelines.

4.2. Consent forms

Anyone running experiments in the HLP/Jaeger lab paid for out of Florian's account must use a Cross-linguistic Studies on Human Communication or Linguistic Adaptation in a Variable Environment consent form.

Only consent forms with an RSRB stamp are valid. Even if a copy is translated literally from an approved consent form, it is only valid to use that form if it itself has been approved and hence has an RSRB stamp on it. Also, the first page of the consent form must be on University of Rochester letterhead.

We currently have consent forms in a variety of languages and for studies in several languages, as well as an Amazon Mechanical Turk consent form (English only) online at http://www.hlp.rochester.edu/consent/. Before you run an experiment, make sure our RSRB protocol covers that language and that you are using the consent form in the right language.

If you are on the protocol as a co-investigator, you can go to the RSRB website, sign in and go to the Documents tab on the protocol to download the current approved consent form. These are not on letterhead, so you'll have to print them on letterhead if you download them yourself. The lab manager will also keep master copies of the current consent forms in the RSRB folder on Dropbox (ask if you need access). The lab manager will also email the lab if the consent forms change so you always have the most up to date.

Consent forms expire once a year after the Continuing Review, at which point we get new versions with the next year's stamp. Make sure that you have the current version of the consent form. The expiration date is on the RSRB stamp on the form. The Amazon Mechanical Turk consent forms are stored on the webserver with the expiration date in the filename, i.e ProtocolName-YYYY-MM-DD.pdf

Each time you run a subject, you are required to get a consent form for them, even if they have run in experiments in the past, or even another day of the same experiment. (i.e. If you have a 5 day experiment, you need 5 consent forms for each subject.) If the consent form changes while you are in the middle of a study, you need to start using the new consent form immediately, not when you finish the current study. The RSRB also now requires that we give each subject a copy of the consent form. Previously they only required that we offer the subject a copy. You can give them an unsigned consent form; you don't have to photocopy the signed one. Most subjects don't actually want a copy of the consent form, so please stress that we are required to give it to them, but inform them that if they don't want it they can leave it on the table on their way out.

Do not leave forms with personally identifying information (e.g. consent forms and subject running lists) out in the lab! When you leave the lab, make sure all paperwork is in its folder in a filing drawer.

4.3. RedCap

We have started using the RedCap Database system (https://redcap.urmc.rochester.edu/redcap/). In order to access it you need to get a log in. If you follow this page and go to the If you are an external user please click here to register for access. Fill out the form and you will get your log-in information over the course of the week.

RedCap currently has the Cross-Linguistic and Adaptation protocols. We are using these two studies and this database to keep track of the people who have come in to do a particular study and their demographic and language information. This information is helpful in knowing the people who have done past studies and makes it easier on us to do our continuing reviews. Please send the survey before the subject comes in, if the subject hasn't done the survey by the time they come into lab, have them fill it out online in lab.

Coming soon: Instructions to send out the demographic survey before the subject comes into the lab.

4.4. Ensuring that participants pay attention

IMPORTANT: Participants do not necessarily understand that they need to be completely focused on the task they are doing for your experiment. Please follow the steps outlined below to increase our chances to get meaningful results. These techniques have been employed by other labs successfully for many years:

1) Inform participants that they are participating in a science experiment. Their attention is appreciated and crucial to the success of the experiment.

2) ALWAYS explicitly remind subjects that they are not allowed to use their cell phones (for any reason) or open any other applications on the computer while they are doing the experiment. The best strategy is to have them leave all their personal belongings in the lab, not in the experimental booth.

3) Whenever possible, sit in the same room as the participant. This may not be possible in the booths. If so, inform participants that you will be in a neighboring room, if they need any help. Generally, we want participants to be aware that we are monitoring them. That's part of being in an experiment. So, it's perfectly fine, if not preferred, to look through the windows into the booth, just to make sure that participants are actually focusing on the task.

4.5. Paying Participants (Who has been paid what?)

Every time you pay a participant, you also need to have them sign a payment sheet. That is our receipt and it allows us to get reimbursed. It also is necessary in case we get audited (about every 2-3 years).

Each subject must sign a payment form and provide their home address, which cannot be a P.O. Box or campus box. If the subject is a student and willing, it would be helpful for accounting purposes to get their student ID number. In the past, we got SSNs, but that is no longer allowed. Payment forms are available from Andrew or Dana, as well as being available on this page: Subject_Payment_Form.xls. Payment forms must be returned to Meliora 360, so the department can be reimbursed and we can keep getting money to pay subjects.

Money for paying subjects comes from Chris Dambra or Jennifer Gillis in the BCS office in Meliora 360. You can only get money or return payment forms from 10am-12pm and 2pm-4pm Monday through Friday. Be sure to get the money before your subjects are scheduled.

Since the lab has begun recruiting participants from other college campuses, these participants may need a parking pass. Let them know before they arrive that they need to stop at the booth by the front entrance of the campus to get a guest parking pass for $3, and bring the receipt for a reimbursement (on top of the payment that they would be receiving). If they do not bring a receipt they do not get a reimbursement for the pass!

4.6. Bi- and multilingual participants / Language background

Judith Kroll (plenary talk at AMLaP 2010) has presented convincing evidence that bi/multilinguals process their own (dominant native) language differently than monolinguals do. We should take these differences seriously. There are several steps we can take:

For all experiments:

• Prior to experiment, record information on language background. This is done via Redcap.
• ADD SCREEN SHOT
• Import this information into your primary data sheet.

• Consider it in your analyses. For example, test whether interactions of the Languages variable with your design factors add the model fit. If so, make sure to understand how mono- and multi-linguals differ from each other.

For experiments that do not have language background as a variable of interest (most of our experiments):

• Administer only monolinguals or very weakly multilingual participants. That is, do not administer participants who any of the following for a language other than their native language:

  • prolonged exposure of several years to the language before the age of 7 (e.g., because they grew up in another country; their family, frequently present relatives or friends regularly spoke another language even if not to them; radio/TV or other media were too large parts presented in another language).

  • a total of 6 months of experience speaking a language other than their native language on a daily basis before the age of 20

  • a total of 3 years of listening to/reading a language other than their native language on a daily basis before the age of 20

  • a total of 12 months of experience speaking a language other than their native language on a daily basis before their current age.

• a total of 1+ years of classroom instruction in a language other than their native language.

4.7. Participant sheets (Who is run in what list?)

Keep a sheet with information about subjects that run in your experiment. Write the date, the list number, the subject's name, which computer, and any relevant notes for each subject. For future reference, keep this list in the same folder as the consent forms.

4.8. Where to store all these forms

Please follow these guidelines as to where to store consent forms, payment forms, and subject sheets.

• Each (series of) experiment(s) should have a green folder in the filing cabinet in the main room to the left of Andrew's office door. That folder needs to be accurately labeled with the name of the experiment, the abbreviation code for the experiment if there is one, and the name of the main investigator (the student(s) who designed the study).
• Within each of these green folders, there should be separate beige folders for each experiment and for (a) consent forms, (b) payment forms, and (c) whatever other documents you use to keep track of how many subjects have been run in which list of the experiment. Each folder should be labeled with the experiment name, experiment code, experimenter's name, and the name(s) of the RAs running the subjects.
• Participant surveys should be stored in the folder in that cabinet (on the right side in the top drawer). Since they are the same for all experiments, they should all be stored in the same folder.

• Please do not keep copies of unused forms in the experiment folders. That leads to outdated forms being used. Please always use forms from the "Unused Forms" folder (on the right side in the top drawer). If you need to make more copies, only use the master copies in the the RSRB Dropbox folder. (Ask if you need it to be shared with you.)

If this suggested routine causes problems, please inform Florian about them, so that we can adapt the procedure.

4.9. Treating identifying information adequately

Coming soon!

5. Coordinating with other experiments

We want to use the time we have with each subject as much as possible, so please coordinate with other experimenters to arrange running subjects in multiple experiments at the same time, especially if each experiment is under one hour. Even if the experiment is long and it doesn't make sense to combine, try to get the subject scheduled for another experiment they haven't done, if there are any they can do. (Coordinate with the other experimenter, of course.) Keep in mind though that some experiments shouldn't be run together and that some experiments shouldn't be done if particular experiments have been done in the past. For the time being, we have a spreadsheet on Google Docs with the experiments. The MailTo(hlpstudy AT gmail DOT com) account has viewing and editing access. Check it to see what all is running. Whoever runs subjects for an experiment has to keep this document up to date at least weekly. If you need access, contact Andrew to get it. Eventually we will have a subject scheduling system that keeps track of what experiments subjects have run and what experiments block other experiments.

6. After the experiment is completed

This means after all subjects have run, not after each subject...

FILL THIS IN

6.1. At the end of a Mechanical Turk experiment

• Make sure to submit the invoice from Amazon to Kathy for reimbursement as soon as you get it. Also CC Florian on that email, as Mechanical Turk charges his personal American Express card, and he needs to know the charge occurred.

• Copy your results file(s) to /p/hlp/experiments/mturk_results/<your username>/. If a directory with your username doesn't exist, try to make it, and failing that, ask Andrew to create it for you.