• BCS152SPRExperiment

Creating Stimuli For Web-base Self Paced Reading

Terminology

• Participant: a participant in our experiments

• Session: a session for which a participant comes in. This may contain one experiment, or it may combine several experiments

• Experiment: a set of lists of stimuli, of which each participant sees one. Each experiment may contain several embedded experiments. That is, we often run stimuli from several experiments mixed together, so that the item stimuli from one experiment can serve as fillers for the other and vice versa.

• Lists: define in which order a participant will see the stimuli. An experiment typically has several lists, but each participant only sees one list. This is used to balance conditions across lists (and hence participants).

• Design: Experiments are determined by their design. We typically fully cross manipulations, leading to factorial designs, e.g. a 2 x 2 (manipulating two binary factors) leading to 4 conditions.

• Stimuli: anything that constitutes one presentation (one trial), e.g. a picture to describe, a sentence to read, a word to remember, etc.

• Item: an item usually is a set of stimuli that only differ along the dimensions (manipulations) that define the conditions of the experiment. So, in a 2 x 2 design, each item has 4 conditions (= four different versions). There are between-item designs where this logic breaks down, but let's ignore this for now.

• Filler: a stimulus that is not of primary interest for the question we want to address with the experiment. See the Section on Fillers below.

Design Types

Latin square design

The idea behind Latin square design (a between-subject design) is to have each participant see each item exactly once AND to see all conditions equally often AND across lists each item should be seen equally often in all its conditions. Since each participants sees exactly one list of the experiment, this means that each list should contain each item only once (in one of its condition) and all conditions equally often. One thing that follows from this is that the number of items that an experiment should have should be a multiple of the number of the experiment's conditions. For power consideration, we often have between 3- to 6-times as many items as there are conditions in the experiment (and an equal number of subjects, but that's another matter). For more subtle effects, you may need even more items.

How should the items and conditions be distributed across lists? Let's consider a 2 x 2 design, with the four conditions a1, b1, a2, and b2. The minimum number of lists we will need is four. Let's say that we use 8-times more items than condition (a.k.a. 8 items per condition) and that List1 looks like this (prior to sorting and prior to the inclusion of fillers):

• List1:
  • Item1 in condition a1
  • Item2 in condition b1
  • Item3 in condition a2
  • Item4 in condition b2
  • Item5 in condition a1
  • Item6 in condition b1
  • Item7 in condition a2
  • Item8 in condition b2
  • ...
  • Item29 in condition a1
  • Item30 in condition b1
  • Item31 in condition a2
  • Item32 in condition b2

So, List1 contains each item only once, and each conditions (a1, b1, a2, b2) occurs equally often (8 times). Now we want a second list that fulfills these constraints and brings us closer to the third constraints stated above, that --across lists-- each item should be seen equally often in all its conditions. To achieve this, we construct List2 by simply shifting the condition one up. So the condition of Item1 in List2 will be the one of Item2 in List1, etc. The last item of List2 will occur in the same condition as the first item of List1

• List2:
  • Item1 in condition b1
  • Item2 in condition a2
  • Item3 in condition b2
  • Item4 in condition a1
  • Item5 in condition b1
  • Item6 in condition a2
  • Item7 in condition b2
  • Item8 in condition a1
  • ...
  • Item29 in condition b1
  • Item30 in condition a2
  • Item31 in condition b2
  • Item32 in condition a1

If we repeat this for List3 and List4 (see below), you will see that --across lists-- each item occurs exactly once in each of its conditions, and --within lists-- each item occurs once and all conditions occur equally often across items. That's what we want. (Latin square designs are most powerful, if each item is seen equally often in each condition across all participants. That is, we want each list to be seen by equally many participants, but that is a matter to be kept track of later when we run the experiment)

• List3:
  • Item1 in condition a2
  • Item2 in condition b2
  • Item3 in condition a1
  • Item4 in condition b1
  • ...
  • Item29 in condition a2
  • Item30 in condition b2
  • Item31 in condition a1
  • Item32 in condition b1

etc.

NB: A convenient way to achieve this balancing in Excel is to follow the following steps:

1. Have all your items in one sheet, row by row, and, for all item, conditions always are ordered in the same way. For example, if you have 32 items in 4 conditions (A, B, C, D), e.g. from a 2 x 2 design, then the rows would look like this:
   • Item 1 in condition A
   • Item 1 in condition B
   • Item 1 in condition C
   • Item 1 in condition D
   • Item 2 in condition A
   • ...
   • Item 2 in condition D
   • ...
   • Item 32 in condition A
   • ...
   • Item 32 in condition D.

2. Insert a header row naming all the variables in your file. In the lab, we follow a naming scheme like VariableName, which is easy to read and avoid spaces, special symbols, etc. that would lead to problems later when we import results into the script running the experiment or the analysis software (even later).

3. Create a column called Item. Enter the item numbers. E.g. for the example above, it would be 1,1,1,1,2,2,2,2,3,...,32,32,32,32 in that column.

4. Create a column called List.

5. Copy the following formulate into the List field in the first data row (second row in the sheet overall): =MOD(ROW()-1+k+ReferenceToItemCellInSameRow,k)+1, where ReferenceToItemCellInSameRow refers to the cell with the item number in the same row and k is the number of conditions of your experiment. So, for example, if List in column A and Item is in column B, you would enter the following formula into the List cell in row 2: =MOD(ROW()-1+4+B2,4)+1, which will assign this stimulus to list 1.

6. Now copy the cell to the remaining cells in the List column. This should assign lists in the desired way. For example, for the example above with 4 conditions and 32 items, you should see the following numbers in the List column:

   • 1
   • 2
   • 3
   • 4
   • 2
   • 3
   • 4
   • 1
   • 3
   • 4
   • 1
   • 2
   • 4
   • 1
   • 2
   • 3
   • 1
   • 2
   • 3
   • 4
   • ...

7. Check!

Now we almost have the four lists, but we still need to fill in fillers and we need to determine the order of items. Usually, no two items should follow each other immediately.

Making the final lists: Adding fillers and determining the order of stimuli

All lists have the same fillers. Recall that fillers have no conditions. So, yes, all lists actually have the very same filler stimuli. It's a good idea to have at least one filler between two items in the list. Since we usually have at least twice as many fillers as items that is an easy thing to construct. What should be avoided, however, is that fillers and items are distributed according to some pattern, e.g. filler, filler, item, filler, filler, item, filler, filler, item, ... That is not good, since subjects may pick up on such patterns.

So, here's a good way to create lists where each item and each filler occurs in the same position across lists:

1. Create an excel sheet that contains all stimuli for List1 and has an extra column for a random number (use the random number creation function of excel).

2. Sort the fillers and items a couple of times according to the random number column, until it seems that there is a nice pseudo-random order to things, e.g. filler, item, filler, filler, item, filler, item, filler, filler, filler, item, filler, filler, item, etc. That's good. If not mentioned otherwise, there are a couple of important constraints that should be considered:

   • Avoid two adjacent items. That is, always have at least one filler intervene between two items
   • Avoid patterns. For example, if the experiment has four conditions avoid orders where the items would be going through conditions a,b,c,d,a,b,c,d,a,b,c,d,.... Instead, we want orders that order conditions in an unpredictable way, e.g. a,b,c,d,b,a,d,c,a,d,c,b,...
   • As in the example pattern in the previous line, we prefer orders where the closest items is not in the same condition as the current item.
3. Create a new column called Trial and enter numbers from 1 ... n in ascending order. This now encodes in which order the stimuli of List1 will be presented. (If you use formula to create these numbers: be careful when you resort of copy the numbers!)
4. Sort the list back into the order it had in step 1 above.
5. Create an excel sheet for each of the remaining lists, as you have done for List1 in step 1 above.
6. Copy the Trial column from the List1 sheet into all the other List sheets.

Fillers

Fillers are often just considered stimuli that aren't of further interest to the experimenter. It is dangerous though to underestimate the importance of fillers. There are many examples where fillers matter a lot. Fillers may determine what participants take to be the task. Consider, for example, the difference between having only grammatical or also ungrammatical fillers in some task paradigms.

Generally, fillers have two main functions:

• distract from the items, which may otherwise make it to obvious what we are interested in
• counter-balance some aspects of items to prevent that participants (subconsciously) learn about the distribution of stimuli in the experiments and then start acting strategically based on information that reflects their understanding of the experiment rather than the question of interest.

Several consequences follow from these two central purposes of fillers:

• In order to distract from item, ...
  • fillers need to be similar to the items ?(e.g. in terms of complexity and structure) --though, of course, they also need to be different.
  • fillers need to form groups/clusters just like items. Imagine seeing a fourth of 40 items with a certain lexical property (e.g. a certain verb) because one of four conditions of an experiment requires that lexical property. In that case, groups of fillers of similar size (10ish) should also be created distracting from the repeated property of items that would otherwise stand out.

What the final spreadsheet is like

Now let's start by downloading some example stimuli from a real experiment that was run in Professor Jaeger's lab: ExampleLists.xlsx
It's in the current Excel format (XLSX), although we can also handle the older (XLS) format. Excerpted below are the first four rows of the first sheet from the example:

• There are three (3) required columns: Experiment, ItemName, and Condition

  • Experiment distinguishes between items from different sub-experiments being run in the same script. We won't be doing that (strictly speaking) but we will be using filler versus whatever you decide to call your experiment.

  • ItemName is to identify an individual item within an experiment. Generally this will just be a number.

  • Condition is to identify which condition the item is in within the experiment. For filler items this should be "-". In theory you can have multiple Conditions of the same ItemName and have the applet randomize the stimuli presentation and show each subject only one Condition of each ItemName, but we will be randomizing the lists ourselves and only doing one per list.

• After that you can add an arbitrary number of other columns, but they need to fit the naming scheme of SentenceN, QuestionN, and AnswerN, where N is an integer starting at 1. For this assignment we will probably only have Sentence1, Question1, and Answer1, but it's possible to have multiple sentences and question/answer pairs for each item.

  • SentenceN is the stimulus item that will be read self paced.

  • QuestionN is a comprehension question to ensure that subjects are actually paying attention and not just holding down the space bar.

  • AnswerN is the correct answer for the comprehension question.

• Notes:

  • If you have a QuestionN, make sure you also have a AnswerN for any given N.

  • You essentially do have to have at least a Sentence1 in addition to the required columns.

  • Condition names should be just upper and lowercase letters and periods (although you can just stick with letters)., e.g. UNINF.HIGH, UNINF.LOW, INF.HIGH, INF.LOW
  • Filler items should be the same across all sheets (albeit not necessarily in the same order).

• Here is an example of stimuli from a 2x2 design with boxes around critical words

  • The two independent variables in this design are the verb and the final noun
    • The verb levels are "uninformative unconstraining" ("saw"; you can see many things and it doesn't cause you to generate any constraining expectations) and "informative constraining" ("ate"; there is a much more constrained set of edible things in the world, generating an expectation of an edible noun)
    • The two noun levels are "high frequency" ("apple") and "low frequency" ("kiwi")
  • The word we are interested in the reading time of is the noun at the end of sentence

  • n.b We hold everything constant that we aren't interested, e.g. I __ the __.

    • Also kept parts that vary maximally similar with regards to all properties not of interest that are still known to affect RT. i.e. verbs are both one syllable and equal frequency, nouns are two syllables.
  • n.b. target region is equally far from beginning in all conditions

Entering Items

• Create a new Excel document
• Name each sheet what you want the list to be called
• Put the required and any optional headers you need in the first row of each sheet
• Enter each of your items
• Ordering
  • The order that your items are in each sheet are the order they will be presented
  • One trick you can use to get the items randomized is to enter them sequentially and then add another column and fill it with "=rand()" (sans quotes), and then use "Custom Sort..." in Excel to sort on that column to randomize. You'll probably want to go through after randomizing to make sure you don't have too many experimental items together, etc.
    • n.b. the program that generates the lists for the applet based on your Excel sheet will ignore any column other than the ones described above, so feel free to have some extra columns if they help you, but also keep in mind that if you screw up the spelling of your required columns they will be ignored too.
• Make extra sheets for each set of conditions for items (four sheets for a 2x2 design) and for each ordering of each set (e.g., each list forward and backward or two randomizations of each list, both yielding a total of eight sheets).

What the applet will do

• Word by word moving window self paced reading

• Comprehension questions

• Randomization
  • In theory the applet can create and randomize lists
  • In practice, it's too brittle to use currently, so we will be randomizing by hand
• Each subject will see each item in only one order (assuming your lists are done correctly; they will see only one list)