Creating Stimuli For Web-base Self Paced Reading

Terminology

• Participant: a participant in our experiments (sometimes called a subject).

• Session: a session for which a participant comes in. This may contain one experiment, or it may combine several experiments. For our purposes we will only have one session per participant.

• Experiment: a set of lists of stimuli, of which each participant sees one. Each experiment may contain several embedded experiments. That is, we often run stimuli from several experiments mixed together, so that the item stimuli from one experiment can serve as fillers for the other and vice versa.

• Lists: define in which order a participant will see the stimuli. An experiment typically has several lists, but each participant only sees one list. This is used to balance conditions across lists (and hence participants).

• Design: Experiments are determined by their design. We typically fully cross manipulations, leading to factorial designs, e.g. a 2 x 2 (manipulating two binary factors) leading to 4 conditions.

• Stimuli: anything that constitutes one presentation (one trial), e.g. a picture to describe, a sentence to read, a word to remember, etc.

• Item: an item usually is a set of stimuli that only differ along the dimensions (manipulations) that define the conditions of the experiment. So, in a 2 x 2 design, each item has 4 conditions (= four different versions). There are between-item designs where this logic breaks down, but let's ignore this for now.

• Filler: a stimulus that is not of primary interest for the question we want to address with the experiment. See the Section on Fillers below.

Latin square design

The idea behind a Latin square design (a between-subject design) is to have each participant see each item exactly once AND to see all conditions equally often AND across lists each item should be seen equally often in all its conditions. Since each participants sees exactly one list of the experiment, this means that each list should contain each item only once (in one of its condition) and all conditions equally often. One thing that follows from this is that the number of items that an experiment should have should be a multiple of the number of the experiment's conditions. For power consideration, we often have between 3- to 6-times as many items as there are conditions in the experiment (and an equal number of subjects, but that's another matter). For more subtle effects, you may need even more items.

Items and lists

How should the items and conditions be distributed across lists? Let's consider a 2 x 2 design, with the four conditions a1, b1, a2, and b2. The minimum number of lists we will need is four. Let's say that we use 8-times more items than condition (a.k.a. 8 items per condition) and that List1 looks like this (prior to sorting and prior to the inclusion of fillers, more on that later):

• List1:
  • Item1 in condition a1
  • Item2 in condition b1
  • Item3 in condition a2
  • Item4 in condition b2
  • Item5 in condition a1
  • Item6 in condition b1
  • Item7 in condition a2
  • Item8 in condition b2
  • ...
  • Item29 in condition a1
  • Item30 in condition b1
  • Item31 in condition a2
  • Item32 in condition b2

So, List1 contains each item only once, and each condition (a1, b1, a2, b2) occurs equally often (8 times). Now we want a second list that fulfills these constraints and brings us closer to the third constraints stated above, that --across lists-- each item should be seen equally often in all its conditions. To achieve this, we construct List2 by simply shifting the condition one up. So the condition of Item1 in List2 will be the one of Item2 in List1, etc. The last item of List2 will occur in the same condition as the first item of List1

• List2:
  • Item1 in condition b1
  • Item2 in condition a2
  • Item3 in condition b2
  • Item4 in condition a1
  • Item5 in condition b1
  • Item6 in condition a2
  • Item7 in condition b2
  • Item8 in condition a1
  • ...
  • Item29 in condition b1
  • Item30 in condition a2
  • Item31 in condition b2
  • Item32 in condition a1

If we repeat this for List3 and List4 (see below), you will see that --across lists-- each item occurs exactly once in each of its conditions, and --within lists-- each item occurs once and all conditions occur equally often across items. That's what we want. (Latin square designs are most powerful, if each item is seen equally often in each condition across all participants. That is, we want each list to be seen by equally many participants, but that is a matter to be kept track of later when we run the experiment)

• List3:
  • Item1 in condition a2
  • Item2 in condition b2
  • Item3 in condition a1
  • Item4 in condition b1
  • ...
  • Item29 in condition a2
  • Item30 in condition b2
  • Item31 in condition a1
  • Item32 in condition b1

etc.

Fillers

All lists have the same fillers. Lists typically have at least one filler between two items in the list. Since we usually have at least twice as many fillers as items that is easy to satisfy. What should be avoided, however, is that fillers and items are distributed according to some pattern, e.g. filler, filler, item, filler, filler, item, filler, filler, item, ... That is not good, since participants may pick up on such patterns. One way to avoid this is to fully randomize lists, however, that might result in situations where participants see a long chain of items (and even worse if they are all the same condition). An alternative is to use a pseudo-random order where the items and fillers are intermixed somewhat randomly but avoiding excessively long chains of similar trials. For the purposes of this experiment we will pseudo-randomize stimuli using pre-written scripts so don't worry about this, but be aware that this means that you cannot specify your items or fillers to appear in a particular order.

Fillers are often just considered stimuli that aren't of further interest to the experimenter. It is dangerous though to underestimate the importance of fillers. There are many examples where fillers matter a lot. Fillers may determine what participants take to be the task. Consider, for example, the difference between having only grammatical or also ungrammatical fillers in some task paradigms.

Generally, fillers have two main functions:

• distract from the items, which may otherwise make it to obvious what we are interested in
• counter-balance some aspects of items to prevent that participants (subconsciously) learn about the distribution of stimuli in the experiments and then start acting strategically based on information that reflects their understanding of the experiment rather than the question of interest.

Several consequences follow from these two central purposes of fillers:

• In order to distract from items, ...
  • fillers need to be similar to the items (e.g. in terms of complexity and structure) --though, of course, they also need to be different.
  • fillers need to form groups/clusters just like items. Imagine seeing a 10 of 40 items with a certain lexical property (e.g. a certain verb) because one of four conditions of an experiment requires that lexical property. In that case, groups of fillers of similar size (10ish) should also be created distracting from the repeated property of items that would otherwise stand out.

What your sheet should look like

Creating lists for latin square design with a good mix of fillers between items can be done automatically. However, the format given to those scripts needs to follow a very specific format. Here is an example from a real experiment run in Professor Jaeger's lab on what your sheet should look like: ExampleLists.xlsx
It's in the current Excel format (XLSX), although we can also handle the older (XLS) format, and the comma separated value (CSV) format. Excerpted below are the first four rows of the first sheet from the example:

• There are three (3) required columns: Experiment, ItemName, and Condition.

• Experiment distinguishes between items from different sub-experiments being run in the same script. We won't be doing that (strictly speaking) but we will be using filler versus whatever you decide to call your experiment (in the example it is just called control.

• ItemName is to identify an individual item within an experiment. Generally this is a number.

• Condition is to identify which condition the item is in within the experiment. For filler items this should be "-". We will be randomizing the lists and each item will appear once in each list (with a different condition across lists) as specified in the latin square design above.

• After that you can add an arbitrary number of other columns, but they need to fit the naming scheme of SentenceN, QuestionN, and AnswerN, where N is an integer starting at 1. For this assignment we will probably only have Sentence1, Question1, and Answer1, but it's possible to have multiple sentences and question/answer pairs for each item.

  • If you want to specify an order of some sentences you have to make multiple Sentence columns, treat the sentence pairs as a single item

• SentenceN is the stimulus item that will be read self paced.

• QuestionN is a comprehension question to ensure that subjects are actually paying attention and not just holding down the space bar.

• AnswerN is the correct answer for the comprehension question.

• Notes:

  • If you have a QuestionN, make sure you also have a AnswerN for any given N.

  • You essentially do have to have at least a Sentence1 in addition to the required columns.

  • Condition names should be just upper and lowercase letters and periods (although you can just stick with letters)., e.g. UNINF.HIGH, UNINF.LOW, INF.HIGH, INF.LOW
  • Filler items will be the same across all the lists

• Here is an example of stimuli from a 2x2 design with boxes around critical words

  • The two independent variables in this design are the verb and the final noun
    • The verb levels are "uninformative unconstraining" ("saw"; you can see many things and it doesn't cause you to generate any constraining expectations) and "informative constraining" ("ate"; there is a much more constrained set of edible things in the world, generating an expectation of an edible noun)
    • The two noun levels are "high frequency" ("apple") and "low frequency" ("kiwi")
  • The word we are interested in the reading time of is the noun at the end of sentence (your experiment could have a different region of interest)

  • n.b We hold everything constant that we aren't interested, e.g. I __ the __.

    • Also kept parts that vary maximally similar with regards to all properties not of interest that are still known to affect RT. i.e. verbs are both one syllable and equal frequency, nouns are two syllables.
  • n.b. target region is equally far from beginning in all conditions

Entering Items

• Create a new Excel document (or clear out the example provided above)
• Put the required and any optional headers you need in the first row of each sheet

• Enter each of your items (make sure each item has a different ItemName and the same number of conditions, one per row)

• Enter your fillers on the same sheet
  • n.b. the program that generates the lists for the applet based on your Excel sheet will ignore any column other than the ones described above, so feel free to have some extra columns if they help you when making your stimuli, but also keep in mind that if you screw up the spelling of your required columns they will be ignored too.
• Save the file and send it to us.

What the web experiment applet will do

• Your stimuli file will be converted into lists that follow a latin square design (one list per condition)
• Items and fillers will be pseudo-randomly ordered (so don't expect Item 1 to be read after Item 2)

  • n.b. All participants will see the fillers and items in the same pseudo-random order

• Each participant will only see one list

• Each Sentence will be presented via word by word moving window self paced reading

• Each Question will appear after