Creating Stimuli For Web-base Self Paced Reading

To run an experiment we need to create stimuli (made up of critical items that have differences in line with our experimental maniuplation and fillers), order them into trials and balance within and across participants the order and type of stimuli they see. This guide will teach you some terminology, give you a short course in experimental list design and tell you how to create your stimuli for use with our web-based self-paced reading software.

In creating stimuli for your experiment, you might also find the following page useful, with links to tools that let you obtain word frequencies, predictability, neighborhood density, and so on.

Terminology

Latin square design

We will use a Latin square experimental design in your studies. We have scripts that will automatically create a Latin square from a specially formatted list of stimuli described later. But, knowing what a Latin square design is and how to make one is something you should understand. First it will be important to know the specifics of how participants saw your stimuli when you sit down to analyze your data. Second, it will let you interpret your findings with other experiments that both use this design or alternatives.

The idea behind a Latin square design (a between-subject design) is to have each participant see each item exactly once AND to see all conditions equally often AND across lists each item should be seen equally often in all its conditions. Since each participants sees exactly one list of the experiment, this means that each list should contain each item only once (in one of its condition) and all conditions equally often. One thing that follows from this is that the number of items that an experiment should have should be a multiple of the number of the experiment's conditions. For power consideration, we often have between 6- to 8-times as many items as there are conditions in the experiment and an equal number of subjects, but that's another matter. For more subtle effects, you may need even more items.

Items and lists

How should the items and conditions be distributed across lists? Let's consider a 2 x 2 design, with the four conditions a1, b1, a2, and b2. The minimum number of lists we will need is four. Let's say that we use 8-times more items than condition (a.k.a. 8 items per condition) and that List1 looks like this (prior to sorting and prior to the inclusion of fillers, more on that later):

So, List1 contains each item only once, and each condition (a1, b1, a2, b2) occurs equally often (8 times). Now we want a second list that fulfills these constraints and brings us closer to the third constraints stated above, that --across lists-- each item should be seen equally often in all its conditions. To achieve this, we construct List2 by simply shifting the condition one up. So the condition of Item1 in List2 will be the one of Item2 in List1, etc. The last item of List2 will occur in the same condition as the first item of List1

If we repeat this for List3 and List4 (see below), you will see that --across lists-- each item occurs exactly once in each of its conditions, and --within lists-- each item occurs once and all conditions occur equally often across items. That's what we want. (Latin square designs are most powerful, if each item is seen equally often in each condition across all participants. That is, we want each list to be seen by equally many participants, but that is a matter to be kept track of later when we run the experiment)

etc.

Fillers and lists

All lists have the same fillers. Lists typically have at least one filler between two items in the list. Since we usually have at least twice as many fillers as items that is easy to satisfy. What should be avoided, however, is that fillers and items are distributed according to some pattern, e.g. filler, filler, item, filler, filler, item, filler, filler, item, ... That is not good since participants may pick up on such patterns. One way to avoid this is to fully randomize lists, however, that might result in situations where participants see a long chain of items (and even worse if they are all the same condition). An alternative is to use a pseudo-random order where the items and fillers are intermixed somewhat randomly but avoiding excessively long chains of similar trials. For the purposes of this experiment we will pseudo-randomize stimuli using pre-written scripts so don't worry about this, but be aware that this means that you cannot specify your items or fillers to appear in a particular order.

Making stimuli

Here is an example of stimuli from a 2x2 design with a box around the critical words:

SPRexample_condition_75.png

In this design the experimenters are manipulating the verb ('saw', 'ate') and final noun ('apple', 'kiwi'). This is an example of one item with four conditions:

In a within-participant Latin square design (which is what you will be running), participants see this item once in one and only one of its conditions. The first manipulation is to make the verb either uninformative unconstraining (e.g. 'saw'; you can see many things and it doesn't cause you to generate any constraining expectations) or informative constraining ('ate'; there is a much more constrained set of edible things in the world, generating an expectation of an edible noun). The second manipulation is to make the final noun either high frequency (e.g. 'apple') or low frequency (e.g. 'kiwi'). In this design the experimenters are interested in reading times of the final noun (RT = reading time); you can have a different region of interest in your own study.

Some notes on critical items

The sentences here are only different as it relates to the manipulations. That is everything is held constant except for the portions that are part of the experimental design: I __ the __. Design your items to be like this too (for another example see the stimuli below). Notice that the manipulations are also very controlled. In the example above the verbs are both one syllable and equally frequent, the difference between them is minimized as much as possible to how constraining they are. The final nouns are also very similar: both have two syllables. The region of interest is held constant too, the final noun is the same distance from both the start and end of the sentence.

Some notes on filler items

Fillers are often just considered stimuli that aren't of further interest to the experimenter. It is dangerous though to underestimate the importance of fillers. There are many examples where fillers matter a lot. Fillers may determine what participants take to be the task. Consider, for example, the difference between having only grammatical or also ungrammatical fillers in some task paradigms.

Generally, fillers have two main functions:

Several consequences follow from these two central purposes of fillers:

Self-paced reading and questions

After participants read through a sentence they will have to answer a Yes/No question. Questions are primarily to make sure participants are actually reading your sentences (you can also use questions to see how participants are interpreting your sentences but that requires very careful designing so it is not recommended). Just like fillers you have to put some care into how questions are designed. Here are some important things to do:

  1. To avoid answer bias you should have an equal number of 'yes' and 'no' responses, doing so also makes it more likely participants will pay attention.
    • Since fillers will be the same for all participants have half of the filler questions have a 'yes' and half have a 'no' response.
    • For each item keep the question and response the same, this will make it easy to have an equal number of 'yes' and 'no' responses across items. If it's not possible to keep the question exactly the same try to keep the questions as similar as possible while still keeping the response the same.
  2. Make sure your questions ask about different parts of the sentences. Otherwise participants might ignore all of your sentences except for the part necessary to answer your questions.

What your stimuli file should look like

Creating lists for latin square design with a good mix of fillers between items can be done automatically. However, the format given to those scripts needs to follow a very specific format. Here is an example from a real experiment run in Professor Jaeger's lab on what your sheet should look like: ExampleLists.xlsx
It's in the current Excel format (XLSX), although we can also handle the older (XLS) format, and the comma separated value (CSV) format. Excerpted below are the first few rows of from the example file:

Experiment

ItemName

Condition

Sentence

Question

Answer

control

1

AMV

The excited fans moved through the crowd and sauntered to front stage.

Was the place empty?

N

control

1

ARC

The excited fans moved through the crowd broke apart from Jane accidentally.

Was the place empty?

N

control

1

UMV

The excited fans flew through the crowd and sauntered to front stage.

Was the place empty?

N

control

1

URC

The excited fans that were moved through the crowd broke apart from Jane accidentally.

Was the place empty?

N

control

2

AMV

The aging professors phoned about the midterm and were surprised by the workload.

Was somebody concerned about a homework set?

N

control

2

ARC

The aging professors phoned about the midterm were surprised by all the questions.

Was somebody concerned about a homework set?

N

...

filler

41

-

The parcel arrived too late to be of any use.

Did the package arrive on time?

N

filler

42

-

The espresso machine was broken for months before finally being fixed.

Was the espresso machine fixed?

Y

...

Entering Items

What the web experiment applet will do

BCS152SPRExperiment (last edited 2014-11-25 16:36:05 by slate)

MoinMoin Appliance - Powered by TurnKey Linux