Reliability in Psychology Experiments

1. Define key variables.

1. Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of inappropriate content.
2. Consult definitions created by the organization TV Parental Guidelines and approved by the Federal Communications Commission.
3. Inappropriate content is any crude or rude behavior (e.g., toilet humor), depictions of verbal or physical aggression (e.g., name calling, hitting, etc.), bad language (e.g., curse words), or references to drug use, violence, or sex.

2. Create coding categories from the operational definition of inappropriate content.

1. Design a systematic process for the research participants (referred to here as the raters) to identify instances of the targeted, inappropriate behavior (see Table 1).

Coding Categories	Themes and Exemplars	Count
Crude Behavior	Toilet humor Purposefully disgusting behaviors
Rude Behavior	Disrupting others Poor Manners
Language	Using curse words
Verbal Aggression	Insults Yelling Name-Calling
Physical Aggression	Hitting Pushing/Shoving Tripping
Drug References	Verbal (suggestive statements/conversation) Nonverbal (mimicking drug use)
Sexual References	Verbal (suggestive statements/conversation) Nonverbal (mimicking sexual acts)

Table 1. Example of how to record instances of inappropriate behaviors. This log can be systematically used across raters.

3. Instruct raters to separately watch the same episode of SpongeBob SquarePants and provide coding counts.

4. Instruct raters to separately watch the same episode of Caillou and provide coding counts.

5. Compare ratings to see if the Raters came up with similar ratings for each show.

1. Reliability is the ability to consistently measure the variable—inappropriate content.
2. Inter-rater reliability is the ability for more than one person to measure the variables and for their measurements to be in accord.