What Is Interobserver Agreement?
Interobserver agreement (IOA) refers to the extent to which two or more independent observers record the same target behavior(s) during the same observation period. In applied behavior analysis (ABA), IOA is a core measure of data reliability. Without adequate IOA, decisions based on data may be flawed, undermining the validity of a behavioral intervention.
Table of Contents
- What Is Interobserver Agreement?
- How to Calculate Interobserver Agreement
- Real ABA Examples of IOA in Action
- Exam Relevance and Common Traps
- Quick Checklist for Mastering IOA
IOA is not the same as treatment integrity or procedural fidelity. While treatment integrity measures whether the intervention was implemented as planned, IOA focuses on the consistency of measurement across observers. For the BCBA exam, you need to understand both the calculation methods and the practical implications of low or high IOA.
Why IOA Matters in Behavior Analysis
High IOA values indicate that the target behavior is well-defined and the measurement system is objective. Low IOA suggests operational definitions are unclear, observers need retraining, or the behavior is difficult to capture reliably. On the exam, expect questions linking IOA to trustworthy data and sound clinical decisions.
- Ensures data accuracy: Consistent observations reduce measurement error.
- Supports experimental control: Reliable data are the foundation of functional relations.
- Facilitates replication: High IOA allows other researchers to repeat procedures.
- Protects client welfare: Inaccurate data could lead to ineffective or harmful interventions.
How to Calculate Interobserver Agreement
Several IOA calculation methods exist, each suited for different data types. The BCBA exam often asks you to compute IOA or interpret a given method. The most common formulas are point-by-point, total count, mean count, interval-by-interval, scored-interval, and unscored-interval IOA. Below, we break down each approach.
Common Formulas You Need to Know
| Method | Formula | Best Used For |
|---|---|---|
| Point-by-Point (Exact Agreement) | (Agreements / (Agreements + Disagreements)) x 100% | Event recording with multiple opportunities per trial |
| Total Count | (Smaller count / Larger count) x 100% | When total number of behaviors is the key metric |
| Mean Count | Sum of (smaller/larger) per session / Number of sessions x 100% | When data are collected across multiple sessions |
| Interval-by-Interval | (Number of intervals with agreement / Total intervals) x 100% | Time sampling or interval recording |
| Scored-Interval | (Intervals both observers recorded behavior / Intervals either observer recorded behavior) x 100% | When one observer rarely sees the behavior |
| Unscored-Interval | (Intervals both agreed behavior did not occur / Total intervals) x 100% | When behavior occurs at high rates |
Real ABA Examples of IOA in Action
Understanding IOA in context is critical for the exam. Below are two detailed examples illustrating how IOA is collected and analyzed in common clinical scenarios.
Example 1: Aggression in a Classroom Setting
Setting: A 7-year-old student with autism engages in hitting peers. Antecedent: Peer denies access to a toy. Behavior: Hitting (operationally defined as open-hand contact). Consequence: Teacher removes the student from the area. Hypothesized function: Escape from social demands.
Two observers record the frequency of hitting for 15 minutes. Observer A records 12 instances, Observer B records 9. Using total count IOA: (9/12)x100 = 75%. This is below the 80% threshold often recommended. The disagreement suggests the operational definition may be ambiguous (e.g., does light contact count?). Revising the definition to specify ‘forceful contact sufficient to make an audible sound’ could improve IOA.
Example 2: Self-Injurious Behavior in a Clinic
Setting: A 4-year-old with no functional communication engages in head banging. Antecedent: Alone time (no attention). Behavior: Head banging (defined as any contact of the head with a surface). Consequence: Caregiver provides eye contact and verbal redirection. Hypothesized function: Automatic reinforcement (sensory stimulation).
Using interval recording (10-sec intervals, 60 intervals total), both observers agree on 55 intervals (both recorded behavior present or both absent). Interval-by-interval IOA = (55/60)x100 = 91.7%. This high IOA indicates a clear, reliable definition. However, note that if the behavior is extremely rare, scored-interval IOA might be more accurate.
Example 3: Vocal Stereotypy in a Home Setting
Setting: A 6-year-old child with autism produces repetitive vocalizations (‘ee-ee-ee’). Antecedent: Low-demand situation (free play). Behavior: Any non-contextual vocalization lasting >1 second. Consequence: No social reaction from parent (planned ignoring). Hypothesized function: Automatic reinforcement.
Observers use partial-interval recording (15-sec intervals, 40 intervals). Observer A scores behavior in 30 intervals, Observer B in 28. Disagreements occur in 5 intervals. Using interval-by-interval IOA: (35/40) = 87.5%. But because behavior occurs frequently, unscored-interval IOA might be high if both agree on non-occurrence. Always choose the method that minimizes chance agreement.
Exam Relevance and Common Traps
The BCBA exam tests IOA in several ways: calculating IOA from raw data, selecting the appropriate formula, and interpreting IOA values in context. Many students stumble on the nuances. Below are two frequent traps.
Traps: Misinterpreting Acceptable IOA Thresholds
You will often see ‘80% or higher is acceptable.’ However, context matters. For high-risk behaviors (e.g., self-injury), you may aim for 90% or higher. For low-rate behaviors, even 70% might be acceptable if the definition is clear. The exam will present scenarios where the ‘best’ action is to re-train observers, not just accept borderline IOA.
Traps: Confusing IOA with Interobserver Reliability
On the exam, the term ‘interobserver reliability‘ is often used interchangeably with IOA, but technically, reliability refers to the consistency of a measurement system over time. IOA is a specific type of reliability check. Do not confuse it with test-retest reliability or split-half reliability. When you see ‘reliability’ in a question about two observers, it’s likely IOA.
Additional Trap: Using the Wrong Formula
For example, using total count when the behavior is measured via interval recording. The exam might present data on a partial-interval recording and ask you to calculate IOA. If you use total count instead of interval-by-interval, you will get a misleading value. Always match the formula to the data collection method.
Quick Checklist for Mastering IOA
Use this checklist when reviewing IOA for the BCBA exam or planning a data-collection session.
- Define behavior: Ensure operational definitions are objective and measurable.
- Train observers: Provide practice until IOA reaches at least 80% during training.
- Choose the right formula: For event recording -> total count or point-by-point; for interval -> interval-by-interval; for rare behaviors -> scored-interval; for frequent -> unscored-interval.
- Collect IOA data: Aim for 20-30% of sessions, or per BACB guidelines.
- Interpret values: Below 80% = revisit definitions and retrain; 80-90% = good; above 90% = excellent.
- Document all: Record IOA calculations in your report and note any disagreements.
For more on data reliability, see our guide on treatment integrity and social validity. Also read the official BACB Ethics Code for guidelines on data recording.
Final thoughts: IOA is not just an exam topic—it is a daily practice in ABA. By mastering these calculations and pitfalls, you will be better prepared to ensure your data are trustworthy and your interventions are effective. Good luck on the BCBA exam!
