Quasi-Experimental and Interrupted Time-Series Designs Overview

Explore the various quasi-experimental designs, control groups, pretests, and outcome patterns in research methodologies. Understand the implications of different outcome patterns on causal interpretation and validity threats in experimental studies.

• Experimental designs
• Quasi-experimental
• Control groups
• Outcome patterns
• Causal interpretation

heer521 Follow

Uploaded on Sep 22, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013

2. Agenda Quasi-experimental designs that use both control groups and pretests Interrupted time-series designs Design and power problems

3. Designs that Use Both Control Groups and Pretests

4. Untreated Control Group Design with Dependent Pretest and Posttest Samples NR O1 X O2 NR O1 O2 A selection bias is always present, but the pretest observation allows for determining the magnitude and direction of bias

5. Outcome Pattern 1 Both groups grow apart at different average rates in the same direction Treatment Control This pattern is consistent with treatment effects and can sometimes be causally interpreted, but it is subject to numerous threats, especially selection-maturation

6. Outcome Pattern 2 Spontaneous growth only occurs in the treatment group Treatment Control Not a lot of reliance can be placed on this pattern as the reasons why spontaneous growth only occurred in the treatment group must be explained (e.g., selection-maturation)

7. Outcome Pattern 3 Initial pretest differences favoring the treatment group diminish over time Treatment Control Same internal validity threats as outcome patterns #1 and #2 except that selection- maturation threats are less plausible

8. Outcome Pattern 4 Control Initial pretest differences favoring the control group diminish over time Treatment Subject to numerous validity threats (e.g., selection-instrumentation, selection-history), but generally can be causally interpreted

9. Outcome Pattern 5 Outcomes that crossover in the direction of relationships Treatment Control Most amenable to causal interpretation and most threats cannot plausibly explain this pattern

10. Modeling Selection Bias Simple matching and stratifying Overt biases with respect to measured variables/characteristics Instrumental variable analysis Statistical modeling of covariates believed to explain selection biases Hidden bias analysis Difference with respect to unmeasured variables/characteristics Sensitivity analysis (how much hidden bias would need to be present to explain observed differences) Propensity score analysis Predicted probabilities of group membership Propensities then used for matching or as covariate

11. Effect-Decay Functions Immediate Effect, Large Large Delayed Effect No Decay Response Response Small Small Program Onset Program Termination Program Onset Program Termination Time Time Immediate Effect, Early Effect, Slow Large Large Rapid Decay Decay Response Response Small Small Program Onset Program Termination Program Onset Program Termination Time Time

12. Untreated Control Group Design with Dependent Pretest and Posttest Samples Using a Double Pretest NR O1 O2 X O3 NR O1 O2 O3 Permits assessment of selection-maturation on the assumption that the rates between O1 and O2 will continue between O2 and O3 Testable only on the control group

13. Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Switching Replications NR O1 X O2 O3 NR O1 O2 X O3 A strong design and only a pattern of historical changes that mimics the time sequence of the treatment introductions can serve as an alternate explanation The addition of treatment removal (X) can strengthen cause-effect claims

14. Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Reversed Treatment Control Group NR O1 X+ O2 NR O1 X- O2 Interpretation of this design depends on producing two effects with opposite signs Adding a control is useful Ethically, often difficult to use a reversed treatment

15. Interrupted Time-Series Designs

16. Interuppted Time-Series A large series of observations made on the same variable consecutively over time Observations can be made on the same units (e.g., people) or on constantly changing units (e.g., populations) Must know the exact point at which a treatment or intervention occurred (i.e., the interruption) Interrupted time-series designs are powerful cause-probing designs when experimental designs cannot be used and when a time series is feasible

17. Types of Effects Form of the effect (slope or intercept) Permanence of the effect (continuous or discontinuous) Immediacy of the effect (immediate or delayed)

18. Analytic Considerations Independence of observations (Most) statistical analyses assume observations are independent (one observation is independent of another) In interrupted time-series, observations are autocorrelated (related to prior observations or lags) Requires a large number of observations to estimate autocorrelation Seasonality Observations that coincide with seasonal patterns Seasonality effects must be modeled and removed from a time-series before assessing treatment impact

19. Simple Interrupted Time- Series Design O1 O2 O3 O4 O5 X O6 O7 O8 O9 O10 The basic interrupted time-series design requires one treatment group with many observations before and after a treatment

20. Change in Intercept 20 Intervention 18 16 14 12 10 8 6 Change in intercept 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

21. Change in Slope 35 Intervention 30 25 20 15 10 Change in slope 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

22. Weak and Delayed Effects 35 Intervention 30 25 20 15 10 5 Impact begins 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

23. Validity Threats With most interrupted time-series designs, the major validity threat is history Events that occur at the same time as the treatment was introduced Instrumentation is also often a threat Over long time periods, methods of data collection may change, how variables are defined and/or measured may change Selection is sometimes a threat If group membership changes abruptly

24. Additional Designs O1 O1 O2 O2 O3 O3 O4 O4 O5 O5 X O6 O6 O7 O7 O8 O8 O9 O9 O10 O10 OA1 OB1 OA2 OB2 OA3 OB3 OA4 OB4 OA5 OB5 X X OA6 OB6 OA7 OB7 OA8 OB8 OA9OA10 OB9OB10 O1 O2 O3 O4 X O5 O6 O7 O8 X O9O10O11O12 (1) nonequivalent control group, (2) nonequivalent dependent variable, and (3) removed treatment

25. Nonequivalent Control Group 80 Intervention 70 Treatment group 60 50 40 Control group 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13

26. Nonequivalent Dependent Variable Intervention 35 30 25 Nonequivalent dependent variable 20 Dependent variable 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13

27. Removed Treatment Introduction Removal 30 25 Treatment period 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13

28. Design and Power Problems

29. Problem #1 A school administrator wants to know whether students in his district are scoring better or worse than the national norm of 500 on the SAT He decides that a difference of 20-25 points or more from this normative value would be important to detect He anticipates that the standard deviation of scores in his district is about 80 points Determine the number of students necessary for power at 95% to detect a difference of 20 and 25 points Graph both Diagram the design of the study

30. Problem #2 Patients suffering from allergies are nonrandomly assigned to a treatment and placebo condition and asked to rate their comfort level on a scale of 0 to 100 The expected standard deviation is 20 and a difference of 10-20 is expected (treatment = 50-60 and placebo = 40) Determine the number of patients necessary for power at 95% to detect a difference of 10 and 20 points Graph both Diagram the design of the study

31. Problem #3 The cure rate for two current treatments are 10% and 60%, respectively The alternative treatments are expected to increase the cure rate by 10% Determine the number of patients necessary for power at 95% to detect a difference of 10% for both scenarios Graph both Diagram the design of the studies