Understanding Contamination in Research Studies
Understand the concept of contamination in research studies, its impact on the effectiveness of Randomized Controlled Trials (RCTs), and different analysis methods to handle contamination, such as Intention-to-Treat (ITT) and Contamination-Adjusted Intention-to-Treat (CA-ITT) analyses. Learn how to estimate treatment effects in the presence of contamination using Stata software. Differentiate between contamination and missing data in research studies, and explore the implications of each.
- Contamination in Research
- RCTs
- Intention-to-Treat Analysis
- Contamination Adjusted Analysis
- Treatment Effects
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dealing with contamination Dr. John Jerrim UCL Institute of Education
Aims To understand what contamination is . To understand why contamination reduces the effectiveness of RCTs What is meant by intention-to-treat (ITT) analysis, and how to calculate ITT What is meant by contamination adjusted intention-to-treat (CA-ITT) and how to produce these estimates Gain experience of estimating treatment effects from RCTs where contamination is present using Stata
Set-up Example RCT with two arms and 16 participants (8 Treatment and 8 Control). Outcome = Scores on a mathematics test Homogeneous treatment effect. -Anyone who receives the intervention scores 10 points on the test -Anyone who receives the control scores 0 points on the test In following slide: RED = Assigned to treatment group BLUE = Assigned to control group
If things work perfectly.. Treatment received Control received Gender M M M M F F F F Outcome 10 10 10 10 10 10 10 10 Gender M M M M F F F F Outcome 0 0 0 0 0 0 0 0
Assume no contamination Balance check Treatment group: 4 male (50%) and 4 female (50%) Control group: 4 male (50%) and 4 female (50%) Treatment effect Mean score treatment group = 10 Mean score control group = 0 Estimated treatment effect = 10
What might contamination look like? Treatment received Control received Gender M M M M F Mean 10 10 10 10 10 Gender M M M M F F F F F F F Mean 0 0 0 0 0 0 0 0 0 0 0
Contamination vs missing data NOTE: These are different things!!! Often confused within the literature. Contamination: We have outcome data for all study participants .. ..but some may have moved from T to C (or vice-versa) Missing data: We do not have outcome data for some study participants . ..and some could have changed from T to C (or vice-versa) as well! Today = About contamination. Not missing data .
How do we handle this? Run through four options Treatment received analysis Per-protocol analysis Intention-to-treat (ITT) analysis Contamination-adjusted intention-to-treat (CA-ITT) analysis Each has assumptions .. Each has strengths and limitations
Treatment received analysis Basically, ignore initial allocation . Treatment received Control received Gender M M M M F Mean 10 10 10 10 10 Gender M M M M F F F F F F F Mean 0 0 0 0 0 0 0 0 0 0 0 These 3 girls were coloured red (assigned treatment) but now coloured BLUE
Treatment received analysis Balance check X X Treatment group: 4 male (80%) and 1 female (20%) Control group: 4 male (36%) and 7 female (64%) Treatment effect Mean score treatment group = 10 Mean score control group = 0 Estimated treatment effect = 10 (Yes but only because of how I set up the data!) Balance check violated. Randomisation mechanism broken!!! Back to the selection problem. Note: No guarantee treatment effect will be unbiased!
What does per-protocol mean Only include in analysis individuals very faithful to the treatment and the experiment Individuals who have remained within their randomly assigned group .. Individuals who have implemented the programme as intended . Example Person assigned to drug treatment takes only half the pills In a per-protocol analysis, these individuals are dropped!
Per-protocol analysis Treatment received Control received Gender M M M M F Mean 10 10 10 10 10 Gender M M M M F F F F Mean 0 0 0 0 0 0 0 0 The 3 girls that were not faithful to their initial random allocation have been dropped from the analysis!
Per-protocol analysis Balance check X Treatment group: 4 male (80%) and 1 female (20%) Control group: 4 male (50%) and 4 female (50%) Treatment effect Mean score treatment group = 10 Mean score control group = 0 Estimated treatment effect = 10 (Yes but only because of how I set up the data!) Balance check violated. Randomisation mechanism broken!!! Back to the selection problem. Note: No guarantee treatment effect will be unbiased!
Per-protocol analysis More generally (in the real world) Per-protocol analysis likely to give you an upper-bound on treatment effect I.e. This is as good as its ever going to get! Why? Excluding individuals who received only partial treatment (and therefore likely to have experienced less effect from the programme) .. Exclude people who have selected out of the programme (also likely to select out because programme less effective for them)
What does ITT mean? Remember analyse as randomised . Analyse individuals according to the group they were allocated to . This is the most common method of analysis within RCTs! People will expect to see the ITT estimates in any RCT that you run ..
ITT analysis Treatment received Control received Gender M M M M F F F F Mean 10 10 10 10 10 0 0 0 Gender M M M M F F F F Mean 0 0 0 0 0 0 0 0 The 3 girls that moved from T to C are included in the treatment group within your analysis.
Assume no contamination Balance check Assigned treatment group: 4 male (50%) and 4 female (50%) Assigned control group: 4 male (50%) and 4 female (50%) Treatment effect Mean score treatment group = 6.25 Mean score control group = 0 Estimated treatment effect = 6.25 Random assignment has been maintained (good!) But our estimate of the treatment effect has fallen .
ITT analysis More generally (in the real world) ITT analysis likely to give you a lower-bound on the effectiveness of the treatment . (as long as there is not significant heterogeneity of treatment effects) Why? We only test an intervention if we think it is going to work . Expect those who receive T to score higher than C . Contamination Some we analyse within T won t have received this benefit . Hence will underestimate effect of completing the treatment
Relationship between % contamination and ITT estimate NOTE As non-compliance then ITT estimate But no impact upon the CI
What does ITT estimate? ITT gives you an unbiased estimate (i.e. causal effect ) of being OFFERED the treatment .. .not the effectiveness of the treatment per se In other words, average effect upon those who take up the treatment is likely to be higher than ITT.
Is ITT relevant? Depends upon the programme / intervention! When rolling out, is it going to be optional or mandatory? Optional programme ITT will give average impact of the programme within population of interest ..recognising that take up will not be 100% Hence ITT will be very relevant! Mandatory programme E.g. Change to the national curriculum in England If assuming homogeneous treatment effect ..ITT will probably underestimate impact
Contamination-Adjusted Intention-To-Treat (CA-ITT)
CA-ITT ITT = Effect of being offered the treatment . CA ITT = Effect of actually receiving the treatment . If I am a school / patient / parent I probably care more about CA-ITT than ITT!! CA-ITT is an instrumental variable approach It basically adjusts the ITT estimate by the rate of non-compliance .
Rephrasing the problem Want to estimate the effect of treatment receipt: (1) ? = ? + ?.? + ? Where Y = Test score T = Treatment receipt (0 = Control; 1 = Treatment) Where T is randomly allocated (no contamination) we can safely assume: ??? (?,?) = 0
But now there is contamination.. Concern that there is some unobserved variable (C) that is associated with both treatment receipt (T) and the outcome (Y) . I.E. There is a factor (C) that encourages people to not adhere to random assignment. (2) ? = ? + ?.? + ?.? + ? Where C = Some unobserved confounder that is associated with both T and Y. As C is unobserved, it becomes part of the error term (?). We can then re-write (2) as: (3) ? = ? + ?.? + ? Because ??? (?,?) 0, we can not consistently estimate ? (the effect of treatment receipt).
The IV solution We have access to another variable in our dataset (Z). Can re-write equation (3) in terms of each variable with Z: ??? ?,? = ?.??? ?,? + ???(?,?) Now say that our IV satisfies these two assumptions: ???(?,?) = 0 (Exclusion restriction) ??? ?,? 0 (Strong instrument) Then we can consistently estimate ? (effect of treatment receipt) as: ???(?,?) ???(?,?) =
What could act as an IV in an RCT? Need a variable Z that meets two key assumptions 1. Is strongly associated with treatment receipt .. 2. Has no direct effect on the outcome (Y) Are we likely to have such a variable in an RCT? Yes!!! Random assignment of individuals! 1. Those assigned to T group should be more likely to receive T 2. As random, no reason why assignment related to Y Random allocation usually a very good IV for treatment receipt!
How do we implement CA- ITT in our analysis?
1. Use Stata and apply TSLS Use Two-Stage Least Squares (TSLS). Stage 1. Regress treatment receipt upon random assignment ? = ? + ?.? Where T = Treatment Receipt Z = Dummy variable for trial arm randomly allocated to Based upon this regression model, created predicted values ?. Stage 2. Estimate equation (1) with ? in place of T ? = ? + ?. ? + ?
2. Use common sense! Association between random allocation and the outcome = ITT estimate ???(?,?) ???(?,?) = Association between random allocation and treatment receipt = % swapping T and C = % receive T who were assigned to T MINUS % receive T who were assigned to C
CA-ITT analysis Allocation (Z) T T T T T T T T Receipt (T) T T T T T C C C Score 10 10 10 10 10 0 0 0 Allocation (Z) C C C C C C C C Receipt (T) C C C C C C C C Score 0 0 0 0 0 0 0 0 ITT estimate = 6.25 Difference in % who receive T across groups = (5 / 8) (0 / 8) = 0.625 CA ITT = = 6.25 / 0.625 = 10
CA ITT estimate Balance check Treatment group: 4 male (50%) and 4 female (50%) Control group: 4 male (50%) and 4 female (50%) Treatment effect Estimated treatment effect = 10
Key assumption.. CA ITT analysis seems great for estimating the impact of treatment among those who received the treatment .. But there is a BIG assumption behind this . Assumption Non-compliers would have received the same benefit of the intervention as the compilers , had they received the treatment. Questionable? People more likely switch from C to T if believe intervention will work for them? People more likely switch from T to C if think intervention won t work on them?
Summary of different analyses What does it estimate Impact upon anyone who received T Impact upon those who did as instructed Impact of the offer of T Impact of reciept of T Randomisation maintained No No Yes Yes As treated Per-protocol ITT CA-ITT
Conclusions Contamination can be a major problem in RCTs . Potential to over-ride the randomisation mechanism that is the major attraction of RCTs . Various type of analyses one can conduct. Different strengths and limitations . My advice. - Present all 4 different types of estimates discussed - ITT will probably take precedent as most widely used and most conservative
