Targeted Learning Framework for Causal Effect Estimation Using Real World Data
Hana Lee, Ph.D., presents a webinar on the Targeted Learning Framework for Causal Effect Estimation using Real World Data (TMLE). The project aims to help the FDA develop a structured approach to incorporating real-world data into regulatory decision-making. TMLE offers a systematic roadmap aligned with regulatory requirements, leveraging machine learning for more reliable inference. The presentation covers motivations, goals, and the potential of TMLE for regulatory purposes, emphasizing its theory-based and doubly-robust methodology. Resources for FDA staff and comparisons with propensity score-based methods are also discussed.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
BAA: A Targeted Learning Framework for Causal Effect Estimation Using Real World Data TMLE Webinar Series #12 August 4, 2021 Hana Lee, Ph.D. Hana.Lee@fda.hhs.gov
Disclaimer This presentation reflects the view of the author and should not be construed to represent FDA s views or policies. 2
Outline Motivation & overarching aim of the project Targeted Maximum Likelihood Estimation (TMLE): Potential for Regulatory Purposes Specific aims Task 1, 2, 3, 4: Findings and take away messages Resources for FDA staffs 3
Goal of BAA TMLE To help FDA develop a principled approach to incorporating RWD into regulatory decision making To apply a targeted learning (TL) perspective to define the causal problem and target estimand, and identify potential threats to interpretability as a causal estimate To compare the performance of TMLE with two popular propensity score (PS) based methods; PS matching and PS weighting (aka inverse probability weighting ) 6
TMLE: Potential for Regulatory Purposes 1. TMLE follows a systematic roadmap (TL framework) which aligns well with ICH E9 R1 estimand framework to estimate the target quantity 2. Satisfies regulatory interest in machine learning (ML): TMLE can utilize ML or an ensemble of ML algorithms called Super Learning for modeling outcome and propensity score 3. Theory-based, doubly-robust method that can utilize ML Valid uncertainty (variance) estimation is possible Inference may be more reliable Good finite sample performance as well Unbiased when model for either propensity score or mean outcome is correct. Efficient when both are correct 4. R Packages are available: tmle, ltmle (for longitudinal data analyses) 8
TL Roadmap: Overview Mark van der Laan. Targeted Machine Learning for Causal Inference based on Real World Data. https://youtu.be/PrPNP5RVcLg 9
Specific Tasks, Findings, and Key Takeaway Messages
Specific Tasks Task 1: Analysis of high quality randomized clinical trial (RCT) data and simulated data with synthetic selection bias and/or informative missingness in outcome Task 2: Conduct analysis of RCT data with substantial loss to follow up (LTFU) Task 3: Highlight the importance of following the targeted learning (TL) roadmap to assess the validity and interpretability of study findings, using an observational study as a running example Task 4: Education & dissemination of concepts in TL, causal inference, and ML (Webinar series) 11
Task 1: Analysis of high quality RCT data Task 1.1: Conduct analyses of high quality RCT data to estimate ATE using various methods including TMLE Can we reproduce published findings for high quality RCTs? Can we potentially use PS methods or TMLE for analysis of RCT data? Task 1.2 - 1.4: Simulate selection bias, informative missingness, or both in outcome using the RCT data and conduct analyses Can we potentially use PS methods or TMLE to address some methodological issues in traditional clinical trials? Does TMLE perform better than PS methods? 12
Task 1: Analysis of high quality RCT data International Stroke Trial*: RCT data on ~20,000 stroke patients, 99.99% 14-day follow-up Goal: Estimate additive effect of aspirin vs. no aspirin on stroke patient 1) 14-day mortality (9% event rate) 2) 14-day recurrent ischemic stroke (3% event rate) Estimand: Average treatment effect (ATE) Analyses: Unadjusted (conventional) PS matching - with and without adjusting for covariates PS weighting TMLE 13 *International Stroke Trial Collaborative Group (Lancet 1997): https://www.sciencedirect.com/science/article/pii/S0140673697040117
Task 1: Key Take-Away Messages Task 1.1: Conduct analyses of high quality RCT data to estimate ATE using various methods including TMLE Can we reproduce published findings for high quality RCTs? YES Can we potentially use PS methods or TMLE for analysis of RCT data? YES Task 1.2 - 1.4: Simulate selection bias, informative missingness, or both in outcome using the RCT data and conduct analyses Can we potentially use PS methods or TMLE to address some methodological issues in traditional clinical trials? YES Does TMLE perform better than PS methods? YES (for most cases) 14
Task 2: Analysis of RCT data with substantial LTFU Acupuncture for Chronic Headache in Primary Care*: Pragmatic RCT data among 401 patients with a history of chronic headache Goal: Estimate effect of acupuncture vs. usual care on average weekly headache score at 12 months Estimand: ATE Data collection: At 3, 5, 9, 12 months; ~25% missing outcome by end of 12- month follow-up Analyses: Study Authors: Complete case & sensitivity analysis using imputation method PS matching - with and without adjusting for baseline covariates IPW using both inverse probability of treatment & censoring weights TMLE used the intermediate data 15 *AJ Vickers, et al. Acupuncture for chronic headache in primary care: large, pragmatic, randomised trial. BMJ, 2004.
Task 2: Key Take-Away Messages PS Matching Post-match adjustment was clearly helpful (and recommended by experts) Results sometimes varied substantially depending on the order of matching PS weighting Using SL improved (slightly) performance Relatively stable performance TMLE + SL Quite stable across the different analytic specifications Most efficient; smallest standard errors (SEs) and narrowest confidence intervals 16
Task 2: Key Take-Away Messages TMLE + SL: Powerful and flexible enough to handle complex longitudinal data structure (e.g., treatment non-compliance, loss to follow-up) and produces reliable, stable, efficient results Pre-specifying ML algorithms defines the analysis before any data are collected, while preserving the ability to fit flexible models in a data- adaptive fashion 17
Task 3: Analysis of Real World Data Ritodrine hydrochloride and pulmonary edema in twin pregnancy*: retrospective observational cohort study on 225 Japanese women Goal: Estimate effect of a 1-unit increase in ritodrine dose on the risk of pulmonary edema Estimand: Conditional effect (measured in odds ratio), interpreted as an association measure Analyses: Study authors: Adjusted analysis using baseline, post-treatment, and post- outcome covariate adjustment (highly problematic) PS matching - with and without adjusting for baseline covariates PS weighting using both inverse probability of treatment & censoring weights TMLE *Shinohara S et al. Association between total dose of ritodrine hydrochloride and pulmonary oedema in twin pregnancy: A retrospective cohort study in Japan. BMJ Open 2017. 18
Task 3: Key Take-Away Messages Authors analysis: Data does not support their model assumptions (e.g., monotonic dose-response relationship) Flawed covariate adjustment (post-treatment, post-outcome covariate adjustment) Conclusion:Following the TL-roadmap, which involves the assessment of underlying assumptions, revealed that the authors target estimand is not identifiable (i.e., the author s study question is not addressable) from the data without making strong modeling assumptions that are not supported by the data. Therefore, the validity and interpretability of study findings are questionable. 19
Task 3: Key Take-Away Messages Following the TL-roadmap helps us think through about what types of questions can be addressed by the available data. For example, 1) Consider a point-treatment effect (a binary treatment indicator; yes/no) instead of dose 2) Consider a realistic treatment rule that conforms to how treatment is administered in the real world: e.g., compare Always treat with ritodrine unless it is expressly contra-indicated vs Never treat with ritodrine 20
Project Deliverables and Useful Resources for FDA Staffs
TMLE: Deliverables 1. Completed Task 1,2, and 3. Manuscripts are in preparation 2. Task 4: TL Webinars: Recordings are available on FDA CDER RWE SP1 and youtube2 Instructional videos on TMLE and Super Learning will be available on Youtube3 3. TL-based annotated SAP Targeted Learning Roadmap to achieve goals and standards in ICH E9 (R1) Respecting intercurrent events typically necessitates a longitudinal viewpoint Checklists for reviewers / sponsors to completely pre-specify the analysis Draft will be available to the TMLE short course* attendees at the 2021 ASA biopharmaceutical Section Regulatory-Industry Statistics Workshop SC07: Beyond Propensity Score Methods for Real-World Evidence: Targeted Maximum Likelihood Estimation (TMLE) with an Ensemble of Machine Learning Algorithms Instructor: Susan Gruber, Putnam Data Sciences, LLC; Martin Ho, Google Health 22
# 23 TMLE: Useful Resources FDA BAA TMLE Webinars/Training List # Presenter Title URL Date Recorded Date made Public 1 Dr. Mark van der Laan Targeted Machine Learning for Causal Inference based on Real World Data https://youtu.be/PrPNP5RVcLg 02/05/2020 04/24/2020 2 Dr. Susan Gruber An Introduction toTargeted Maximum Likelihood Estimation of Causal Effects https://youtu.be/8Q9dfW3oOi4 03/11/2020 04/24/2020 3 Dr. Eric Polley An Introduction to Super Learning 04/08/2020 04/24/2020 4 Dr. Nicole Bohme Carnegie Introduction to Bayesian Additive Regression Trees (BART) for Causal Inference https://youtu.be/9d5-3_7u5a4 05/13/2020 05/15/2020 5 Dr. David Benkeser Practical Issues in Targeted Learning https://youtu.be/t0ggpi-G_Cs 06/03/2020 08/14/2020 6 Dr. Mireille Schnitzer The Causal Inference of Longitudinal Exposures with Marginal Structural Models: An Overview and Application of Longitudinal TMLE https://youtu.be/_C3BRowGGfI 07/22/2020 08/03/2020 7 Dr. Iv n D az Advantages of covariate adjustment in randomized studies with time-to-event endpoints https://youtu.be/wAYEDWSab24 09/30/2020 10/05/2020 8 Dr. Alan Hubbard Cross-validated Targeted Maximum Likelihood Estimation (CV-TMLE) https://youtu.be/MDmddX267Ys 10/21/2020 10/29/2020 9 Dr. Mark van der Laan Highly Adaptive Lasso (HAL) in Causal Inference https://youtu.be/YnXVAtzF4ss 12/09/2020 12/18/2020 10 Dr. Romain Pirracchio Targeted Machine Learning in action in the ICU https://youtu.be/fJ_p-a37Q3Y 03/24/2021 03/31/2021 11* Dr. Susan Gruber Developing a Targeted Learning-Based Statistical Analysis Plan https://youtu.be/ANuvSrkJ3Ag 04/29/2021 05/03/2021 12* Rachael Phillips Practical Considerations for Specifying a Super Learner https://youtu.be/WYnjja8DKPg 05/20/2021 updated 5/29/2021 6/29/2021 13 Dr. Laura Balzer Challenges and Solutions in the Analysis of Cluster Randomized Trials https://youtu.be/B8TZaBgA3nc 06/23/2021 23 www.fda.gov *TMLE Trainings
# 24 TMLE: Upcoming Resources 1. Findings from BAA Tasks 1, 2, 3 2. TL-based annotated SAP 3. Guidance on using TMLE + SL in practice Practical considerations for using TMLE and SL to estimate causal effects 4. Truncation bound for PS weighting and TMLE General purpose bounding strategy that minimizes MSE 5. In person, hands-on training 24 www.fda.gov
Acknowledgment PI: Susan Gruber, Putnam Data Sciences, LLC. Consultant: Mark van der Laan, U of California, Berkeley Research assistant: Rachael Phillips, U of California, Berkeley Observer: Martin Ho, Google Health (former FDA/CBER) FDA/CDER/OMP: John Concato, Kenneth Quinto, Dianne Paraoan, Nahleen Lopez 25
# 26 Reference 1. FDA CDER RWE SharePoint Website: https://fda.sharepoint.com/sites/CDER-Boards- MPC/RWE/Lists/Materials%20from%20Public%20Meetings%20and%20 Workshops/AllItems.aspx 2. FDA BAA TMLE: TMLE webinars on Youtube: https://www.youtube.com/channel/UC6Cg1XjzX-MlyxKIWfHezFQ (Presentation slides are available for download at putnamds.com/#TargetedLearning) 3. Instructional videos on TMLE and Super Learning on Youtube: https://www.youtube.com/channel/UC6Cg1XjzX-MlyxKIWfHezFQ/videos 26 www.fda.gov
