Interactive Web-Based Shiny Apps for Meta-Analysis of Diagnostic Test Accuracy
Interactive, web-based
Shiny apps to conduct
meta-analysis of
diagnostic test accuracy data
through a ‘
point and click’
interface
and create novel
data visualisations
Presented by Nicola Cooper, Alex Sutton, Suzanne Freeman, Clareece Nevill, Ryan Field
•
Enzo Cerullo, Tom Morris, Janion Nevill (Leicester)
•
Amit Patel (Birmingham)
•
Terry Quinn, Olivia Wu (Glasgow)
•
All members of the NIHR Complex Reviews
Support Unit
•
Feedback from users
•
This project is funded by the
National Institute
for Health and Care Research (NIHR) Complex
Reviews Support Unit
(project number
14/178/29). The views expressed are those of the
author(s) and not necessarily those of the NIHR
or the Department of Health and Social Care
.
A
c
k
n
o
w
l
e
d
g
e
m
e
n
t
s
•
Originally set up as a support group to assist NIHR review groups
including Cochrane (2015-2023)
•
Advice and training:
•
Refining review questions and scope
•
Applications, protocols and report writing
•
Consideration of types of data and structure
•
Appropriate methodological approaches
•
Apps developed to help with implementation of more advanced
evidence synthesis methods for complex data structures
Online Evidence
Synthesis Analysis Apps
Bayesian Meta-Analysis
of DTA studies
(basic –
advanced approaches)
Network
Meta-Analysis (NMA):
NMA (frequentist &
Bayesian approaches)
Evidence
Based Research:
Sample size calculator
for new study based on
adding it to an existing
pairwise meta-analysis
http://www.nihrcrsu.org/guidance/
DTA Primer
Educational
primer on DTA
studies
Diagnostic Test Accuracy (DTA):
Meta-Analysis of DTA studies
(frequentist & mostly
standard approaches)
Feasibility into online
interactive publication
of a living NMA
App Principles
•
Free to use and
open source
•
Where possible
utilise existing R
packages
•
Point and click
interface
•
Emphasis on
visualization and
methods for
sensitivity analysis
D
i
a
g
n
o
s
t
i
c
t
e
s
t
s
:
O
v
e
r
v
i
e
w
•
Diagnostic tests are routinely used in
healthcare for confirming the presence or
absence of disease
•
Diagnostic tests rarely 100% accurate
•
Diagnostic test accuracy (DTA) measures the
ability of a test to detect a condition when it
is present and detect the absence of a
condition when it is absent
Diseased
Non-diseased
For an interactive explorable explanation of DTA evaluation see our DTA
primer available at
https://crsu.shinyapps.io/diagprimer/
M
e
t
a
D
T
A
:
M
o
t
i
v
a
t
i
o
n
•
Meta-analysing diagnostic test accuracy (DTA) data is more
challenging than for effectiveness data:
•
Two dependent variables:
•
Sensitivity
–The proportion of people with the disease who are
correctly diagnosed as positive by the test; i.e. the true positive rate.
•
Specificity
– The proportion of people without the disease who are
correctly diagnosed as negative by the test; i.e. the true negative rate
•
Requires fitting relatively complex bivariate statistical models
•
Software such as Stata, R and SAS require statistical knowledge
whilst RevMan cannot fit bivariate models
M
e
t
a
D
T
A
:
A
i
m
s
•
To develop a
freely
available
user friendly
,
web-based
“point
and click”
interactive tool
which allows users to input their
diagnostic test accuracy (DTA) study data and conduct meta-
analyses for DTA reviews
•
Including incorporation of quality assessment (via QUADAS-2) and
sensitivity analysis
•
To develop
interactive
graphical displays
to facilitate
exploration of the DTA data and effective communication of the
results
•
Customise, explore and export plots
M
e
t
a
D
T
A
:
I
n
t
e
r
f
a
c
e
•
We used the statistical software R and the existing packages
Shiny and lme4.
•
Shiny allowed the creation of a web application with interactive user
interfaces
•
lme4 is a package in R that fits generalised linear mixed effect models.
•
MetaDTA
is hosted on the shinyapps server and is available to
any user with a web browser, without requiring any specialist
statistical software. The application is available at
https://crsu.shinyapps.io/dta_ma/
M
e
t
a
D
T
A
v
2
.
0
.
5
D
e
m
o
:
h
t
t
p
s
:
/
/
c
r
s
u
.
s
h
i
n
y
a
p
p
s
.
i
o
/
d
t
a
_
m
a
/
•
Load data tab
•
Preloaded example dataset
•
Load own dataset
•
Meta-analysis tab
•
Study-level outcomes table
•
SROC plot – confidence region, predictive region, risk of bias
•
Statistics – options
•
Parameter estimates
•
Parameters for RevMan
•
Forest plots – sensitivity and specificity
•
Sensitivity analysis tab
•
Excluding studies, same options as for Meta-analysis
•
Prevalence tab
•
Schematic plot for communicating results
•
Options to set number of patients and prevalence
S
u
m
m
a
r
y
•
Advantages of MetaDTA
•
Fits bivariate meta-analysis model for diagnostic test accuracy data
•
Includes risk of bias and sensitivity analysis
•
Includes interactive graphics for data exploration
•
Facilitates effective communication of results
•
Limitations of MetaDTA
•
Imperfect gold standard
•
Different thresholds
•
Comparative analysis
•
Subgroup analysis /meta-regression
•
MetaBayesDTA
https://crsu.shinyapps.io/MetaBayesDTA/
https://crsu.shinyapps.io/MetaBayesDTA/
Overcomes many of the limitations of MetaDTA (but does
not allow data at multiple thresholds per study)
M
e
t
a
B
a
y
e
s
D
T
A
(
c
u
r
r
e
n
t
l
y
i
n
l
a
t
e
B
e
t
a
)
•
Written by Enzo
Cerullo
•
Create an extended version of MetaDTA which fits more complex
models
•
Uses Bayesian methods as implemented in STAN software
exclusively – bespoke STAN code used throughout
•
Encourage good practices
•
Automatically report predictive intervals
•
Prior distributions visualised before model fitting
•
Model and sampler diagnostics automatically presented
•
Updated user interface over MetaDTA
M
e
t
a
B
a
y
e
s
D
T
A
v
1
.
4
(
B
e
t
a
)
D
e
m
o
:
h
t
t
p
s
:
/
/
c
r
s
u
.
s
h
i
n
y
a
p
p
s
.
i
o
/
M
e
t
a
B
a
y
e
s
D
T
A
M
e
t
a
B
a
y
e
s
D
T
A
:
F
u
r
t
h
e
r
I
n
f
o
•
A document showing how analyses from the Cochrane Handbook can be
replicated in MetaBayesDTA is near completion – and should be available soon
•
An early version is available for use today in the practical
F
i
n
a
l
C
o
m
m
e
n
t
s
•
Both apps:
•
have links to YouTube tutorials within them
•
source
code is freely available on GitHub
•
can be run locally in R if speed / internet connection / memory / confidentiality concerns are a
problem
•
Testing and bug fixing of MetaBayesDTA ongoing – hope full release soon
•
MetaDTA has a manual within app
•
A document showing how all possible analyses from the Cochrane Handbook can be
replicated in MetaBayesDTA is being written – and should be available soon
•
If anyone is interested in contributing, please get in touch
P
r
a
c
t
i
c
a
l
•
Worksheet for MetaDTA
•
Document outlining how to replicate analyses in Cochrane
Handbook using MetaBayesDTA
•
Both can be downloaded from:
http://bit.ly/cochrane-colloquium
A
n
y
Q
u
e
s
t
i
o
n
s
?
Resources
Links
•
MetaDTA App:
https://crsu.shinyapps.io/dta_ma/
•
GitHub:
https://github.com/CRSU-Apps/MetaDTA
•
MetaBayesDTA App:
https://crsu.shinyapps.io/MetaBayesDTA/
•
GitHub:
https://github.com/CRSU-Apps/MetaBayesDTA
Future Work
•
Binary & Continuous Outcomes
•
Frequentist and Bayesian CNMA
•
Frequentist - Additive model
•
Bayesian – 4 Models from Welton
et al.
M
e
t
a
C
N
M
A
C
o
m
p
o
n
e
n
t
N
e
t
w
o
r
k
A
n
a
l
y
s
i
s
-
S
i
m
p
l
i
f
i
e
d
F
e
e
d
b
a
c
k
References
•
Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton
AJ
.
Development of an interactive web-based tool to conduct
and interrogate meta-analysis of diagnostic test accuracy
studies: MetaDTA
.
BMC Medical Research Methodology
(2019);
19
:
81
https://doi.org/10.1186/s12874-019-0724-x
•
Patel A, Cooper NJ, Freeman SC, Sutton AJ.
Graphical
enhancements to summary receiver operating characteristic
plots to facilitate the analysis and reporting of meta-analysis
of diagnostic test accuracy data
.
Research Synthesis Methods
(2021); 12: 34-44.
https://doi.org/10.1002/jrsm.1439
•
Cerullo E, Sutton AJ, Jones HE, Wu O, Quinn TJ, Cooper NJ.
MetaBayesDTA: codeless Bayesian meta-analysis of test
accuracy, with or without a gold standard
. BMC Medical
Research Methodology (2023); 127.
https://doi.org/10.1186/s12874-023-
01910-y
Open Source
•
Source code for all apps available on GitHub:
•
https://github.com/CRSU-Apps
Explore interactive Shiny apps presented by Nicola Cooper and team for conducting meta-analysis of diagnostic test accuracy data with a point-and-click interface. Find educational primers, principles, and tools to enhance evidence synthesis methods and visualization in healthcare.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Interactive, web-based Shiny apps to conduct meta-analysis of diagnostic test accuracy data through a point and click interface and create novel data visualisations Presented by Nicola Cooper, Alex Sutton, Suzanne Freeman, Clareece Nevill, Ryan Field
Enzo Cerullo, Tom Morris, Janion Nevill (Leicester) Acknowledgements Acknowledgements Amit Patel (Birmingham) Terry Quinn, Olivia Wu (Glasgow) All members of the NIHR Complex Reviews Support Unit Feedback from users This project is funded by the National Institute for Health and Care Research (NIHR) Complex Reviews Support Unit (project number 14/178/29). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
Originally set up as a support group to assist NIHR review groups including Cochrane (2015-2023) Advice and training: Refining review questions and scope Applications, protocols and report writing Consideration of types of data and structure Appropriate methodological approaches Apps developed to help with implementation of more advanced evidence synthesis methods for complex data structures
Diagnostic Test Accuracy (DTA): DTA Primer Educational primer on DTA studies Bayesian Meta-Analysis of DTA studies (basic advanced approaches) Meta-Analysis of DTA studies (frequentist & mostly standard approaches) App Principles Free to use and open source Where possible utilise existing R packages Online Evidence Synthesis Analysis Apps http://www.nihrcrsu.org/guidance/ Point and click interface Network Meta-Analysis (NMA): Evidence Emphasis on visualization and methods for sensitivity analysis Based Research: Sample size calculator for new study based on adding it to an existing pairwise meta-analysis NMA (frequentist & Bayesian approaches) Feasibility into online interactive publication of a living NMA
Diagnostic tests: Overview Diagnostic tests: Overview Diagnostic tests are routinely used in healthcare for confirming the presence or absence of disease Diagnostic tests rarely 100% accurate Diagnostic test accuracy (DTA) measures the ability of a test to detect a condition when it is present and detect the absence of a condition when it is absent Diseased Non-diseased For an interactive explorable explanation of DTA evaluation see our DTA primer available at https://crsu.shinyapps.io/diagprimer/
MetaDTA MetaDTA: Motivation : Motivation Meta-analysing diagnostic test accuracy (DTA) data is more challenging than for effectiveness data: Two dependent variables: Sensitivity The proportion of people with the disease who are correctly diagnosed as positive by the test; i.e. the true positive rate. Specificity The proportion of people without the disease who are correctly diagnosed as negative by the test; i.e. the true negative rate Requires fitting relatively complex bivariate statistical models Software such as Stata, R and SAS require statistical knowledge whilst RevMan cannot fit bivariate models
MetaDTA MetaDTA: Aims : Aims To develop a freely available user friendly, web-based point and click interactive tool which allows users to input their diagnostic test accuracy (DTA) study data and conduct meta- analyses for DTA reviews Including incorporation of quality assessment (via QUADAS-2) and sensitivity analysis To develop interactivegraphical displays to facilitate exploration of the DTA data and effective communication of the results Customise, explore and export plots
MetaDTA MetaDTA: Interface : Interface We used the statistical software R and the existing packages Shiny and lme4. Shiny allowed the creation of a web application with interactive user interfaces lme4 is a package in R that fits generalised linear mixed effect models. MetaDTA is hosted on the shinyapps server and is available to any user with a web browser, without requiring any specialist statistical software. The application is available at https://crsu.shinyapps.io/dta_ma/
MetaDTA MetaDTA v2.0.5 Demo: v2.0.5 Demo: https://crsu.shinyapps.io/dta_ma/ Load data tab Preloaded example dataset Load own dataset Meta-analysis tab Study-level outcomes table SROC plot confidence region, predictive region, risk of bias Statistics options Parameter estimates Parameters for RevMan Forest plots sensitivity and specificity Sensitivity analysis tab Excluding studies, same options as for Meta-analysis Prevalence tab Schematic plot for communicating results Options to set number of patients and prevalence
Summary Summary Advantages of MetaDTA Fits bivariate meta-analysis model for diagnostic test accuracy data Includes risk of bias and sensitivity analysis Includes interactive graphics for data exploration Facilitates effective communication of results Limitations of MetaDTA Imperfect gold standard Different thresholds Comparative analysis Subgroup analysis /meta-regression MetaBayesDTA https://crsu.shinyapps.io/MetaBayesDTA/
https://crsu.shinyapps.io/MetaBayesDTA/ Overcomes many of the limitations of MetaDTA (but does not allow data at multiple thresholds per study)
MetaBayesDTA MetaBayesDTA (currently in late Beta) (currently in late Beta) Written by Enzo Cerullo Create an extended version of MetaDTA which fits more complex models Uses Bayesian methods as implemented in STAN software exclusively bespoke STAN code used throughout Encourage good practices Automatically report predictive intervals Prior distributions visualised before model fitting Model and sampler diagnostics automatically presented Updated user interface over MetaDTA
MetaBayesDTA MetaBayesDTA v1.4 (Beta) Demo: v1.4 (Beta) Demo: https://crsu.shinyapps.io/MetaBayesDTA https://crsu.shinyapps.io/MetaBayesDTA
MetaBayesDTA MetaBayesDTA : Further Info : Further Info A document showing how analyses from the Cochrane Handbook can be replicated in MetaBayesDTA is near completion and should be available soon An early version is available for use today in the practical
Final Comments Final Comments Both apps: have links to YouTube tutorials within them source code is freely available on GitHub can be run locally in R if speed / internet connection / memory / confidentiality concerns are a problem Testing and bug fixing of MetaBayesDTA ongoing hope full release soon MetaDTA has a manual within app A document showing how all possible analyses from the Cochrane Handbook can be replicated in MetaBayesDTA is being written and should be available soon If anyone is interested in contributing, please get in touch
Practical Practical Worksheet for MetaDTA Document outlining how to replicate analyses in Cochrane Handbook using MetaBayesDTA Both can be downloaded from: http://bit.ly/cochrane-colloquium
Any Questions? Any Questions?
Resources Links MetaDTA App: https://crsu.shinyapps.io/dta_ma/ GitHub: https://github.com/CRSU-Apps/MetaDTA MetaBayesDTA App: https://crsu.shinyapps.io/MetaBayesDTA/ GitHub: https://github.com/CRSU-Apps/MetaBayesDTA
Future Work MetaCNMA MetaCNMA Binary & Continuous Outcomes Frequentist and Bayesian CNMA Frequentist - Additive model Bayesian 4 Models from Welton et al. Component Component Network Analysis Network Analysis - - Simplified Simplified
Feedback Feedback
References Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton AJ. Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Medical Research Methodology (2019); 19: 81 https://doi.org/10.1186/s12874-019-0724-x Patel A, Cooper NJ, Freeman SC, Sutton AJ. Graphical enhancements to summary receiver operating characteristic plots to facilitate the analysis and reporting of meta-analysis of diagnostic test accuracy data. Research Synthesis Methods (2021); 12: 34-44. https://doi.org/10.1002/jrsm.1439 Cerullo E, Sutton AJ, Jones HE, Wu O, Quinn TJ, Cooper NJ. MetaBayesDTA: codeless Bayesian meta-analysis of test accuracy, with or without a gold standard. BMC Medical Research Methodology (2023); 127. https://doi.org/10.1186/s12874-023- 01910-y
Open Source Source code for all apps available on GitHub: https://github.com/CRSU-Apps
