Understanding Measurement Error, Reliability, and Validity in Research
This content dives into the concepts of measurement error, reliability, and validity in research studies. It discusses the impact of measurement error on study validity, explains the importance of reliability in obtaining consistent results, and highlights the significance of validity in accurately representing the intended attribute. The content also covers practical considerations and approaches related to design and analysis in research studies.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
QSU Seminar Reliability and Validity: Design and Analytic Approaches Practical Considerations Rita Popat, PhD Dept of Health Research & Policy Division of Epidemiology rpopat@stanford.edu
What do we want to know about the measurements? Why? Dependent variable (outcome) Independent variable (risk factor or predictor) JAMA. 2004; 292:1188-1194 2
What are other possible explanations for not detecting an association?
Outline Definitions: Measurement error, reliability, validity Why should we care about measurement error? Effects of measurement error on Study Validity (Categorical Exposures) Effects of measurement error on Study Validity (Continuous Exposures) Measures (or indices) for reliability and validity
Measurement error For an individual, measurement error is the difference between his/her observed and true measurement. Measurement error can occur in dependent (outcome) or independent (predictor or exposure) variables For categorical variables, measurement error is referred to as misclassification Measurement error is an important source of bias that can threaten internal validity of a study 6
Reliability (aka reproducibility, consistency) Reliability is the extent to which repeated measurement of a stable phenomenon- by the same person or different people and instruments, at different times and places- obtain similar results. A precise measurement is reproducible, that is, has the same (or nearly the same) value each time it is measured. The higher the reliability, the greater the statistical power for a fixed sample size Reliability is affected by random error 7
Validity or Accuracy The accuracy of a variable is the degree to which it actually represents what it is intended to represent That is: The extent to which the measurement represents the true value of the attribute being assessed. 8
Precise (Reliable) and Accurate (Valid) measurements are key to minimizing measurement error Precision, no accuracy Accuracy, low precision Precision and accuracy No precision, no accuracy 9
Measurement error in Categorical Variables Referred to as Misclassification and could be in the Outcome variables, or Exposure variables How do we know misclassification exists? When method used for classifying exposure lacks accuracy
Assessment of Accuracy Criterion validity (compare against a reference or gold standard) True classification Present + Absent - Present + a b a+b Imperfect Classification TP FP Present - c d c+d FN TN a+c b+d Sensitivity = a / (a+c) Specificity = d / (b+d) False negative = c / (a+c) False positive = b / (b+d) 11
Misclassification of exposure Cases Controls (outcome -) b (outcome +) a Exposure + c Exposure - d Non-differential Differential 12 12
Misclassification of exposure True exposure Cases Controls Exposure + 50 20 50 ( )( 80 ) = = 0 . 4 OR ( 20 )( 50 ) Exposure - 50 80 Reported exposure:90% sensitivity & 80% specificity in cases & controls Cases Controls Exposure + 55 34 55 ( )( 66 ) = = 4 . 2 OR 34 ( )( 45 ) Exposure - 45 66 Attenuation of true association due to misclassification of exposure 13
Misclassification of the exposure Cases Controls (outcome -) b (outcome +) a Exposure + c Exposure - d Non-differential misclassification occurs when the degree of misclassification of exposure is independent of outcome/disease status Tends to bias the association toward the null Occurs when the sensitivity and specificity of the classification of exposure are same for those with and without the outcome but less than 100% 14 14
Bias toward the null hypothesis Underestimation of a relative risk or odds ratio for True value Observed value 0 1 2 A. Risk factor Bias toward the null hypothesis True value Observed value 0 1 2 B. Protective factor Modified from Greenberg. Fig 10-4, chapter 10
Misclassification of the exposure True exposure Cases Controls Exposure + 50 20 50 ( )( 80 ) = = 0 . 4 OR ( 20 )( 50 ) Exposure - 50 80 Reported exposure: Cases - 96% sensitivity and 100% specificity Controls- 70% sensitivity and 100% specificity Cases Controls Exposure + 48 14 ( 48 )( 86 ) = 7 . 5 = OR 52 ( )( 14 ) Exposure - 52 86 16
Misclassification of the exposure True exposure Cases Controls Exposure + 50 20 50 ( )( 80 ) = = 0 . 4 OR ( 20 )( 50 ) Exposure - 50 80 Reported exposure: Cases - 96% sensitivity and 100% specificity Controls- 70% sensitivity and 80% specificity Cases Controls Exposure + 48 30 ( 48 )( 70 ) = = 1 . 2 OR 52 ( )( 30 ) Exposure - 52 70 17
Misclassification of the exposure Cases Controls (outcome -) b (outcome +) a Exposure + c Exposure - d Differential misclassification occurs when the degree of misclassification differs between the groups being compared. May bias the association either toward or away from the null hypothesis Occurs when the sensitivity and specificity of the classification of exposure differ for those with and without the outcome 18 18
Bias away from the null hypothesis Overestimation of a relative risk or odds ratio for Observed value True value 0 1 2 A. Risk factor Bias away from the null hypothesis Observed value True value 0 1 2 B. Protective factor Modified from Greenberg. Fig 10-4, chapter 10
Cases Controls Index Index Proxy (~25%) Proxy (~25%) Hormone therapy Never Former Current accuracy? Pharmacy database
Summary so far. Misclassification of exposure is an important source of bias Good to know something about the validity of measurement for exposure classification before the study begins Almost impossible to avoid misclassification, but try to avoid differential misclassification If the study has already been conducted, develop analytic strategies that explore exposure misclassification as a possible explanation of the observed results (especially for a primary exposure of interest)
Measurement error in Continuous Variables Physiologic measures (SBP, BMI) Biomarkers (hormone levels, lipids) Nutrients Environmental exposures Outcome measures (QOL, function)
Measurement Theory: Example contd.. Measurement error Ti Ti + b Ti + b + Ei _________ b _______ ______________ E _______________ + systematic error in + additional "random error" for subject i instrument (bias) EXAMPLE: One measured diastolic blood pressure (DBP) as indicator of 2-year average DBP. BD cuff miscalibrated -- measures everyone's diastolic BP as 10 mm Hg less + randomness in BP cuff mechanics + subject i's 10 mmHg increase over 2-year average + subject intimidated - diastolic BP 20 mmHg higher than usual + misreading by interviewer + random fluctuations in current BP 24
Non- Differential Measurement error The effects of non-differential measurement error on the odds ratio. ORT is the true odds ratio for exposure versus reference level r. ORO is the observable odds ratio for exposure versus reference level r.
2 XT = = = / 1 O / 1 . 0 62 . 0 82 . 0 73 RR RR T
Summary so far. Measurement error is an important source of bias Good to know something about the validity of measurement for exposure before the study begins Almost impossible to avoid misclassification, but try to avoid differential misclassification! Non-differential measurement error will attenuate the results towards the null, resulting in loss of power for a fixed sample size This should be taken into account when estimating sample size during the planning stage and Interpretation of results and determining internal validity of a study
So why should we evaluate reliability and validity of measurements? If it precedes the actual study, it tells us whether the instrument/method we are using is reliable and valid This information can help us run sensitivity analysis or correct for the measurement error in the variables after the study has been completed
Outline Definitions: Measurement error, reliability, validity Why should we care about measurement error? Effects of measurement error on Study Validity (Categorical Exposures) Effects of measurement error on Study Validity (Continuous Exposures) Measures (or indices) for reliability and validity
Choice of reliability and validity measures depend on type of variable . . . Type of Variable Reliability Measure(s) Validity Measure(s) Dichotomous Kappa sensitivity, specificity misclassification matrix Ordinal weighted kappa ICC* ICC * Bland Altman Plots Continuous Pearson correlation (see note) Bland-Altman Plots *ICC intraclass correlation coefficient Note: in inter-method reliability studies, inferences about validity can be made from coefficients of reproducibility (such as the Pearson s correlation ) 38
Assessing Accuracy (Validity) of continuous measures Bias: difference between the mean value as measured and the mean of the true values So bias = Standardized bias = x X x X x X SD X Bland-Altman plots 39
Bland and Altman plots Take two measurements (different methods or instrument) on the same subject For each subject, plot the difference b/w the two measures (y axis) vs. the mean of the two measures We expect the mean difference to be 0 We expect 95% of the differences to be within 2 standard deviations (SD)
Suppose there is no gold standard, then how do we evaluate validity? .. We make inferences from inter-method reliability studies! Note: will not be able to estimate bias when the two measures are based on different scales
Inferences about validity from inter-method reliability studies Suppose two different methods (instruments) are used to measure the same continuous exposure. Let X1 denote the measure of interest (i.e., the one to be used to measure the exposure in the study) and X2 is the comparison measure We have the reliability coefficient 1x x 2 However, we are actually interested in the validity coefficient: Tx 1 Example: Is self-reported physical activity valid? Compare it to the 4-week diary. 44
Relationship of Reliability to Validity Errors of X1 and X2 are: Relationship b/w reliability and validity Usual application 1. Uncorrelated and both measures are equally precise Intramethod study = = Tx Tx x x 1 2 1 2 2. Uncorrelated , X2 is more precise than X1 Intermethod study x x Tx x x 1 2 1 1 2 3. Uncorrelated , X1 is more precise than X2 Intermethod study Tx x x 1 1 2 4. Correlated errors and both measures are equally precise Intramethod study Intermethod study Tx x x 1 1 2 Take home message: In most situations the square root of the reliability coefficient can provide an upper limit to the validity coefficient 45
Inferences about validity from inter-method reliability studies In our example, X1 is measure of interest (i.e., the one to be used to measure the exposure in the study: self-reported activity) and X2 is the comparison measure (4-wk diaries) We have the reliability coefficient x = 0.79 1x 2 Errors in X1 and X2 are likely to be uncorrelated and X2 is more precise than X1, so 0.79 < < 0.89 1 Tx So, self-reported activity appears to be a valid measure 46
Summary ofInferences From Reliability to Validity Reliability studies are used to interpret validity of x. Reliability is necessary for validity (instrument cannot be valid if it is not reproducible). Reliability is not sufficient for validity - repetition of test may yield same result because both X1 and X2 measure some systematic error (i.e., errors are correlated). Reliability can only give an upper limit on validity. If the upper limit is low, then the instrument is not valid. An estimate of reliability (or validity) depends on the sample (i.e., may vary by age, gender, etc.) 47
Reliability of continuously distributed variables Pearson product-moment correlation? Spearman rank correlation? 48
Butdoes correlation tell you about But does correlation tell you about relationship or agreement? relationship or agreement? 190 Measure 1 Measure 2 150 155 160 163 170 174 185 155 158 165 170 176 184 180 175 Measure 2 170 165 160 155 150 145 150 155 160 165 170 175 180 Measure 1 Pearson s Correlation coefficient=0.99 Is this measure reliable? 49
Reliability of continuously distributed variables Other methods generally preferred for intra or inter- observer reliability when same method/instrument is used - Intraclass correlation coefficients (ICC): is calculated using variance estimates obtained through an analysis of variance (ANOVA) - Bland-Altman plots Correlation coefficient useful for inter-method reliability to make inferences about validity (especially when the measurement scale differs for the two methods) 50
