Understanding Error Correction and Reproducibility in Science
Explore the importance of severe testing, statistical crisis of replication, and the American Statistical Association's stance on P-values in ensuring reproducibility and error correction in scientific research. Delve into the philosophical, statistical, and historical aspects of error statistical methods for drawing accurate inferences from data while controlling error probabilities.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Severe Testing: The Key to Error Correction Deborah G Mayo Virginia Tech March 17, 2017 Understanding Reproducibility and Error Correction in Science
Statistical Crisis of Replication O Statistical findings disappear when others look for them. O Beyond the social sciences to genomics, bioinformatics, and medicine (Big Data) O Methodological reforms (some welcome, others radical) O Need to understand philosophical, statistical, historical issues 2
American Statistical Association (ASA):Statement on P-values The statistical community has been deeply concerned about issues of reproducibility and replicability of scientific conclusions. . much confusion and even doubt about the validity of science is arising. Such doubt can lead to radical choices such as to ban p-values (ASA, Wasserstein & Lazar 2016, p. 129) 3
I was a philosophical observer at the ASA P-value pow wow 4
Dont throw out the error control baby with the bad statistics bathwater The American Statistician 5
OThe most used methods are most criticized O Statistical significance tests are a small part of a rich set of: techniques for systematically appraising and bounding the probabilities of seriously misleading interpretations of data (Birnbaum 1970, p. 1033) O These I call errorstatisticalmethods (or sampling theory) . 6
Error Statistics OStatistics: Collection, modeling, drawing inferences from data to claims about aspects of processes O The inference may be in error OIt s qualified by a claim about the method s capabilities to control and alert us to erroneous interpretations (error probabilities) 7
p-value. to test the conformity of the particular data under analysis with H0in some respect: we find a function T = t(y) of the data, to be called the test statistic, such that the larger the value of T the more inconsistent are the data with H0; The random variable T = t(Y) has a (numerically) known probability distribution when H0is true. the p-value corresponding to any t0bsas p = p(t) = Pr(T t0bs; H0) (Mayo and Cox 2006, p. 81) 8
Testing Reasoning O If even larger differences than t0bsoccur fairly frequently under H0 (i.e.,P-value is not small), there s scarcely evidence of incompatibility with H0 O Small P-value indicates some underlying discrepancy from H0because very probably you would have seen a less impressive difference than t0bswere H0true. OThis indication isn t evidence of a genuine statistical effect H, let alone a scientific conclusion H* Stat-Sub fallacy H => H* 9
OIm not keen to defend many uses of significance tests long lampooned O I introduce a reformulation of tests in terms of discrepancies (effect sizes) that are and are not severely-tested O The criticisms are often based on misunderstandings; consequently so are many reforms 10
Replication Paradox (for Significance Test Critics) Critic:It s much too easy to get a small P- value You: Why do they find it so difficult to replicate the small P-values others found? Is it easy or is it hard? 11
Only 36 of 100 psychology experiments yielded small P-values in Open Science Collaboration on replication in psychology OSC: Reproducibility Project: Psychology: 2011-15 (Science 2015): Crowd-sourced effort to replicate 100 articles (Led by Brian Nozek, U. VA) 12
OR.A. Fisher: its easy to lie with statistics by selective reporting, not the test s fault O Sufficient finagling cherry-picking, P- hacking, significance seeking, multiple testing, look elsewhere may practically guarantee a preferred claim H gets support, even if it s unwarranted by evidence (biasing selection effects, need to adjust P-values) Note: Support for some preferred claim H is by rejecting a null hypothesis O H hasn t passed a severe test 13
Severity Requirement: If the test procedure had little or no capability of finding flaws with H (even if H is incorrect), then agreement between data x0 and H provides poor (or no) evidence for H ( too cheap to be worth having Popper) O Such a test fails a minimal requirement for a stringent or severe test O My account: severe testing based on error statistics (requires reinterpreting tests) 14
This alters the role of probability: typically just 2 OProbabilism. To assign a degree of probability, confirmation, support or belief in a hypothesis, given data x0 (e.g., Bayesian, likelihoodist) with regard for inner coherency O Performance. Ensure long-run reliability of methods, coverage probabilities (frequentist, behavioristic Neyman-Pearson) 15
What happened to using probability to assess the error probing capacity by the severity requirement? ONeither probabilism nor performance directly captures it O Good long-run performance is a necessary, not a sufficient, condition for severity 16
O Problems with selective reporting, cherry picking, stopping when the data look good, P-hacking, are not problems about long-runs OIt s that we cannot say the case at hand has done a good job of avoiding the sources of misinterpreting data Key to revising the role of error probabilities 17
A claim C is not warranted _______ O Probabilism: unless C is true or probable (gets a probability boost, is made comparatively firmer) O Performance: unless it stems from a method with low long-run error O Probativism (severe testing) unless something (a fair amount) has been done to probe ways we can be wrong about C 18
O If you assume probabilism is required for inference, error probabilities are relevant for inference only by misinterpretation False! O I claim, error probabilities play a crucial role in appraising well-testedness OIt s crucial to be able to say, C is highly believable or plausible but poorly tested 19
Biasing selection effects: O One function of severity is to identify problematic selection effects (not all are) OBiasing selection effects: when data or hypotheses are selected or generated (or a test criterion is specified), in such a way that the minimal severity requirement is violated, seriously altered or incapable of being assessed O Picking up on these alterations is precisely what enables error statistics to be self-correcting 20
Nominal vs actual Significance levels Suppose that twenty sets of differences have been examined, that one difference seems large enough to test and that this difference turns out to be significant at the 5 percent level. .The actual level of significance is not 5 percent, but 64 percent! (Selvin, 1970, p. 104) From (Morrison & Henkel s Significance Test controversy 1970!) 21
O Morrison and Henkel were clear on the fallacy: blurring the computed or nominal significance level, and the actual level O There are many more ways you can be wrong with hunting (different sample space) 22
Spurious P-Value You report: Such results would be difficult to achieve under the assumption of H0 When in fact such results are common under the assumption of H0 (Formally): O You say Pr(P-value Pobs; H0) ~ Pobs small O But in fact Pr(P-value Pobs; H0) = high 23
Scapegoating ONowadays, we re likely to see the tests blamed OMy view: Tests don t kill inferences, people do O Even worse are those statistical accounts where the abuse vanishes! 24
On some views, taking account of biasing selection effects defies scientific sense Two problems that plague frequentist inference: multiple comparisons and multiple looks, or, as they are more commonly called, data dredging and peeking at the data. The frequentist solution to both problems involves adjusting the P- value But adjusting the measure of evidence because of considerations that have nothing to do with the data defies scientific sense, belies the claim of objectivity that is often made for the P-value (Goodman 1999, p. 1010) (To his credit, he s open about this; heads the Meta-Research Innovation Center at Stanford) 25
Technical activism isnt free of philosophy Ben Goldacre (of Bad Science) in a 2016 Nature article, is puzzled that bad statistical practices continue even in the face of the new "technical activism : The editors at Annals of Internal Medicine, repeatedly (but confusedly) argue that it is acceptable to identify prespecified outcomes [from results] produced after a trial began; .they say that their expertise allows them to permit and even solicit undeclared outcome-switching 26
His paper: Make journals report clinical trials properly OHe shouldn t close his eyes to the possibility that some of the pushback he s seeing has a basis in statistical philosophy! 27
Likelihood Principle (LP) The vanishing act links to a pivotal disagreement in the philosophy of statistics battles In probabilisms, the import of the data is via the ratios of likelihoods of hypotheses The data x0 are fixed, while the hypotheses vary P(x0;H1)/P(x0;H0) 28
All error probabilities violate the LP (even without selection effects): Sampling distributions, significance levels, power, all depend on something more [than the likelihood function] something that is irrelevant in Bayesian inference namely the sample space (Lindley 1971, p. 436) The LP implies the irrelevance of predesignation, of whether a hypothesis was thought of before hand or was introduced to explain known effects (Rosenkrantz, 1977, p. 122) 29
Paradox of Optional Stopping: Error probing capacities are altered not just by cherry picking and data dredging, but also via data dependent stopping rules: Xi ~ N( , 2), 2-sided H0: = 0 vs. H1: 0. Instead of fixing the sample size n in advance, in some tests, n is determined by a stopping rule: 30
Trying and trying again O Keep sampling until H0 is rejected at 0.05 level i.e., keep sampling until M 1.96 / n O Trying and trying again: Having failed to rack up a 1.96 difference after 10 trials, go to 20, 30 and so on until obtaining a 1.96 difference 31
Nominal vs. Actual significance levels again: O With n fixed the Type 1 error probability is 0.05 O With this stopping rule the actual significance level differs from, and will be greater than 0.05 OViolates Cox and Hinkley s (1974) weak repeated sampling principle 32
OThe ASA (p. 131) correctly warns that [c]onducting multiple analyses of the data and reporting only those with certain p- values leads to spurious p-values (Principle 4) OThey don t mention that the same p- hacked hypothesis can occur in Bayes factors, credibility intervals, likelihood ratios 33
With One Big Difference: O The direct grounds to criticize inferences as flouting error statistical control is lost O They condition on the actual data, O Error probabilities take into account other outcomes that could have occurred but did not (sampling distribution) 34
How might probabilists block intuitively unwarranted inferences (without error probabilities)? A subjective Bayesian might say: If our beliefs were mixed into the interpretation of the evidence, we wouldn t declare there s statistical evidence of some unbelievable claim (distinguishing shades of grey and being politically moderate, ovulation and voting preferences) 35
Rescued by beliefs O That could work in some cases (it still wouldn t show what researchers had done wrong) battle of beliefs O Besides, researchers sincerely believe their hypotheses OSo now you ve got two sources of flexibility, priors and biasing selection effects 36
No help with our most important problem O How to distinguish the warrant for a single hypothesis H with different methods (e.g., one has biasing selection effects, another, pre-registered results and precautions)? 37
Most Bayesians use default priors O Eliciting subjective priors too difficult, scientists reluctant to allow subjective beliefs to overshadow data O Default, or reference priors are supposed to prevent prior beliefs from influencing the posteriors (O-Bayesians, 2006) 38
OThe priors are not to be considered expressions of uncertainty, ignorance, or degree of belief. Default priors may not even be probabilities (Cox and Mayo 2010, p. 299) ODefault Bayesian Reforms are touted as free of selection effects O Bayes factors can be used in the complete absence of a sampling plan (Bayarri, Benjamin, Berger, Sellke 2016, p. 100) 39
Granted, some are prepared to abandon the LP for model testing In an attempted meeting of the minds Andrew Gelman and Cosma Shalizi say: O [C]rucial parts of Bayesian data analysis, such as model checking, can be understood as error probes in Mayo s sense which might be seen as using modern statistics to implement the Popperian criteria of severe tests. (2013, p.10). OAn open question 40
The ASA doc highlights classic foibles that block replication In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result (Fisher 1935, p. 14) isolated low P-value > H: statistical effect 41
Statistical > substantive (H > H*) [A]ccording to Fisher, rejecting the null hypothesis is not equivalent to accepting the efficacy of the cause in question. The latter...requires obtaining more significant results when the experiment, or an improvement of it, is repeated at other laboratories or under other conditions (Gigerentzer 1989, pp. 95-6) 42
The Problem is with so-called NHST ( null hypothesis significance testing ) O NHSTs supposedly allow moving from statistical to substantive hypotheses O If defined that way, they exist only as abuses of tests O ASA doc ignores Neyman-Pearson (N-P) tests 43
Neyman-Pearson (N-P) tests: A null and alternative hypotheses H0, H1 that are exhaustive H0: 12 vs H1: > 12 O So this fallacy of rejection H impossible O Rejecting the null only indicates statistical alternatives (how discrepant from null) H*is 44
P-values Dont Report Effect Sizes (Principle 5) Who ever said to just report a P-value? O Tests should be accompanied by interpretive tools that avoid the fallacies of rejection and non-rejection. These correctives can be articulated in either Fisherian or Neyman-Pearson terms (Mayo and Cox 2006, Mayo and Spanos 2006) 45
To Avoid Inferring a Discrepancy Beyond What s Warranted: large n problem. OSeverity tells us: an -significant difference is indicative of less of a discrepancy from the null if it results from larger (n1) rather than a smaller (n2) sample size (n1 > n2 ) 46
OWhats more indicative of a large effect (fire), a fire alarm that goes off with burnt toast or one so insensitive that it doesn t go off unless the house is fully ablaze? O [The larger sample size is like the one that goes off with burnt toast] 47
What About Fallacies of Non-Significant Results? O They don t warrant 0 discrepancy O Use the same severity reasoning to rule out discrepancies that very probably would have resulted in a larger difference than observed- set upper bounds O If you very probably would have observed a more impressive (smaller) p-value than you did, if > 1( 1= 0+ ), then the data are good evidence that < 1 O Akin to power analysis (Cohen, Neyman) but sensitive to x0 48
OTheres another kind of fallacy behind a move that s supposed improve replication but it confuses the notions from significance testing and it leads to Most findings are false O Fake replication crisis. 49
Diagnostic Screening Model of Tests: urn of nulls ( most findings are false ) O If we imagine randomly select a hypothesis from an urn of nulls 90% of which are true O Consider just 2 possibilities: H0: no effect H1: meaningful effect, all else ignored, O Take the prevalence of 90% as Pr(H0 you picked) = .9, Pr(H1)= .1 O Rejecting H0 with a single (just) .05 significant result, Cherry-picking to boot 50
