Understanding HARKing and the Importance of Exploratory vs. Confirmatory Research
HARKing (Hypothesizing After the Results are Known) involves presenting unexpected findings as if they were predicted and treating exploratory work as confirmatory. Exploratory research allows for hypothesis generation, while confirmatory research focuses on hypothesis testing. It is crucial to differentiate between the two to avoid research bias and misleading interpretations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Questionable Research Practices Part 4: HARKing 1
HARKing: Hypothesizing After the Results are Known AKA: Forking Paths Researchers Degrees of Freedom 2
HARKing: Hypothesizing After the Results are Known HARKing includes [i] presenting ad hoc and/or unexpected findings as though they had been predicted all along (Kerr 1998); and [ii] presenting exploratory work as though it was confirmatory hypothesis testing (Wagenmakers et al. 2012). 3
HARKing: Hypothesizing After the Results are Known 4
Exploratory vs. Confirmatory Research Exploratory research: Analysis steps not specified in advance. Confirmatory research: Analysis steps have been specified in advance of data collection. 5
Exploratory vs. Confirmatory Research. Both are essential. Exploratory research: Explore ( get to know ) your biological system: Correlations? Which dependent variables seem to respond to independent variables? Anything unusual or surprising? What appears to be happening? Generate new hypotheses Confirmatory research: A focused test of that hypothesis. We obtain a dataset in order to test a focal hypothesis in a pre- determined way. 6
Exploratory vs. Confirmatory Research. Both are essential. Exploratory research: Explore ( get to know ) your biological system: Correlations? Which dependent variables seem to respond to independent variables? Anything unusual or surprising? What appears to be happening? Generate new hypotheses Confirmatory research: A focused test of that hypothesis. We obtain a dataset in order to test a focal hypothesis in a pre- determined way. Problems arise when an exploratory analysis is confused for, or explained as, a confirmatory hypothesis. 7
Exploratory Confirmatory 8
HARKing: Why a problem? (At least) Two, non-mutually exclusive problems: Problem 1: - In an exploratory analysis, our interaction with the data will generate hypotheses. - Generally, generating hypothesis is useful. But we may generate and test many more hypotheses than we would have if we had a single hypothesis in mind before looking at the data. - Testing multiple hypotheses increases opportunities for False Positives (Type 1 error). - If we present an exploratory result as confirmatory, we hide the fact that multiple hypotheses were investigated (and associated risks). 9
HARKing: Why a problem? (At least) Two, non-mutually exclusive problems: Problem 2: - As we look at the data to find interesting patterns we use the data to generate hypotheses. - At this point, the data are not longer appropriate to test the hypothesis that was generated by these data. Using the data to test a hypothesis generated by the data involves circular reasoning. - We need a fresh dataset to test the hypothesis that was generated by exploring the data. - If we partake in this circular reasoning (and do not obtain a fresh dataset), this process inflates type 1 errors and p-values cannot be trusted. 10
How common is HARKing? n = 573 Ecologists, 299 Evolutionary biologists Q3: Reporting an unexpected finding or a result from exploratory analysis as having been predicted from the start. 13
HARKing Reported having done at least once 14
HARKing 15
Why is HARKing relatively common? Training may not equip to avoid HARKing. Training might encourage HARKing. 16
How should we do science? - Only confirmatory? This video might give that impression, but no. - Confirmatory analyses are essential, but we need exploratory analyses, too. - It is vital to distinguish between the two. - Declare which analyses were planned a priori in a paper - Pre-registration provides evidence to support such claims. 17
