Dismissive Reviews, Citation Cartels, and the Replication Crisis
The study delves into the replication crisis in research, citing factors influencing research credibility such as sample size, effect size, and financial interests. It highlights issues like gaming the system, P-hacking, and data sharing reluctance. The growth of Metascience as a response to these challenges is depicted through the establishment of various scholarly platforms. The escalating information proliferation poses additional challenges to scholarly integrity and credibility.
- Replication Crisis
- Research Credibility
- Metascience Growth
- Scholarly Integrity
- Information Proliferation
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dismissive Reviews, Citation Cartels, and the Replication Crisis Richard P. Phelps
According to John Ioannidis research findings are less likely to be true, the smaller the sample size. smaller the effect. greater the number of tested relationships. greater the flexibility in design, definitions, outcomes, and analytical modes. greater the financial and other interests and prejudices. hotter the scientific field. greater the publicity.
Ioannidis exaggerated, but much of what he wrote was true Other researchers found gaming the system to be widespread P-hacking Only requiring P < .05 Non-replicable findings Many scholars refusing to share data sets I add 2 more: Dismissive Reviews, Citation Cartels
Proportion of studies replicated at p < .05, in three fields of study, 2015 to 2018 1 0 Nature/Science Economics Psychology NOT Replicated Replicated
Number of citations to replicated and not replicated studies, as of 2021
The growth of Metascience as a field of study Research Policy (journal), 1971 Scientometrics (journal), 1978 Cochrane Library, 1993 Metascience (journal), 1996 Campbell Collaboration, 2000 Research Ethics (journal), 2005 Centre for Journalology, 2005 Journal of Informetrics, 2007 Retraction Watch, 2010 PubPeer, 2012 Science Advances (journal), 2015 Research Waste/EQUATOR, 2015 Research Integrity and Peer Review (journal), 2016
Information proliferation Pro-Quest UMI dissertation publishing: 3 + 5 million dissertations and theses 250,000 new works each year STM reports: 34,000 journals +4%/year 10,000 publishers
Still more proliferation As of 2021 3 million articles are published in peer- reviewed journals every year. More than 5 new articles per minute.
The proliferation of researchers Residing in the United States alone (2008): 2.5 million with doctoral degrees 5.5 million with professional degrees 14.9 million with masters degrees
Knowing ALL the research literature on a topic There is so much, is anyone qualified to speak for all of it? It is genuinely difficult to do something new and unique
Knowledge is Unlimited? It may be, but there are limits to the amount that we can use. So, we filter it. Two ways to filter: Summarize all of it Accept only a certain amount, a certain type, or only from certain people
The World Wide Web is an information filter 20th Century: Reference librarians provides wide range of information 21st Century: Search Engine Optimization (SEO) narrows range
Definitions: Firstness Claims & Dismissive Reviews With a firstness claim, a researcher insists that s/he is the first to study a topic. With a dismissive literature review, a researcher assures the reader that no one else has conducted a study on a topic.
The Effect of Firstness Claims and Dismissive Reviews The public is told that no other research exists on a topic, ergo, there is no reason to look for it.
How difficult is a literature review? Not analytically difficult But a genuinely thorough review requires a substantial amount of time, and some money Generally, neither search nor ethics are part of US professors training.
Professional incentives to do a thorough literature review THERE ARE NONE? Scholars get little credit for a thorough literature review, much more for original work In publish or perish environments, lit reviews are impediments to progress
Why do a thorough lit review? huge burden in time and distraction little to no benefit professionally no punishment for not doing it
Dismissive Reviews and Citation Cartels in Education Policy Research A List https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm A database of > 1,000 dismissive reviews from top US scholars in education policy 3 types: Firstness Claims Dismissals Denigrations
Examples of Firstness Claims This report is unique. This is the only study to We provide the first evidence We construct the first nationwide measures of Our study is the first to precisely estimate We are the first to isolate the impact of Ours is the first comprehensive analysis of To our knowledge, this has not been studied before
Examples of Dismissals There is very little research Surprisingly little is known about The debate is mostly supported by anecdotes. Scant empirical evidence has been provided. Studies of are only relatively recent. Knowledge of has remained nonexistent. There has been little systematic effort to study
Examples of Denigrations Previous work is very limited To provide more rigorous evidence, we Existing studies focus on a very small number of . [we] found the bulk of studies to be flawed significant measurement problems in previous studies. [earlier] studies were unable to control for .... [earlier] studies had major deficiencies There are statistical shortcomings in the studies.
absence of evidence is not evidence of absence Eyewitness fallacy what you see represents all Composition fallacy the whole must have a property because its parts have the property. Amazing Familiarity the speaker seems to have information that there is no possible way for him to get Argumentum ad ignorantiam a proposition is true if it has not been proven false, or false if not proven true. Wishful Thinking things are (or will be) some way because that is how we wish them to be.
Key Characteristics of Dismissive Reviews Raw Declaration No evidence of a literature search provided Dismissed Work Not Identified Providing no help in finding it Discouraging search for it Avoiding debate
Search engine counts (Google Scholar), 9.5.2023 absence of research ~33,000 absence of studies ~26,000 this is the first study ~1,610,000 little research ~931,000 paucity of research ~121,000
Dismissive reviews worse than plagiarism Misrepresent the work of one person (by plagiarizing) Misrepresent the work of hundreds (in dismissive reviews) reward is small reward is large (for being first & unopposed) (saves some work & time) risk is large risk is nil (could ruin one s reputation and career)
Ethics of dismissive reviews Whatever you allow, you encourage. Michael Josephson
Definition: Citation Cartels In a Citation Cartel, scholars cite and reference each other and ignore, dismiss, or denigrate other research.
Definition: Sincere Scholarship Sincere scholars behave ethically: Conduct forthright and thorough literature reviews, Cite all other, relevant research regardless of: their own personal opinions of the individual scholars their own personal preferences for their research results. They provide full and accurate citations of all that previous work, so that readers will have no trouble finding it. Focus is on accuracy and adding knowledge
Definition: Strategic Scholarship Strategic scholars behave strategically: Avoid literature searches and reviews Either: Declare firstness no previous research Reference only work from within their cartel Do not pass up opportunities to promote their work even if they lack expertise on a topic, they will say something Focus is on advancing their careers
Comparing Strategic and Sincere Scholarship: Citation accumulation over thirty years 50,000 Strategic Scholars 40,000 Sincere Scholars 30,000 20,000 10,000 0 0 5 10 15 20 25 30 Years
How can they do this? No legal restrictions on education research quality no Hippocratic Oath, no license or certification Rewards still lie within one s primary field economics for economists, psychology for psychologists, etc. Scholars are rewarded for what they add; but they are not punished for what they subtract Many who could blow the whistle are afraid, or they might dislike the research literature and like dismissing it
Journalists help to suppress information When they print one researcher s firstness claim or dismissive review, they help to suppress others work and competing evidence
Benefits accrue to individual scholars and small groups: * saves time, avoids tedium of reading the research literature * adds to own citation totals, or those of one s citation cartel, while not adding to rivals citation totals * gives readers no help in finding rival evidence (by not even citing it) * establishes (false) bona fides as an expert on the topic * attention by allegedly being "first," "original," "a pioneer." * increases the likelihood of press coverage for the same reason. * increases prospects for grant funding to "fill knowledge gaps."
Costs accrue to society Society loses information; remaining information is skewed in favor of the powerful Policy decisions are based on information that is limited and skewed Government and foundations may pay again for research that has already been done
Research most vulnerable to dismissal That done by those below the celebrity threshold * Studies by civil servants (government agencies do not promote or defend their work) That done by the deceased All become: Zombie Researchers * Researchers below the celebrity threshold lack the resources and media access to successfully counter dismissals of their work they can easily be ignored.
Paradox of research proliferation As the amount of research grows so does the amount declared nonexistent so does the incentive to dismiss it so does the opportunity to dismiss it
Transparency Too easy to publish dismissive reviews: Find 1 editor and 1-2 reviewers among the hundreds of thousands of journals available If first journal one tries objects, there are many others. Just submit same paperwork to another journal. A genuine lit review takes weeks or months. Preprints and PubPeer are the only current, effective deterrents
Ban firstness claims and dismissive reviews Add ban to the ethics codes of journalists foundation research funders government research funders In most cases, editors, reviewers, & journalists have neither the time nor the resources to verify
Real punishment for false firstness claims and dismissive reviews Make literature reviews optional for getting funding, but make their accuracy mandatory, suspend violators from any further funding
Remove any literature review obligation from research articles Removes temptation Most do more harm than good anyway because they are partial and selective
Is meta-analysis the solution? Let meta-analysts do all literature reviews Meta-analysis review model: Identify where you have looked before making summary claims
