Dismissive Reviews, Citation Cartels, and the Replication Crisis
Dismissive Reviews, Citation Cartels, and
the Replication Crisis
Richard P. Phelps
The Replication Crisis
… smaller the sample size.
… smaller the effect.
… greater the number of tested relationships.
… greater the flexibility in design, definitions, outcomes,
and
analytical modes.
… greater the financial and other interests and prejudices.
… hotter the scientific field.
… greater the publicity.
According to John Ioannidis …
… research findings are less likely to be true, the …
Other researchers found “gaming the system” to be
widespread
“P-hacking”
Only requiring P < .05
Non-replicable findings
Many scholars refusing to share data sets
I add 2 more: Dismissive Reviews, Citation Cartels
Ioannidis exaggerated,
but much of what he wrote was true
Proportion of studies replicated at p < .05,
in three fields of study, 2015 to 2018
Number of citations to replicated and not replicated studies, as of 2021
The growth of “Metascience” as a
field of study
Research Policy
(journal), 1971
Scientometrics
(journal), 1978
Cochrane Library, 1993
Metascience
(journal), 1996
Campbell Collaboration, 2000
Research Ethics
(journal), 2005
Centre for Journalology, 2005
Journal of Informetrics
, 2007
Retraction Watch, 2010
PubPeer, 2012
Science Advances
(journal), 2015
Research Waste/EQUATOR, 2015
Research Integrity and Peer Review
(journal), 2016
Information proliferation
•
Pro-Quest UMI dissertation publishing:
–
3 + 5 million dissertations and theses
–
250,000 new works each year
•
STM reports
:
–
34,000 journals
•
+4%/year
–
10,000 publishers
Still more proliferation
As of 2021…
…3 million articles are
published in peer-
reviewed journals every
year.
More than 5 new articles
per minute.
The proliferation of researchers
•
Residing in the United
States alone (2008):
–
2.5 million with doctoral
degrees
–
5.5 million with professional
degrees
–
14.9 million with masters’
degrees
Knowing ALL the
research literature on a
topic
•
There is so much, is
anyone qualified to speak
for all of it?
•
It is genuinely difficult to do
something new and unique
Knowledge is Unlimited?
•
It may be, but there are limits to the
amount that we can use.
•
So, we
filter
it.
Two ways to filter:
Summarize all of it
Accept only a certain amount, a
certain type,
…or only from
certain people
The World Wide Web is an
information filter
20
th
Century:
Reference librarians provides wide range of information
21
st
Century:
Search Engine Optimization (SEO) narrows range
Definitions: Firstness Claims &
Dismissive Reviews
•
With a firstness claim, a researcher insists
that s/he is the first to study a topic.
•
With a dismissive literature review, a
researcher assures the reader that no one
else has conducted a study on a topic.
The Effect
of Firstness
Claims and
Dismissive
Reviews
The public is told that no other
research exists on a topic, ergo,
there is no reason to look for it.
How difficult is a literature review?
•
Not analytically difficult
•
But a
genuinely thorough
review requires a
substantial amount of
time, and some money
•
Generally, neither search
nor ethics are part of US
professors’ training.
Professional incentives to do a
thorough literature review
THERE ARE NONE?
•
Scholars get little credit for a
thorough literature review,
much more for
“
original work
”
•
In
“
publish or perish
”
environments, lit reviews are
impediments to progress
Why do a thorough lit review?
•
huge burden in time and distraction
•
little to no benefit professionally
•
no punishment for not doing it
Dismissive Reviews and Citation Cartels in
Education Policy Research – A List
https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm
A database of > 1,000 dismissive reviews from top
US scholars in education policy
3 types:
•
Firstness Claims
•
Dismissals
•
Denigrations
Examples of Firstness Claims
“This report is unique.”
“This is the only study to …”
“We provide the first evidence …”
“We construct the first nationwide measures of …”
“Our study is the first to precisely estimate …”
“We are the first to isolate the impact of …”
“Ours is the first comprehensive analysis of …”
“To our knowledge, this has not been studied before …”
Examples of Dismissals
“There is very little research …”
“Surprisingly little is known about …”
“The debate is mostly supported by anecdotes.”
“Scant empirical evidence has been provided.”
“Studies of … are only relatively recent.”
“Knowledge of … has remained nonexistent.”
“There has been little systematic effort to study …”
Examples of Denigrations
“Previous work is very limited …”
“To provide more rigorous evidence, we …”
“Existing studies focus on a very small number of ….”
“[we] found the bulk of studies to be flawed …”
“… significant measurement problems in previous studies.”
“[earlier] studies were unable to control for ....”
“[earlier] studies had major deficiencies …”
“There are statistical shortcomings in … the studies.”
“absence of evidence is not
evidence of absence”
•
E
y
e
w
i
t
n
e
s
s
f
a
l
l
a
c
y
–
w
h
a
t
y
o
u
s
e
e
r
e
p
r
e
s
e
n
t
s
a
l
l
•
C
o
m
p
o
s
i
t
i
o
n
f
a
l
l
a
c
y
–
t
h
e
w
h
o
l
e
m
u
s
t
h
a
v
e
a
p
r
o
p
e
r
t
y
b
e
c
a
u
s
e
i
t
s
p
a
r
t
s
h
a
v
e
t
h
e
p
r
o
p
e
r
t
y
.
•
A
m
a
z
i
n
g
F
a
m
i
l
i
a
r
i
t
y
–
t
h
e
s
p
e
a
k
e
r
s
e
e
m
s
t
o
h
a
v
e
i
n
f
o
r
m
a
t
i
o
n
t
h
a
t
t
h
e
r
e
i
s
n
o
p
o
s
s
i
b
l
e
w
a
y
f
o
r
h
i
m
t
o
g
e
t
•
A
r
g
u
m
e
n
t
u
m
a
d
i
g
n
o
r
a
n
t
i
a
m
–
a
p
r
o
p
o
s
i
t
i
o
n
i
s
t
r
u
e
i
f
i
t
h
a
s
n
o
t
b
e
e
n
p
r
o
v
e
n
f
a
l
s
e
,
o
r
f
a
l
s
e
i
f
n
o
t
p
r
o
v
e
n
t
r
u
e
.
•
W
i
s
h
f
u
l
T
h
i
n
k
i
n
g
–
t
h
i
n
g
s
a
r
e
(
o
r
w
i
l
l
b
e
)
s
o
m
e
w
a
y
b
e
c
a
u
s
e
t
h
a
t
i
s
h
o
w
w
e
w
i
s
h
t
h
e
m
t
o
b
e
.
R
a
w
D
e
c
l
a
r
a
t
i
o
n
No evidence of a literature search provided
D
i
s
m
i
s
s
e
d
W
o
r
k
N
o
t
I
d
e
n
t
i
f
i
e
d
Providing no help in finding it
Discouraging search for it
Avoiding debate
Key Characteristics of Dismissive Reviews
“absence of research” ~33,000
“absence of studies” ~26,000
“this is the first study” ~1,610,000
“little research” ~931,000
“paucity of research” ~121,000
S
earch engine counts (Google
Scholar), 9
.5.2023
Dismissive reviews worse than plagiarism
Misrepresent the work
of
one
person
(by plagiarizing)
•
reward
is small
(saves some work & time)
•
risk
is large
(could ruin one
’
s reputation
and career)
Misrepresent the work
of
hundreds
(in
dismissive reviews)
•
reward
is large
(for being first & unopposed)
•
risk
is nil
Ethics of
dismissive reviews
“
Whatever you allow, you
encourage.
”
–
Michael Josephson
Definition: Citation Cartels
In a Citation Cartel, scholars…
…
cite and reference each other and
…
ignore, dismiss, or denigrate other
research.
Definition: “Sincere” Scholarship
Sincere
scholars behave
ethically
:
C
onduct forthright and thorough literature reviews,
C
ite all other, relevant research regardless of:
their own personal opinions of the individual scholars
their own personal preferences for their research results.
They provide full and accurate citations of all that previous work,
so that readers will have no trouble finding it.
Focus is on accuracy and adding knowledge
Definition: “Strategic” Scholarship
Strategic
scholars behave
strategically
:
Avoid literature searches and reviews
Either:
Declare ”firstness” – no previous research
Reference only work from within their cartel
Do not pass up opportunities to promote their work…
…even if they lack expertise on a topic, they will say
something
Focus is on advancing their careers
Comparing Strategic and Sincere Scholarship:
Citation accumulation over thirty years
How can they do this?
•
No legal restrictions on education research quality –
•
no Hippocratic Oath, no license or certification
•
Rewards still lie within one’s primary field –
•
economics for economists, psychology for
psychologists, etc.
•
Scholars are rewarded for what they add; but they are
not punished for what they subtract
•
Many who could “blow the whistle” are afraid, or they
might dislike the research literature and like dismissing it
Journalists help to suppress information
When they print one
researcher
’
s firstness
claim or dismissive
review,
they help to suppress
others
’
work and
competing evidence
* saves time, avoids tedium of reading the research literature
* adds to own citation totals, or those of one’s citation cartel,
… while not adding to rivals’ citation totals
* gives readers no help in finding rival evidence (by not even
citing it)
* establishes (false) bona fides as an “expert” on the topic
* attention by allegedly being "first," "original," "a pioneer."
* increases the likelihood of press coverage for the same
reason.
* increases prospects for grant funding to "fill knowledge gaps."
Benefits accrue to…
…individual scholars
and small groups:
Costs accrue
to… society
•
Society loses information; remaining information
is skewed in favor of the powerful
•
Policy decisions are based on information that is
limited and skewed
•
Government and foundations may pay again for
research that has already been done
Research most vulnerable to dismissal
•
That done by those below the
“
celebrity threshold
”
*
•
Studies by civil servants
(government agencies do not
promote or defend their work)
•
That done by the deceased
All become:
“
Zombie Researchers
”
* Researchers below the celebrity threshold lack the resources and media access to
successfully counter dismissals of their work – they can easily be ignored.
Paradox of
research
proliferation
As the amount of research grows…
…so does the amount declared nonexistent
…so does the incentive to dismiss it
…so does the opportunity to dismiss it
What Can be Done?
Transparency
Too easy to publish dismissive reviews:
•
Find 1 editor and 1-2 reviewers among the
hundreds of thousands of journals available
•
If first journal one tries objects, there are
many others.
•
Just submit same paperwork to another
journal. A genuine lit review takes weeks
or months.
P
r
e
p
r
i
n
t
s
a
n
d
P
u
b
P
e
e
r
a
r
e
t
h
e
o
n
l
y
c
u
r
r
e
n
t
,
e
f
f
e
c
t
i
v
e
d
e
t
e
r
r
e
n
t
s
Ban firstness claims
and dismissive reviews
Add ban to the ethics codes of…
…journalists
…foundation research funders
…government research funders
In most cases, editors, reviewers, & journalists
have neither the time nor the resources to verify
Real punishment for false firstness
claims and dismissive reviews
Make literature reviews optional
for getting funding, but…
…make their accuracy mandatory,
…suspend violators from any
further funding
Remove any literature review
obligation from research articles
•
Removes temptation
•
Most do more harm than
good anyway because
they are partial and
selective
Is
meta-analysis the solution?
•
Let meta-analysts do all literature
reviews
•
Meta-analysis review model:
Identify where you have looked
before making summary claims
Merci !
The study delves into the replication crisis in research, citing factors influencing research credibility such as sample size, effect size, and financial interests. It highlights issues like gaming the system, P-hacking, and data sharing reluctance. The growth of Metascience as a response to these challenges is depicted through the establishment of various scholarly platforms. The escalating information proliferation poses additional challenges to scholarly integrity and credibility.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dismissive Reviews, Citation Cartels, and the Replication Crisis Richard P. Phelps
According to John Ioannidis research findings are less likely to be true, the smaller the sample size. smaller the effect. greater the number of tested relationships. greater the flexibility in design, definitions, outcomes, and analytical modes. greater the financial and other interests and prejudices. hotter the scientific field. greater the publicity.
Ioannidis exaggerated, but much of what he wrote was true Other researchers found gaming the system to be widespread P-hacking Only requiring P < .05 Non-replicable findings Many scholars refusing to share data sets I add 2 more: Dismissive Reviews, Citation Cartels
Proportion of studies replicated at p < .05, in three fields of study, 2015 to 2018 1 0 Nature/Science Economics Psychology NOT Replicated Replicated
Number of citations to replicated and not replicated studies, as of 2021
The growth of Metascience as a field of study Research Policy (journal), 1971 Scientometrics (journal), 1978 Cochrane Library, 1993 Metascience (journal), 1996 Campbell Collaboration, 2000 Research Ethics (journal), 2005 Centre for Journalology, 2005 Journal of Informetrics, 2007 Retraction Watch, 2010 PubPeer, 2012 Science Advances (journal), 2015 Research Waste/EQUATOR, 2015 Research Integrity and Peer Review (journal), 2016
Information proliferation Pro-Quest UMI dissertation publishing: 3 + 5 million dissertations and theses 250,000 new works each year STM reports: 34,000 journals +4%/year 10,000 publishers
Still more proliferation As of 2021 3 million articles are published in peer- reviewed journals every year. More than 5 new articles per minute.
The proliferation of researchers Residing in the United States alone (2008): 2.5 million with doctoral degrees 5.5 million with professional degrees 14.9 million with masters degrees
Knowing ALL the research literature on a topic There is so much, is anyone qualified to speak for all of it? It is genuinely difficult to do something new and unique
Knowledge is Unlimited? It may be, but there are limits to the amount that we can use. So, we filter it. Two ways to filter: Summarize all of it Accept only a certain amount, a certain type, or only from certain people
The World Wide Web is an information filter 20th Century: Reference librarians provides wide range of information 21st Century: Search Engine Optimization (SEO) narrows range
Definitions: Firstness Claims & Dismissive Reviews With a firstness claim, a researcher insists that s/he is the first to study a topic. With a dismissive literature review, a researcher assures the reader that no one else has conducted a study on a topic.
The Effect of Firstness Claims and Dismissive Reviews The public is told that no other research exists on a topic, ergo, there is no reason to look for it.
How difficult is a literature review? Not analytically difficult But a genuinely thorough review requires a substantial amount of time, and some money Generally, neither search nor ethics are part of US professors training.
Professional incentives to do a thorough literature review THERE ARE NONE? Scholars get little credit for a thorough literature review, much more for original work In publish or perish environments, lit reviews are impediments to progress
Why do a thorough lit review? huge burden in time and distraction little to no benefit professionally no punishment for not doing it
Dismissive Reviews and Citation Cartels in Education Policy Research A List https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm A database of > 1,000 dismissive reviews from top US scholars in education policy 3 types: Firstness Claims Dismissals Denigrations
Examples of Firstness Claims This report is unique. This is the only study to We provide the first evidence We construct the first nationwide measures of Our study is the first to precisely estimate We are the first to isolate the impact of Ours is the first comprehensive analysis of To our knowledge, this has not been studied before
Examples of Dismissals There is very little research Surprisingly little is known about The debate is mostly supported by anecdotes. Scant empirical evidence has been provided. Studies of are only relatively recent. Knowledge of has remained nonexistent. There has been little systematic effort to study
Examples of Denigrations Previous work is very limited To provide more rigorous evidence, we Existing studies focus on a very small number of . [we] found the bulk of studies to be flawed significant measurement problems in previous studies. [earlier] studies were unable to control for .... [earlier] studies had major deficiencies There are statistical shortcomings in the studies.
absence of evidence is not evidence of absence Eyewitness fallacy what you see represents all Composition fallacy the whole must have a property because its parts have the property. Amazing Familiarity the speaker seems to have information that there is no possible way for him to get Argumentum ad ignorantiam a proposition is true if it has not been proven false, or false if not proven true. Wishful Thinking things are (or will be) some way because that is how we wish them to be.
Key Characteristics of Dismissive Reviews Raw Declaration No evidence of a literature search provided Dismissed Work Not Identified Providing no help in finding it Discouraging search for it Avoiding debate
Search engine counts (Google Scholar), 9.5.2023 absence of research ~33,000 absence of studies ~26,000 this is the first study ~1,610,000 little research ~931,000 paucity of research ~121,000
Dismissive reviews worse than plagiarism Misrepresent the work of one person (by plagiarizing) Misrepresent the work of hundreds (in dismissive reviews) reward is small reward is large (for being first & unopposed) (saves some work & time) risk is large risk is nil (could ruin one s reputation and career)
Ethics of dismissive reviews Whatever you allow, you encourage. Michael Josephson
Definition: Citation Cartels In a Citation Cartel, scholars cite and reference each other and ignore, dismiss, or denigrate other research.
Definition: Sincere Scholarship Sincere scholars behave ethically: Conduct forthright and thorough literature reviews, Cite all other, relevant research regardless of: their own personal opinions of the individual scholars their own personal preferences for their research results. They provide full and accurate citations of all that previous work, so that readers will have no trouble finding it. Focus is on accuracy and adding knowledge
Definition: Strategic Scholarship Strategic scholars behave strategically: Avoid literature searches and reviews Either: Declare firstness no previous research Reference only work from within their cartel Do not pass up opportunities to promote their work even if they lack expertise on a topic, they will say something Focus is on advancing their careers
Comparing Strategic and Sincere Scholarship: Citation accumulation over thirty years 50,000 Strategic Scholars 40,000 Sincere Scholars 30,000 20,000 10,000 0 0 5 10 15 20 25 30 Years
How can they do this? No legal restrictions on education research quality no Hippocratic Oath, no license or certification Rewards still lie within one s primary field economics for economists, psychology for psychologists, etc. Scholars are rewarded for what they add; but they are not punished for what they subtract Many who could blow the whistle are afraid, or they might dislike the research literature and like dismissing it
Journalists help to suppress information When they print one researcher s firstness claim or dismissive review, they help to suppress others work and competing evidence
Benefits accrue to individual scholars and small groups: * saves time, avoids tedium of reading the research literature * adds to own citation totals, or those of one s citation cartel, while not adding to rivals citation totals * gives readers no help in finding rival evidence (by not even citing it) * establishes (false) bona fides as an expert on the topic * attention by allegedly being "first," "original," "a pioneer." * increases the likelihood of press coverage for the same reason. * increases prospects for grant funding to "fill knowledge gaps."
Costs accrue to society Society loses information; remaining information is skewed in favor of the powerful Policy decisions are based on information that is limited and skewed Government and foundations may pay again for research that has already been done
Research most vulnerable to dismissal That done by those below the celebrity threshold * Studies by civil servants (government agencies do not promote or defend their work) That done by the deceased All become: Zombie Researchers * Researchers below the celebrity threshold lack the resources and media access to successfully counter dismissals of their work they can easily be ignored.
Paradox of research proliferation As the amount of research grows so does the amount declared nonexistent so does the incentive to dismiss it so does the opportunity to dismiss it
Transparency Too easy to publish dismissive reviews: Find 1 editor and 1-2 reviewers among the hundreds of thousands of journals available If first journal one tries objects, there are many others. Just submit same paperwork to another journal. A genuine lit review takes weeks or months. Preprints and PubPeer are the only current, effective deterrents
Ban firstness claims and dismissive reviews Add ban to the ethics codes of journalists foundation research funders government research funders In most cases, editors, reviewers, & journalists have neither the time nor the resources to verify
Real punishment for false firstness claims and dismissive reviews Make literature reviews optional for getting funding, but make their accuracy mandatory, suspend violators from any further funding
Remove any literature review obligation from research articles Removes temptation Most do more harm than good anyway because they are partial and selective
Is meta-analysis the solution? Let meta-analysts do all literature reviews Meta-analysis review model: Identify where you have looked before making summary claims
