On Content-Oblivious Measures of Research
On Content-Oblivious
Measures of Research
Oded Goldreich
Weizmann Institute od Science
Definition and Types
A
content-oblivious measure
(of scientific quality) is obtained by applying
statistical tools on bibliographic data that refers to scientific works,
where the latter data refers to the identification of the work,
its co-authors, its date of publication, and the publication venue.
The source of this data is the
recognized publication venue
itself
or the bibliographic list that appears in a publication in such a venue.
•
Public and universal measures (or rather their usage): The numerical
value is obtained by using a universal on-line service (Google Scholar,
Web-of-Science, etc.) and is used as a final value (w.o. calibration).
•
Private and specific measures (or rather their usage): The numerical
value is obtained manually and is calibrated by the users for the specific
use by comparison to
norms in the specific discipline
.
The term
content-oblivious
indicates the absolute lack of reference to the actual
contents of the scientific work. Only its “bibliographic signature” counts.
The fantasy of objectivity
and the hidden controversial decisions
•
Controversial decisions underlying the bibliographic data (work & citation)
- Which types of publication venues are recognized?
E.g., books, journals, conferences, chapters, TRs, electronic archives.
- Which of these publication venues are recognized?
E.g., new venues wait for recognition. N.B.: Control by publishers!
•
Controversial decisions regarding the analysis of data
- How to weigh
different venues
(e.g., Impact Factor or “reputation”).
- How to weigh
different publications
in same venue (incl. over
time
).
- How to combine
different counts
(e.g., the H-index, the Qs).
•
Subjective decisions underlying the individual data items
- Why is a work accepted? Who decides that?
- Why is a work cited (i.e., instrumental, convenience, properness, mistake)?
•
When using the measure we typically do not know the answers.
•
Different measures typically given different ranking
(e.g., “recent”, the H-index formula, Google Scholar vs Web of Science [Thom. Reut.] vs Scopus [Els.]).
“recent”
period
Using unreliable measures yield unsound decisions
•
Different disciplines have different publication cultures,
which yield different norms and expectations
(for different application types).
At best, we have to get calibrated by an expert who we trust!
•
Scientific works are extremely diverse,
even those published in the same venue at the same time.
Counting them (resp., the citations they make) as equal is absurd.
•
The “opinion” (i.e., count) of a measure is not articulated,
and cannot be challenged, rebutted, discussed, etc.
If preferred over a
reliable expert
opinion, it leads to mistakes.
•
When using the measure we do not know crucial underlying decisions.
•
Different measures typically given different
ranking
.
“Beware of the temptation of objectivity.”
Indirect (“collateral”) damage caused by the measures
•
From a focus on research,
where
publications are means
of dissemination its results,
to a focus on publications with
research as means
towards them.
(To see it, listen to the “corridor” discussions.)
•
From a meaningful communal project to a mice race,
from collaboration to adversarial competition
(or “too fierce” competition)
.
•
Manipulations (of all sorts: in research itself as well as in publication),
rewarding the “Kings Road” (and “conformism”)
and discouraging innovative explorations.
A proxy that was never tested, and never calibrated
•
Stress: It was never tested!
•
How to test it?
(An infeasible project: Learn and train using reliable experts per each discipline.)
“Additional information never hurts”
except if it makes you disregard more reliable information.
What do I suggest
•
Declare its use as
bad manners
. Make this the norm.
•
Ideally, I’d disallow it altogether…
Caveat: I’m aware that we do not have control on what people do,
even at Weizmann, let alone outside Weizmann.
Furthermore, the use of contents-oblivious measures is aligned
with the Zeitgeist in academia and in society at large.
Still, we can try to do our best.
June 2021, WIS
A content-oblivious measure of scientific quality involves statistical analysis of bibliographic data without reference to actual content. This method raises issues of objectivity, controversial decisions, and the limitations of using numerical values for evaluating research.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
On Content-Oblivious Measures of Research Oded Goldreich Weizmann Institute od Science
Definition and Types A content-oblivious measure (of scientific quality) is obtained by applying statistical tools on bibliographic data that refers to scientific works, where the latter data refers to the identification of the work, its co-authors, its date of publication, and the publication venue. The source of this data is the recognized publication venue itself or the bibliographic list that appears in a publication in such a venue. Public and universal measures (or rather their usage): The numerical value is obtained by using a universal on-line service (Google Scholar, Web-of-Science, etc.) and is used as a final value (w.o. calibration). Private and specific measures (or rather their usage): The numerical value is obtained manually and is calibrated by the users for the specific use by comparison to norms in the specific discipline. The term content-oblivious indicates the absolute lack of reference to the actual contents of the scientific work. Only its bibliographic signature counts.
The fantasy of objectivity and the hidden controversial decisions Controversial decisions underlying the bibliographic data (work & citation) - Which types of publication venues are recognized? E.g., books, journals, conferences, chapters, TRs, electronic archives. - Which of these publication venues are recognized? E.g., new venues wait for recognition. N.B.: Control by publishers! Controversial decisions regarding the analysis of data - How to weigh different venues (e.g., Impact Factor or reputation ). - How to weigh different publications in same venue (incl. over time). - How to combine different counts (e.g., the H-index, the Qs). Subjective decisions underlying the individual data items - Why is a work accepted? Who decides that? - Why is a work cited (i.e., instrumental, convenience, properness, mistake)? recent period When using the measure we typically do not know the answers. Different measures typically given different ranking (e.g., recent , the H-index formula, Google Scholar vs Web of Science [Thom. Reut.] vs Scopus [Els.]).
Using unreliable measures yield unsound decisions When using the measure we do not know crucial underlying decisions. Different measures typically given different ranking. Different disciplines have different publication cultures, which yield different norms and expectations (for different application types). At best, we have to get calibrated by an expert who we trust! Scientific works are extremely diverse, even those published in the same venue at the same time. Counting them (resp., the citations they make) as equal is absurd. The opinion (i.e., count) of a measure is not articulated, and cannot be challenged, rebutted, discussed, etc. If preferred over a reliable expert opinion, it leads to mistakes. Beware of the temptation of objectivity.
Indirect (collateral) damage caused by the measures From a focus on research, where publications are means of dissemination its results, to a focus on publications with research as means towards them. (To see it, listen to the corridor discussions.) From a meaningful communal project to a mice race, from collaboration to adversarial competition (or too fierce competition). Manipulations (of all sorts: in research itself as well as in publication), rewarding the Kings Road (and conformism ) and discouraging innovative explorations.
A proxy that was never tested, and never calibrated Stress: It was never tested! How to test it? (An infeasible project: Learn and train using reliable experts per each discipline.) Additional information never hurts except if it makes you disregard more reliable information.
What do I suggest Declare its use as bad manners. Make this the norm. Ideally, I d disallow it altogether Caveat: I m aware that we do not have control on what people do, even at Weizmann, let alone outside Weizmann. Furthermore, the use of contents-oblivious measures is aligned with the Zeitgeist in academia and in society at large. Still, we can try to do our best.
