Enhancing Peer Review Quality Through Double-Blind Reviewing in ACM Conferences

 
Considering a Mandate for
Double-Blind Reviewing
in ACM Conferences
 
Jonathan Aldrich, Carnegie Mellon University
SGB Liason to the ACM Publications Board
 
Double-Blind Reviewing (DBR) shields author
identities from reviewers
 
Single-Blind Reviewing (SBR)
Reviewer identities hidden from authors
Author identities 
visible 
to reviewers
 
Double-Blind Reviewing (DBR):
Reviewer identities hidden from authors
Author identities 
hidden 
from reviewers
 
There is bias in single-blind reviewing
[Snodgrass, SIGMOD 2006][Tomkins et al., PNAS 2017]
 
Clear literature review findings
Evidence of gender bias
Evidence of pro-US institution bias
Perception that DBR is more fair
 
Additional mixed/surprising results (not always in expected direction)
Bias related to prolific authors
Bias related to institution quality
 
Regardless of the source / kind of bias, DBR can reduce it
 
DBR Improves Quality
 
Articles published in journals using blinded peer review were cited
significantly more than articles published in journals using non-
blinded peer review, controlling for a variety of author, article, and
journal attributes. 
[Laband and Piette, JAMA 1994]
 
DBR reviewers rarely guess author identity
 
Across 3 conferences, reviewers were asked to guess author identity
74-90% of reviews were submitted with no correct guesses
[Le Goues et al., CACM 2018]
 
DBR is practical
 
PC chairs did not report the extra administrative burden was large
[Le Goues et al., CACM 2018]
 
Most current ACM conferences use DBR
80% of those that reported (117 of 146)
Thank you for contributing to this data:
https://docs.google.com/spreadsheets/d/1XqN4PIUYDJAeWoqIzhXQodwJx8b
QfVs9cS31vK0MQe0/edit
 
DBR has variations with tradeoffs
[terminology from double-blind.org]
 
When are author identities revealed?
Partially double blind: author identities revealed after initial review
Allows authors to be considered in discussion, e.g. with respect to prior work
Fully double blind (blind-to-accept): author identities hidden until (cond.) accept
Stronger protection against bias
Most double-blind ACM conferences are blind-to-accept (93 of 117)
Some started partially double-blind, and moved to fully double-blind
 
arXiv-restricted
Submissions to arXiv are restricted for a period before/during review
Can help with blinding, as reviewers may be notified of preprint submissions
Relatively uncommon in ACM conferences (~3 of 127)
Pubs Board encourages arXiv preprints in general, so a total ban is counterproductive
 
Should ACM mandate DBR?
 
Some other publishers are doing it, e.g. IOP in Physics
https://ioppublishing.org/news/iop-publishing-commits-to-adopting-double-
blind-peer-review-for-all-journals/
Some publishers are going the other way, towards Open Peer Review
 
Let’s discuss
 
Some Resources on Double Blind Review
 
https://double-blind.org/
Tracks double-blind reviewing in top CS conferences
Includes resources on DBR (at the bottom of the page)
https://www.cs.utexas.edu/users/mckinley/notes/blind.html
Kathryn McKinley’s site on improving reviewing quality through DBR
Also includes many useful DBR resources
Two resources advocating for double-blind in SE conferences
https://www.cs.cmu.edu/~clegoues/double-blind.html
https://people.cs.umass.edu/~brun/doubleblind.html
 
Slide Note
Embed
Share

Double-blind reviewing (DBR) is increasingly recognized for its effectiveness in reducing biases, improving article quality, and practicality in ACM conferences. Studies show evidence of gender and institutional biases in single-blind reviewing, while DBR enhances fairness and credibility. DBR reviewers rarely guess author identities, contributing to the integrity of the review process. The variations of DBR offer trade-offs in when author identities are revealed.


Uploaded on Jul 11, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Considering a Mandate for Double-Blind Reviewing in ACM Conferences Jonathan Aldrich, Carnegie Mellon University SGB Liason to the ACM Publications Board

  2. Double-Blind Reviewing (DBR) shields author identities from reviewers Single-Blind Reviewing (SBR) Reviewer identities hidden from authors Author identities visible to reviewers Double-Blind Reviewing (DBR): Reviewer identities hidden from authors Author identities hidden from reviewers

  3. There is bias in single-blind reviewing [Snodgrass, SIGMOD 2006][Tomkins et al., PNAS 2017] Clear literature review findings Evidence of gender bias Evidence of pro-US institution bias Perception that DBR is more fair Additional mixed/surprising results (not always in expected direction) Bias related to prolific authors Bias related to institution quality Regardless of the source / kind of bias, DBR can reduce it

  4. DBR Improves Quality Articles published in journals using blinded peer review were cited significantly more than articles published in journals using non- blinded peer review, controlling for a variety of author, article, and journal attributes. [Laband and Piette, JAMA 1994]

  5. DBR reviewers rarely guess author identity Across 3 conferences, reviewers were asked to guess author identity 74-90% of reviews were submitted with no correct guesses [Le Goues et al., CACM 2018]

  6. DBR is practical PC chairs did not report the extra administrative burden was large [Le Goues et al., CACM 2018] Most current ACM conferences use DBR 80% of those that reported (117 of 146) Thank you for contributing to this data: https://docs.google.com/spreadsheets/d/1XqN4PIUYDJAeWoqIzhXQodwJx8b QfVs9cS31vK0MQe0/edit

  7. DBR has variations with tradeoffs [terminology from double-blind.org] When are author identities revealed? Partially double blind: author identities revealed after initial review Allows authors to be considered in discussion, e.g. with respect to prior work Fully double blind (blind-to-accept): author identities hidden until (cond.) accept Stronger protection against bias Most double-blind ACM conferences are blind-to-accept (93 of 117) Some started partially double-blind, and moved to fully double-blind arXiv-restricted Submissions to arXiv are restricted for a period before/during review Can help with blinding, as reviewers may be notified of preprint submissions Relatively uncommon in ACM conferences (~3 of 127) Pubs Board encourages arXiv preprints in general, so a total ban is counterproductive

  8. Should ACM mandate DBR? Some other publishers are doing it, e.g. IOP in Physics https://ioppublishing.org/news/iop-publishing-commits-to-adopting-double- blind-peer-review-for-all-journals/ Some publishers are going the other way, towards Open Peer Review Let s discuss

  9. Some Resources on Double Blind Review https://double-blind.org/ Tracks double-blind reviewing in top CS conferences Includes resources on DBR (at the bottom of the page) https://www.cs.utexas.edu/users/mckinley/notes/blind.html Kathryn McKinley s site on improving reviewing quality through DBR Also includes many useful DBR resources Two resources advocating for double-blind in SE conferences https://www.cs.cmu.edu/~clegoues/double-blind.html https://people.cs.umass.edu/~brun/doubleblind.html

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#