Enhancing Peer Review Quality Through Double-Blind Reviewing in ACM Conferences
Double-blind reviewing (DBR) is increasingly recognized for its effectiveness in reducing biases, improving article quality, and practicality in ACM conferences. Studies show evidence of gender and institutional biases in single-blind reviewing, while DBR enhances fairness and credibility. DBR reviewers rarely guess author identities, contributing to the integrity of the review process. The variations of DBR offer trade-offs in when author identities are revealed.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Considering a Mandate for Double-Blind Reviewing in ACM Conferences Jonathan Aldrich, Carnegie Mellon University SGB Liason to the ACM Publications Board
Double-Blind Reviewing (DBR) shields author identities from reviewers Single-Blind Reviewing (SBR) Reviewer identities hidden from authors Author identities visible to reviewers Double-Blind Reviewing (DBR): Reviewer identities hidden from authors Author identities hidden from reviewers
There is bias in single-blind reviewing [Snodgrass, SIGMOD 2006][Tomkins et al., PNAS 2017] Clear literature review findings Evidence of gender bias Evidence of pro-US institution bias Perception that DBR is more fair Additional mixed/surprising results (not always in expected direction) Bias related to prolific authors Bias related to institution quality Regardless of the source / kind of bias, DBR can reduce it
DBR Improves Quality Articles published in journals using blinded peer review were cited significantly more than articles published in journals using non- blinded peer review, controlling for a variety of author, article, and journal attributes. [Laband and Piette, JAMA 1994]
DBR reviewers rarely guess author identity Across 3 conferences, reviewers were asked to guess author identity 74-90% of reviews were submitted with no correct guesses [Le Goues et al., CACM 2018]
DBR is practical PC chairs did not report the extra administrative burden was large [Le Goues et al., CACM 2018] Most current ACM conferences use DBR 80% of those that reported (117 of 146) Thank you for contributing to this data: https://docs.google.com/spreadsheets/d/1XqN4PIUYDJAeWoqIzhXQodwJx8b QfVs9cS31vK0MQe0/edit
DBR has variations with tradeoffs [terminology from double-blind.org] When are author identities revealed? Partially double blind: author identities revealed after initial review Allows authors to be considered in discussion, e.g. with respect to prior work Fully double blind (blind-to-accept): author identities hidden until (cond.) accept Stronger protection against bias Most double-blind ACM conferences are blind-to-accept (93 of 117) Some started partially double-blind, and moved to fully double-blind arXiv-restricted Submissions to arXiv are restricted for a period before/during review Can help with blinding, as reviewers may be notified of preprint submissions Relatively uncommon in ACM conferences (~3 of 127) Pubs Board encourages arXiv preprints in general, so a total ban is counterproductive
Should ACM mandate DBR? Some other publishers are doing it, e.g. IOP in Physics https://ioppublishing.org/news/iop-publishing-commits-to-adopting-double- blind-peer-review-for-all-journals/ Some publishers are going the other way, towards Open Peer Review Let s discuss
Some Resources on Double Blind Review https://double-blind.org/ Tracks double-blind reviewing in top CS conferences Includes resources on DBR (at the bottom of the page) https://www.cs.utexas.edu/users/mckinley/notes/blind.html Kathryn McKinley s site on improving reviewing quality through DBR Also includes many useful DBR resources Two resources advocating for double-blind in SE conferences https://www.cs.cmu.edu/~clegoues/double-blind.html https://people.cs.umass.edu/~brun/doubleblind.html
