Strategies for Effective Paper Reviewing in Academic Circles
In the world of paper reviewing, the importance of incentivizing good reviews is explored through game theory models. The goal is to ensure quality papers are accepted and bad ones are rejected. Strategies like rewarding good reviewers and proposing innovative incentives like free beers are discussed. The focus is on achieving equilibrium among reviewers to enhance the reviewing process.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Incentive Compatible Reviewing Erez Karpas Technion-Microsoft Electronic-Commerce Research Center Festivus @ ICAPS 2012
Why We Review Papers? Social welfare o Ensure good papers are accepted o Ensure bad papers are rejected o Ensure ICAPS had a good reputation Personal reward o Being on program committees looks good in CV o Early access to innovative research o "You forgot to cite me" But - why should we write good reviews?
Why Write Good Reviews? We use game theory to model reviewing The game is an (over)-simplified model of reality We will draw conclusions from the solution of the (over)-simplified game, and apply them to real life Economists do it all the time, why cant we?
The Reviewing Game Review a single paper Outcome for the paper is either right or wrong 3 players - the reviewers Two actions for each player: SWEAT or COAST The outcome is right iff two or more reviewers SWEAT Reward for right outcome: 2 Cost for SWEAT: 1
The Reviewing Game Matrix Reviewer 3 = SWEAT Reviewer 2 Desired Outcome SWEAT COAST Reviewer 1 1 1 1 1 2 1 SWEAT 2 1 1 0 0 -1 COAST Equilibria Reviewer 3 = COAST Reviewer 2 SWEAT COAST Reviewer 1 1 1 2 -1 0 0 SWEAT 0 -1 0 0 0 0 COAST
Summary The desired outcome is not in equilibrium o One of the reviewers will COAST o Which reviewer? The proposed solution o Reward good reviewers o The best reviewer award is a good start, but there's only one o How about - a free beer to all good reviewers? People will do more for a free beer than for enough money for 2 beers
For A Free Beer I tricked the DC into thinking I wrote an awesome paper I have gone swimming in Antarctica
