Guidelines for Being a CVPR Area Chair Based on ICCV 2019 & CVPR 2019
Serving as an Area Chair for CVPR involves making transparent decisions for the community, ensuring consistency, and handling critical paper evaluations. The decision process includes assigning papers to reviewers, managing reviews, discussions, rebuttals, and final decisions. Area Chairs play a crucial role in the evaluation and selection of papers, guiding the review process, and providing valuable contributions to the community.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
How to Be an Area Chair for CVPR Based on material by ICCV 2019 / CVPR 2019 Program Chairs
Guiding principles We want to make the best decisions we can to serve the community. We want these decisions to be transparent to the authors. While an author may not be happy with a decision, he/she should understand why the decision was made. We want reasonable consistency across area chairs. Much of your work will be routine administration; you're serving the community, and you might be dealing with relatively few papers whose fate you can change. HOWEVER: those are the most important cases, where your contribution is extremely valuable.
The Decision Process: Overview 8. Program chairs finalize oral decisions based on space/time constraints Program Chairs 7. Area chairs discuss with reviewers and each other, make accept/reject decisions and oral recommendations 1. PCs assign papers to ACs Primary Area Chair Secondary Area Chair 2. Primary AC suggests reviewers Authors 5. Authors provide rebuttal to reviewers and ACs 3. Papers are assigned to reviewers using global matching algorithm 6. Reviewers update final reviews 4. Reviewers write reviews, which are released to authors (after AC checking for quality) Reviewers
The Decision Process: In Detail Program chairs (PCs) assign papers to area chairs (ACs), usually not more than 35 papers per AC. ACs suggest 10 reviewers per paper, with help from Toronto Paper Matching Software (TPMS). Papers are assigned to reviewers (3 per paper) using an optimization algorithm that takes into account AC suggestions, paper load and conflict constraints, and the prayers of PCs that nothing goes wrong. Reviewers submit initial reviews, typically handling 6-10 papers each. ACs check quality of reviews, chase late reviewers, and assign emergency reviewers as necessary. Authors receive reviews with a mixture of gasps, grimaces, grumbles, and the occasional grin. After much thought and re-reading of paper and reviews, authors submit rebuttals. Discussion ensues among reviewers and AC, based on all reviews, rebuttal, and paper. Reviewers update their ratings and justification. ACs make decisions and write meta-reviews. The decision and meta-review are recorded by the primary AC for each paper and checked/approved by the secondary AC. Primary and secondary ACs discuss borderline papers. Additional opinions may be sought from other expert ACs after checking for conflicts. In addition to accept/reject decisions, AC pairs provide a roughly ranked list of oral/spotlight nominations to the PCs. PCs make final determination of poster/spotlight/oral for accepted papers, almost entirely based on the recommendations of the ACs but taking into account time and space constraints and topic diversity. 1. 2. 3. 4. 5. 6. 7. 8.
Three phases of AC duties 1. Assign papers to reviewers In some ways, this is actually the most important step. If done well, the rest of the process is a piece of cake. If not, it s a disaster. Do not rely solely on TPMS and topic matching. You are smarter than a weighted histogram matcher, right? The automated tools provide initial rankings, but ultimately you should rely on your judgment. 2. Guide the review/discussion process Check that submitted reviews are complete and respectful. Catching bad reviews early can save you (and the authors) a lot of pain later. Post-review, start discussions where appropriate to clarify any conflicting opinions. Get reviewers to update their final ratings especially to reconsider their positions once they have more context (rebuttal, other reviews) and time to reflect. 3. Decide, write meta-reviews; check decisions and meta-reviews of papers for which you are secondary meta-reviewer
How to choose reviewers 1. Spend a few minutes reading the abstract / intro / figures / citations to understand the key idea, main technical topics, and community involved. While reading the paper, jot down names of reviewers that come to mind. Look through the list of references for more ideas. Finally, look at top-ranked reviewers according to TPMS scores. Suggest a mix of 10 potential reviewers who have expertise in each main technical topic, are likely to enjoy reading the paper, and can give feedback on the big ideas as well as technical details. Pro tips: Do not trust TPMS blindly; many of its suggestions are bad. Verify the suggestions by checking the reviewers Google Scholar profiles. Some reviewers may not show up in the TPMS rankings because they did not create a TPMS profile. If you think of somebody who may be good to review a paper, search for their name directly. 2. 3.
Level of decisions Award: major advances that will heavily impact the field; will be used by many people, create new capabilities, etc. E.g., ResNet (CVPR 2016 Best Paper), Mask R-CNN (ICCV 2017 Best Paper) Oral: potential to be very significant; worthwhile for the whole community to hear about Poster: incremental steps that expand the sum of the community s knowledge or add bricks to the cathedral of knowledge; papers introducing useful tools; papers of interest to a subcommunity Also, creative ideas that are hard to judge but could be promising -- no one knows the future, so we should give the benefit of the doubt to plausible ideas Reject: unlikely to be significant Your job as an AC is mainly to decide acceptance/rejection and to provide PCs with recommendations for orals and awards
Why not accept everything? Papers can have a negative impact: Wrong or fraudulent results mislead the field and damage the reputation of the conference. Misleading evaluation makes it hard to compare with, kills follow-up. Creates bad precedent (weak paper X got in, so this one should too). Fatigue/overload of too many papers, wastes everyone s time. Each weak or mediocre paper we accept hurts the conference a little (though not as much as rejecting a good paper).
The role of experimental evaluation Extraordinary claims require extraordinary evidence. Pierre Simon-Laplace, paraphrased Different papers require different levels of evaluation, for example: Potentially transformative idea: basic proof-of-concept Established problem, plausible idea: benchmark results Weird, overly complex, implausible, and/or seemingly incremental: extraordinary results (which need to be scrutinized carefully) Position piece or theory paper: no experiments Having a good idea is not enough to have a paper. Ideas are cheap.
How to decide Judge each paper based on its own merits, not some ranking with respect to other submissions to the conference. There is no global quota on the number of papers we can accept. For unanimous accepts or rejects, just follow the reviewers (but do a sanity check in case the reviews are completely off or missed something). For all other cases: Read reviews, see if you agree with claims. Which arguments did the reviewers make that you find compelling? Pay more attention to the arguments than the ratings! If you aren t sure: get help from secondary meta-reviewer or another expert AC of your choice (contact the Program Chairs first to check for conflicts).
How to write meta-reviews A meta-review is a well-founded argument that a paper does/doesn t meet the criteria for acceptance. It is not simply a summary of the individual reviews. Especially in the case of difficult reject decisions, you must reconcile the reviews and give explicit justification for the final recommendation. Key principle: would a reasonable author object to the meta-review as the basis for the decision? Use impersonal language: AC, ACs, authors -- not I , you . Take care when reviewers don t agree and especially if your decision goes against the majority. Secondary meta-reviewer must proofread every meta-review to make sure it meets standards.
Meta-reviews for easy cases When all reviewers agree, a couple of sentences will suffice. In cases of acceptance, do not mention oral vs. poster recommendation in your meta-review since this status will not be finalized until later by the PCs. In cases of rejection, briefly mention the main reasons for rejection and acknowledge the rebuttal (if only to say that it was missing or did not change anybody s mind). Examples: Three reviewers agree that the paper contains technical errors and is not clearly written. There is no rebuttal. Three reviewers find that the paper is below ICCV standards due to a lack of novelty and insufficient experimental validation. The rebuttal did not assuage these concerns.
Meta-reviews for difficult cases If referees disagree or make borderline recommendations: 1. Summarize the main arguments of the reviewers. 2. Indicate whether the rebuttal removed any initial concerns (if there was no rebuttal, explicitly state there was no rebuttal ). 3. Summarize any substantive points from the reviewer discussion (authors will not see the discussion). 4. Reconcile all the above information and explain the main factors for your decision. 5. Discuss your decision with the secondary meta-reviewer. Include This paper was discussed by two ACs or similar language.
Good meta-review Initially, R1 and R2 liked the main idea of the paper and gave it a positive recommendation, but R3, who wrote the most in-depth review, found that the experimental results were insufficient. In particular, all the experiments in the paper are on one small dataset [anon]. In the rebuttal, the authors responded that there is no ground truth available for other datasets. But in the subsequent discussion, R3 pointed out that other datasets provide ground truth, e.g. [anon] and [anon], and also that the comparisons given in the paper contradict the reported results in [anon]. R1 and R2 acknowledged the validity of these concerns, but kept the same rating. The AC finds the arguments of R3 to be the most persuasive. To properly validate their approach, the authors need to address the discrepancies with prior work and evaluate on the datasets pointed out by R3. This decision was discussed with and approved by the secondary meta-reviewer. 1. Summarize the main arguments of the reviewers 2. Indicate whether the rebuttal removed any of the original concerns 3. Summarize any substantive points from the reviewer discussion 4. Reconcile the above information and explain the main factors for your decision 5. Discuss your decision with the secondary meta- reviewer
Not good enough R1 liked the paper for reason X. R2 disliked aspect Y. R3 disliked aspect Z. Therefore the paper was rejected. Good start, but you need to explicitly reconcile the three reviews. Reviewers all say that they are OK if the paper is rejected, and we have many good papers. Really bad, reads as though the decision on this paper depended on decisions made on other papers, or as though the AC pressured the reviewers to reject.
Overruling reviewers If all reviewers agree that a paper should be rejected, we expect the paper to be accepted ONLY if there are unusual circumstances, e.g., three exceptionally poor or misguided reviews. If all reviewers agree that a paper should be accepted we expect the paper to be rejected ONLY if there are unusual circumstances, e.g., a major technical error, fraud, or plagiarism that are not detected by reviewers. If you wish to overrule a unanimous reviewer recommendation: Secondary meta-reviewer must agree with decision. You must contact the PCs for approval.
Main take-aways 1. Invest your time in picking the right reviewers for each paper. Don t rely solely on the paper matching tools. 2. Make a well-founded decision for each paper, considering the substantive points raised by the reviewers and authors. Don t rely solely on reviewer ratings. 3. Write a meta-review that fully justifies the decision to the authors. They should understand the reasons for the decision even if they don t like it. 4. Always be respectful to the authors and reviewers, who put much time and trust into the process. And complete your tasks on time!
