Understanding ARC Rejoinder Process in Peer Review: A Comprehensive Overview
The ARC Rejoinder Information Session provides insights into the Peer Review Process of the Australian Research Council (ARC). It explains the role of Assessors, the importance of Rejoinders, and the overall criteria for funding decisions under the National Competitive Grants Program (NCGP).
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
ARC Rejoinder Information Session (June 2014) ARC Rejoinder Information Session Peer review process & How to write a Rejoinder Tina D Urbano Research Funding Team research.funding@federation.edu.au Sources & acknowledgements: ARC website, some information adapted from other institutions 23 June 2014
ARC Rejoinder Information Session (June 2014) What is an Assessor Report? Peer review comments made by the external assessors (CoE Members, Detailed Assessors and General Assessors) in relation to the selection criteria about your proposal What is a Rejoinder? Applicant s response to the Assessor Reports It s your opportunity to constructively respond to comments and criticism made by Assessors Rejoinders provide information for the CoE panel members who decide which proposals are fundable Rejoinders are not viewed by external assessors they are considered by ARC CoE Panels when deciding the overall ranking of proposals
ARC Rejoinder Information Session (June 2014) General overview - NCGP & Peer Review Process http://www.arc.gov.au/general/assessment_process.htm & http://www.arc.gov.au/ncgp/default.htm (19-6-14) The ARC funds research and researchers under the National Competitive Grants Program (NCGP). This comprises two main elements - Discovery and Linkage - under which the ARC funds a range of complementary schemes to: support researchers at different stages of their careers, Relative to opportunity + / - build Australia s research capability, expand and enhance research networks and collaborations, and develop centres of research excellence. The majority of funding decisions under the NCGP are made on the basis of peer review.
ARC Rejoinder Information Session (June 2014) Peer Review Process Peer review of proposals may be undertaken by: members of the ARC College of Experts (CoE); - currently 158 members Assessors and/or other eminent researchers. - approx 19,000 ARC CoE members divided among five panels: Biological Sciences and Biotechnology (BSB), Engineering, Mathematics and Informatics (EMI), Humanities and Creative Arts (HCA), Physics, Chemistry and Earth Sciences (PCE), and Social Behavioural and Economic Sciences (SBE).
ARC Rejoinder Information Session (June 2014) Characteristics & roles: ARC College of Experts (CoE): experts of international standing drawn from the Australian research community, including higher education, industry and public sector research organisations. appointed by the Minister for terms of 1 - 3 years. annually the ARC conducts a competitive nomination process to replace retiring members.
ARC Rejoinder Information Session (June 2014) ARC College of Experts (CoE) cont.: assign Assessors to Discovery Projects proposals participate in selection panels for the ARCs Discovery Projects and Linkage Projects schemes, submit initial assessments, review Assessor assessments and applicant rejoinders, rank proposals, and make funding recommendations to the ARC CEO provide strategic advice to the ARC on emerging disciplinary and interdisciplinary developments, and innovative approaches to research.
ARC Rejoinder Information Session (June 2014) Assessors and/or other eminent researchers: readers whose research interests span the broad field/s of the research projects they are asked to assess (General Assessors), or experts (Detailed Assessors) in the specific field of a proposed research project, assess proposals against selection criteria in the funding rules score, rank and provide written comments for each proposal.
ARC Rejoinder Information Session (June 2014) Assessors and/or other eminent researchers cont.: new Assessors are selected through CoE member nominations each year the ARC reviews successful proposals and, where suitable, adds funded researchers not already listed to the Assessor database.
ARC Rejoinder Information Session (June 2014) Assignment of proposals Proposals are matched to a selection panel using the Fields of Research (FOR) ARC Executive Directors review the appropriateness of the proposals in each panel and may transfer proposals to different panels if required. They also identify interdisciplinary proposals that will require assessors from more than one selection panel. In Discovery Projects, CoE members also assign up to four Assessors per proposal. Assessors may decline to assess proposals if they consider it to be outside their area of expertise or a conflict of interest
ARC Rejoinder Information Session (June 2014) Assessment process Selection and Scoring Assessment forms CoE members and Assessors assess the merits of a proposal against selection criteria in funding rules submit assessment forms through RMS typically assessment forms request scores and written comments for each selection criteria
ARC Rejoinder Information Session (June 2014) Selection criteria track records Selection criteria and weightings vary from scheme to scheme: Most scheme selection criteria include investigator track record and proposed program of research components. When assessing the track records of researchers nominated in a proposal, assessors are advised to take into account the opportunities that the researchers have had to build their research profile. - need to take this on board when considering submitting an application = build your research profile and track record
ARC Rejoinder Information Session (June 2014) Scoring Bands The ARC provides the following advice to assessors on the degree of merit associated with specified scoring bands as a guide. A: Outstanding Of the highest quality and at the forefront of research in the field. (~10%* of Proposals should receive ratings in this band.) B: Excellent High quality research and a strongly competitive Proposal. (~15%*) C: Very good An interesting, sound and compelling Proposal. (~20%*) D: Good A sound research Proposal, but lacks a compelling element. (~ 35%*) E: Uncompetitive: The Proposal is uncompetitive and has significant weaknesses or more fatal flaws. (~20%*) *Percentages are provided as a guide only.
ARC Rejoinder Information Session (June 2014) Rejoinders at last these are released = here s your chance to fight for your proposal Once the initial assessments are submitted to the ARC, researchers are invited to respond to the written comments [only] from the assessors. - Scores and assessor details are not released to applicants. Rejoinders allow applicants and nominated participants the opportunity to clarify any misunderstanding or difference of opinion about perceived weaknesses in the proposal. - It also provides an important feedback mechanism that may help researchers in future proposals. The assessors' reports and the rejoinders also assist the ARC CoE members in reviewing proposals.
ARC Rejoinder Information Session (June 2014) Selection Meetings after rejoinder process CoE members attend selection meetings to rank the proposals and determine to what extent they ought to be funded. A recommendation for funding is made to the ARC CEO.
ARC Rejoinder Information Session (June 2014) Ranking of proposals To determine a final ranked list, the selection panel considers and discusses proposals particularly those around the funding margin. Selection panel members may revise the ranking of a proposal in response to a number of factors including: assessor reports; persuasive rejoinder arguments; and panel discussions comparing the relative merits of proposals against the selection criteria. and taking into account any ranking anomalies.
ARC Rejoinder Information Session (June 2014) Assignment of budgets Once a CoE selection panel has determined the final ranked list of proposals, members determine funding recommendations for all proposals in the fundable range. Awards outcomes - ARC CEO s recommendation On advice from the CoE, the ARC CEO makes funding recommendations to the Minister. Approval of proposal expenditure rests with the Minister. Generally, the Minister acts on advice from the ARC CEO.
ARC Rejoinder Information Session (June 2014) OK, so what s the chances of success? Discovery Projects 2010 4068 925 22.7% 2011 4230 931 22.0% 2012 3544 778 22.0% 2013 3425 732 21.4% 2014 3534 703 19.9% Submitted Funded Success Rate DECRA Projects 2012 2159 227 12.8% 2013 1281 200 15.6% 2014 1468 200 13.6% Submitted Funded Success Rate Source: ARC research funding trend data, Success rate of proposals by scheme, by funding commencement year, http://www.arc.gov.au/general/searchable_data.htm (19-6-14)
ARC Rejoinder Information Session (June 2014) Writing a Rejoinder - How to read your Assessor Reports - Where to start? -Tips what to do /what not to do - Rejoinder Process Instructions
ARC Rejoinder Information Session (June 2014) ARC Rejoinders http://intranet.monash.edu.au/researchadmin/arc/arc-rejoinders.html The purpose of the Rejoinder process is to allow Applicants to respond to assessment comments made by external assessors. Rejoinders are not viewed by external assessors but are considered by an ARC College Panel when deciding the overall ranking of a Proposal. Applicants and named participants are given access in RMS to the Assessor Reports and the Rejoinder form.
ARC Rejoinder Information Session (June 2014) IMPORTANT Normally a 2-week turnaround for rejoinders Rejoinder text is limited to 5000 characters Use it wisely
ARC Rejoinder Information Session (June 2014) Before you start You should definitely write a rejoinder no matter how good (or bad) your assessor reports are can t always assume the assessor s scores match what they write. View it as a bonus , an opportunity to argue your case and, if written well, increase your chances of success. Remember your Rejoinder goes to the CoE panel member not the assessor; give your champion ammunition to help get your proposal over the line. Understand what you are trying to achieve make a plan, what will be the structure, what needs to be addressed and what doesn t.
ARC Rejoinder Information Session (June 2014) How to read your Assessor Reports Read, Read and Re-read try to keep an open mind Think take your time give yourself time to react and neutralise any emotional reaction Digest the comments show it to colleagues, ask for advice Write the first draft and then put it aside Be careful not to misinterpret comments and criticism Check your proposal Was it clear? Did the reviewer miss something important that might help your panel champion? Generally, assessors won t know who you are they are not making personal attacks; think about why they wrote what they did.
ARC Rejoinder Information Session (June 2014) Where to start? Highlight the comments and criticism that need to be addressed. Determine whether the criticisms are accurate or not - if they are, say how the concern will be addressed. - if not, say why not Use the criteria headings to structure your response Be brief and systematic quote and respond, quote and respond Identify the Assessor for each response - similar comments from more than one Assessor group the response (e.g. Assessor 2 similar )
ARC Rejoinder Information Session (June 2014) Where to start? - cont Don t waste space discussing all the positive comments (unless it s all positive!) - they ve read your proposal and reports; - so focus on the criticism and issues that need to be addressed Only if space permits, very briefly acknowledge positive comments Write it put it aside (time permitting) edit Ask colleagues for feedback
ARC Rejoinder Information Session (June 2014) TIPS Dos & Don ts DO: Allow yourself to react, get annoyed and complain then work out a strategy to respond Write in a clear and concise manner Use a positive and conciliatory tone Put emphasis on why your proposal is worth funding Consider each assessor s comments seriously
ARC Rejoinder Information Session (June 2014) DO cont: Respond to every critical comment address them directly Acknowledge something you may have gotten wrong Be succinct and avoid overly technical comments they may have a broad knowledge but may not be an expert in your field Remember reviewers are experts within their fields potentially they are correct Provide an indication of the comment to which you are responding.
ARC Rejoinder Information Session (June 2014) DO cont: Help the College of Experts to see the proposal from your point of view. Stick to the main points. Whenever possible, a short rejoinder is a good rejoinder Put criticisms into perspective Use it as a learning experience what would you do differently next time? Rejoinders are optional but highly recommended. If you don t want to submit you must state this in RMS
ARC Rejoinder Information Session (June 2014) DON T: Don t take it personally or jump to conclusions Don t argue with or be hostile to an Assessor point out any factual errors and respond accordingly Don t play assessors off against each other Don t be mislead by general positive comments may not equal a high ranking Don t try to guess your scores or ranking comments can be misleading focus on the issues that need addressing Don t over react see it as a potential positive Don t nit-pick or complain if they misunderstood something, clarify it (politely)
ARC Rejoinder Information Session (June 2014) DON T cont: Don t regard unexpected criticism as having no ground Don t request that comments be disregarded without providing a good reason Don t criticise or question the assessors competence or expertise Don t hide away from negatives by only focusing on the positives Don t over do it give clear, succinct responses explaining why you should be funded, be careful of using technical details Don t add new information, updates, etc.* assessed by what was in the proposal
ARC Rejoinder Information Session (June 2014) DON T cont: Don t give up! - you ve put the effort into this project; - can it be funded elsewhere? Rejoinders are optional but highly recommended. If you don t want to submit you must state this in RMS
ARC Rejoinder Information Session (June 2014) What if you don t get an Assessor Report? Not likely but still submit a Rejoinder. This is when you can tell the panel about new developments, new work, successes, publications, etc.
ARC Rejoinder Information Session (June 2014) Assessor Reports are released!!!!! Lead CI will receive email notification from ARC. Research Funding Team will also be in touch. How to go about it?
ARC Rejoinder Information Session (June 2014) ARC Rejoinder Process Instructions **Edited by Tina D Urbano; Full document can be found at http://www.arc.gov.au/applicants/assessor_rejoinders.htm Applicants and nominated participantsare given accessto the Rejoinder form in the ARC Research Management System (RMS) - able to access the assessors' reports and - have an adequate, though brief, period for submitting a rejoinder. Queries or requests Applicants should direct queries or requests to the Research Office at the Administering Organisation. Research.funding@federation.edu.au Do not contact ARC directly
ARC Rejoinder Information Session (June 2014) Section 1: Instructions for Applicants 1.1 Log on via the ARC Research Management System (RMS) All participants listed on the Proposal with Edit rights can view, edit and submit Rejoinders to the Research Office. Log on via the RMS at https://rms.arc.gov.au using your User Id and password. Note: If you have forgotten your password or User Id please select Reset Password or Retrieve User Id link on the RMS homepage and follow the prompts.
ARC Rejoinder Information Session (June 2014) 1.2 View Proposals To access your Proposal/s submitted to the ARC please select the Proposals link under Proposals.
ARC Rejoinder Information Session (June 2014) Select the relevant scheme from the Scheme Round drop down list and then click on the Search button.
ARC Rejoinder Information Session (June 2014) A list of your Proposal/s submitted to the ARC for this scheme will appear in this screen. To access the assessment/s for the submitted Proposal please click on the Proposal Id.
ARC Rejoinder Information Session (June 2014) 1.3 Accessing assessor comments After clicking on the Proposal Id you will see a screen listing the Proposal and a series of tabs. The first tab Assessment will show links labelled as Assessor A , B , C or D which contain the assessment comments for your Proposal. ** note: you will receive up to 4 Assessor Reports, some may have less
ARC Rejoinder Information Session (June 2014) To access the assessment comments, click on the individual Assessor links
ARC Rejoinder Information Session (June 2014) Clicking on an Assessor link will show a new screen where the assessment comments can be viewed against the selection criteria.
ARC Rejoinder Information Session (June 2014) Note - Downloading the Assessor Reports: While the assessment text cannot be printed, it can be copied by: selecting all of the text in the Assessment window, right-clicking and selecting Copy , and then pasting the text into a Word document Recommended that you draft your Rejoinder in Word then paste it back into RMS
ARC Rejoinder Information Session (June 2014) Remunerated Participant If there is a remunerated participant request, such as Discovery Indigenous Award (DIA) or Discovery Outstanding Researcher Award (DORA), you will need to select the second tab Remunerated Participations to access the assessment comments for the remunerated participant.
ARC Rejoinder Information Session (June 2014) 1.4 Edit and save a Rejoinder To submit a Rejoinder click on the tab entitled Rejoinder . You will then be able to enter your Rejoinder text to a maximum of 5000 characters (including spaces). As the Rejoinder field is for text only, pictures, graphs or other documents cannot be inserted or attached.
ARC Rejoinder Information Session (June 2014) When you have completed your Rejoinder Click on the Save button to save your Rejoinder text.
ARC Rejoinder Information Session (June 2014) 1.5 Submitting your Rejoinder to the Research Office Once you have saved your Rejoinder the Save & Submit button will no longer be greyed out and will become active. To submit your Rejoinder to the Research Office please click
ARC Rejoinder Information Session (June 2014) The Rejoinder State will change from In Draft to Submitted to Research Office once the Save & Submit button has been selected.
ARC Rejoinder Information Session (June 2014) Note: If you wish to make any further changes to your Rejoinder once the Rejoinder State is at Submitted to Research Office , you will need to contact the Research Office to have the Rejoinder desubmitted back to an In Draft status. If no further changes are needed the Research Office staff will then submit your Rejoinder to the ARC. Alternatively, the Research Office will be able to edit the Rejoinder on your behalf before submitting to the ARC.
ARC Rejoinder Information Session (June 2014) 1.6 Submitting your Rejoinder to the ARC The Administering Organisation s Research Office will submit the Rejoinder to the ARC. - You cannot do this, so please let us know when it is ready.
ARC Rejoinder Information Session (June 2014) Dates & Deadlines Rejoinder Open and Close dates for each scheme are made available on the ARC Important Dates website http://www.arc.gov.au/media/important_dates.htm. Currently only an indicative time period is listed: DP end June, DECRA early July Exact dates will be on the website once the Rejoinder period opens. The first named participant on each proposal will receive an automated email from the RMS informing them of the opening of the Rejoinder period. A network message will also be issued to Administering Organisation Research Offices, informing them of the opening of the Rejoinder period.
ARC Rejoinder Information Session (June 2014) Internal Support & Final Deadlines for Submission (tbc) ARC allow around two weeks to respond to Assessor Reports Internal review deadline ~ 4 days before ARC closing date Expert/Academic review and feedback Administrative and Compliance review
