Needs Assessment Evaluation of NIH Critique Templates
Conducted by Consultant Concierge Corporation in June 2013, the needs assessment evaluates the utilization and potential improvements of NIH Reviewer Critique Templates within the Office of Extramural Programs and Office of Extramural Research at the National Institutes of Health. The evaluation focuses on streamlining the peer review process and ensuring uniformity in critique format.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
NATIONAL INSTITUTES OF HEALTH OFFICE OF EXTRAMURAL RESEARCH OFFICE OF EXTRAMURAL PROGRAMS Critique Templates Needs Assessment: Final Report June 26, 2013 Submitted By: Consultant Concierge Corporation
OPENING The SRO community is satisfied with the critique templates as a whole. Thankful to have a system in place, as compared to what used to be , and understand the limitations of the current system. Spend time thinking about ways to improve the system, as well as the peer review process. Appreciative to be included in the process of improvement, and to have their voices heard. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 2
SCOPEOF WORK 1 The purpose of this task order is to conduct a needs assessment evaluation of the NIH Reviewer Critique Templates for the Office of Extramural Programs, Office of Extramural Research, Office of the Director, National Institutes of Health. The needs assessment evaluation will assess actual utilization of the templates by NIH staff, and determine what if any improvements may be made to the templates to facilitate their use by NIH staff. The focus of this evaluation is the bulleted critique templates that were introduced as an element of the Enhancing Peer Review Initiative. The Enhancing Peer Review Initiative, first implemented in 2009, involved a major overhaul of NIH s Peer Review Process, affecting NIH review staff and peer reviewers who serve for the Center for Scientific Review and all of NIH s 24 grant-making Institutes and Centers (ICs). A secondary purpose of the critique templates is to streamline the summary statement preparation process for NIH staff by ensuring that reviewers prepare their critiques in a format that is uniform and compatible with the automated compilation steps provided by the eRA Internet Assisted Review module as well as a Microsoft Word macro managed by staff in the Center for Scientific Review that corrects common formatting problems. Finally, the critique templates are designed to reflect current regulations and policy governing the NIH peer review process. Their use reduces the risk of having reviewers using inappropriate or incomplete review criteria when assessing the scientific and technical merit of applications. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 3
SCOPEOF WORK 2 The purpose of the proposed evaluation is to obtain OEP s planning and decision-making process. OEP is considering developing the capability for creating a critique template automatically for each Funding Opportunity Announcement (FOA) as it is published in the NIH Guide, including Parent Announcements, RFAs, and PARs. This new capability would be part of a larger effort to re-design the Guide Publishing System, alleviating the burden of producing critique templates manually from both OEP and Scientific Review Officers (SROs). The re-design of the Guide is now collecting business requirements and therefore, this is an opportune time to consider any other features and changes that can be included in the end-to-end to be design of the NIH Guide system. The target population for this needs assessment will be NIH review staff. NIH has about 500 SROs roughly equally split between CSR and IC s. The evaluation study shall employ focus groups and structured interviews to identify potential points for improvement. Then, a questionnaire shall be developed for review staff to prioritize potential improvements. The burden shall be assessed by asking SROs to identify problems that consistently require their intervention and the time that SROs spend resolving problems related to the critique templates. SROs also shall be asked to prioritize the issues identified, from most to least problematic. the causes for each problem and to assess whether an automated solutions is feasible. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 4
FOCUS GROUPS CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 5
FOCUS GROUP QUESTIONS How often do you modify/customize critique templates? What are the major pain-points you experience in the use and customization of the critique templates? Please be more specific about your experiences using the critique templates: How long do you spend modifying a template? How long do you spend troubleshooting? How do you think the system can be optimized? What issues do you have with template usage? What tasks take the most time to complete? What tasks are the most problematic? Can the process of modifying these templates be improved for you? Technology/automation? Training Standardization Other Would automation of updates to the templates be useful? In the case of global changes to NIH review criteria? In the case of global changes to guidelines? In other cases? What would those be? Do you use the hyperlinks to policy documents? Are the hyperlinks to the actual policy useful or would you prefer to see the standards embedded in the template directly? Are you involved in the development and creation of FOAs? Would being involved in that process make your job easier? Are there any other issues you want to bring up related to the critique templates? CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 6
FOCUS GROUP PARTICIPATION 15 focus groups were conducted, with a total of 77 participants All ICs had at least one representative attend, including AHRQ CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 7
FOCUS GROUP SUMMARY Format & Customizing Review Criteria & Scores Drop-down Menus Logic Checks Non-applicable Sections Reviewers FOA Macro Bullets Modifications Hyperlinks Pre-view System & Process Development OEP is not the owner of the end-to-end process Review criteria should appear directly on template, with the exact wording used in the FOA Macros are strangely inconsistent Bullets are both helpful and problematic Each IC handles modification of templates differently. Some have dedicated staff for modification, some do extensive modification and customization, other allow individual SRO to tailor template as they see fit. Overall formatting is an issue; takes hours; SRO spends an inordinate amount of time cleaning up; some IC do not have staff to assist with clean- up Hyperlinks require an internet connection, which is not reflective of how reviewers work Drop-down menus are very often skipped. System should validate whether required sections have been filled in Different sections are relevant for new applications, others for renewals but both appear confusing for reviewers Allow for the output to be shown in a testing environment after modifications to the template Overall the CT must be user friendly and make it easier for the Reviewer; Reviewer burden must be considered when contemplating change Program officers need to understand what the downstream output will be OEP is not the business owner of the systems that support the CT Macro output is not transparent Provide needed improvement in obtaining evaluative language Check boxes would be preferred it is obvious an action needs to be taken at that point. Warnings should be issues when a required section has not been completed, but it should not prevent upload or further entry Program directors don t understand the timeline, so quality of applications often suffers Unclear how modification affects macro and therefore output Overall impact box and additional review criteria are often missed, should be more prominent Some hyperlinks point to incorrect places, or yet another link Some sections are not applicable for a given application, but there is no way to hide them, creating confusion for the reviewer SRO, PO and Applicants do not see what the Reviewer sees End-to-end process needs to be reviewed; eRA responsibility ends with the generation of a document; then CSR s issue Reviewers do not use full sentences, so it can be difficult to determine what drives score Acceptable/not acceptable should still require some explanation, these shouldn t be the only responses given Work habits of reviewers must be considered. They often work off-line ICs do not modify the templates for fear of the impact on the output Most do not like the box format Embedded text is right in front of the reviewers eyes, there is no need to leave the document Better relationship between review and program would benefit all Ability to rename the criteria and headings; reflected in the FOA and IAR Recognition of roles in review not all reviewers are required to complete all sections Not everyone is aware of training and resources for template modification. And even when they are aware, resources are not so easy to find Section breaks are an issue; inserts blank pages Other ICs developed own macros Incomplete sentences make summary statements more difficult to read Reviewers often use templates already stored on local machines, which are outdated. Is there a way to have them expire and no longer open? The system is slow and sometimes not available Hover boxes would be preferable they could give more detailed information without making the template even longer Drop-down menus are not functional on Macs Is non- responsiveness to the program announcement the purview of review? Some criteria and sections are not applicable to certain applications (i.e. fellowships, training, and clinical trials) but cannot be changed or deleted Macros do not work on Macs Greek letters are not recognized by the system Forces strength/ weakness dichotomy, which can be arbitrary and not useful Major compatibility issues with Macs Because macros do not remove extraneous text, SROs spend significant time on cleaning up Summary Statements Spacing and alignment are issues Easy to mis-click inside drop-down menu, resulting in confusing responses requiring follow up Too many disconnected systems (IAR disconnected from the Template generated) Mac incompatibility remains an issue Templates are not designed for modifications and customization Reviewer knowledge should not be assumed, there is a need for standard training A place for reviewers to put scores in to refer to while writing critiques would be helpful, ensure less disconnect between score and written critique Difficult to determine what version of the macro to use Each section of the template seems to be formatted differently New system needs to be downloadable and portable When the Reviewer modifies a critique after it has been uploaded it overwrites the previous CT End products should be easier to get to Cannot link meeting materials with RFA Must retain the ability to edit within the IC Guidance on scoring system would be useful i.e. what might constitute a score of 4 Require eRA to provide improved systems performance measures CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 8
SURVEY ADMINISTRATION AND RESULTS CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 9
SURVEY ADMINISTRATION The survey was administered through SurveyMonkey.com, and SROs were sent the link via an email invitation that went out to groups and listservs to ensure full coverage of the SRO community. The survey was live for 10 business days and a reminder was sent on day 5. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 10
SURVEY RESPONSE There were a total of 214 surveys with at least one response, from 194 unique IP addresses. 6 responses were removed, 4 because the respondent finished a survey at a later time, 2 because they only answered one question. There were 8 responses from one IP address and 4 from another, but they were all unique. The mean time to complete a survey was 17:42, and the median time was 13:11. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 11
SURVEY RESPONSE 1 Question # # Answered Response Rate Notes Question # # Answered Response Rate Notes 1 205 98.6% 12 115 55.3% Follow up text question 2 204 98.1% 13 185 88.9% 3 204 98.1% 14 103 49.5% Follow up text question 4 199 95.7% 15 193 92.8% 5 198 95.2% 16 21 116.7% Logic driven n=18 6 199 95.7% 17 15 83.3% Logic driven n=18 7 197 94.7% 18 130 74.3% Logic driven n=175 8 197 94.7% 19 182 87.5% 41 replied other 9 197 94.7% 20 195 93.8% 10 126 60.6% Follow up text question 21 37 88.1% Logic directed n=42 11 192 92.3% 22 85 55.6% Logic directed n=153 23 46 22.1% Text question CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 12
PAIN POINTS 3 Responses from focus groups were categorized into 14 major themes: Review Criteria Criterion Scores CSR Macro Bullet points Modifications Formatting and customizing Hyperlinks Drop-down menus/check boxes Logic checks Non-applicable criteria Reviewers work habits FOA development Preview after template modification Technology Respondents were asked to identify up to 5 pain points for themselves, and for their reviewers. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 13
PAIN POINTS CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 14
PAIN POINTS ANALYSIS One issue was rated as a Pain Point for both SROs and reviewers: Non-applicable sections Items that may represent areas to be addressed quickly: Criterion Scores (how and where captured) Review Criteria (as presented in critique template) Formatting and Drop-Down Boxes Longer-Term Items to Address Technology Constraints Reviewers Work Habits CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 15
PAIN POINTS SPECIFIC COMMENTS For many of the major categories identified, representative comments from focus group participants were taken and respondents were asked to rate their agreement or disagreement with the statement. Strongly agree was given a score of 4 Somewhat agree was given a score of 3 Neither agree nor disagree was given a score of 0 Somewhat disagree was given a score of 2 Strongly disagree was given a score of 1 Scores were averaged for the number of respondents answering that question, and are shown out of a maximum possible 4 CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 16
SCORED ISSUES 4.5 Strong Agreement Strong Disagreement 4 3.5 Average Score 3 2.5 2 1.5 1 0.5 0 Overall impact section Warning: overwrite Completion of additional scored criteria Consider reviewer burden Warning: required section Clearer guidance on scoring I know where resources are Evaluative language/bullet points View template after modification Significant customization needed CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 17
SCORED ISSUES 1 6 issues received a score >3, indicating strong agreement: It is important to ensure that the overall impact section is completed. (x=3.81) A reviewer modifying a previously uploaded critique should be warned that the new one overwrites the previous critique. (x=3.40) It is important to ensure that additional scored review criteria are addressed. (x=3.39) Reviewer burden must be considered when contemplating change to the templates. (x=3.36) A warning should be issued when a required section has not been completed, but it should not prevent upload or further data entry. (x=3.22) Clearer guidance on the scoring system would be helpful so reviewers understand what scoring levels mean. (x=3.13) CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 18
SCORED ISSUES 2 4 issues received a score <2, indicating strong disagreement I know where the training and resources for template modification are located. (x=1.98) The use of bullet points increases the use of evaluative language in critiques. (x=1.94) An option to view the template after modifications but before saving would be helpful. (x=1.65) I spend a significant amount of time customizing critique templates to facilitate ease of use by my reviewers. (x=1.45) CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 19
SCORED ISSUES 3 The results of both the identification of pain points and scored issues points to the need and desire to have logic checks and warning signals in place when reviewers are uploading critiques. These can ensure that all required sections have been completed, and that previous critiques are not overwritten. The results also show that although there are complaints about customization and its difficulties, this is not a serious issue with the templates. Finding and accessing resources and training materials should be more transparent. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 20
ANATOMYOF CUSTOMIZATION SROs were asked about the usefulness of tailor-made templates Activity code specific Funding Opportunity Announcement (FOA) specific Request for Application (RFA) specific Individual application specific CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 21
ANATOMYOF CUSTOMIZATION 1 197 Total 120 77 Respondents answered YES (60.9%) Respondents answered NO (39.1%) 1. Would it be useful to have tailor made templates for each activity code? Respondents1 A 114 78 192 Total Respondents answered NO (59.4%) Respondents answered YES (40.6%) 2. Would it be useful to have tailor made templates for each FOA? Respondents 84 101 185 Total Respondents answered NO (45.4%) Respondents answered YES (54.6%) 3. Would it be useful to have tailor made templates for each RFA? Respondents 175 18 193 Total Respondents answered NO (90.7%) Respondents answered YES (9.3%) 4. Would it be useful to have tailor made templates for each application? Respondents Considered additional analysis of the 184 respondents that answered all four of the above questions CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 22
ANATOMYOF CUSTOMIZATION Four questions, each with two possible responses - 16 unique answer combinations Answered Yes to Activity Code, N = 111 Answered No to Activity Code, N = 73 a Activity Code Application Application FOA RFA Count Activity Code FOA RFA Count No No No No N = 40 Yes Yes Yes No N = 45 No No Yes No N = 20 Yes No No No N = 35 No Yes Yes No N = 11 Yes Yes Yes Yes N = 13 No No No Yes N = 1 Yes No Yes No N = 10 No Yes No No N = 1 Yes Yes No No N = 4 No No Yes Yes N = 0 Yes No No Yes N = 2 No Yes No Yes N = 0 Yes Yes No Yes N = 1 No Yes Yes Yes N = 0 Yes No Yes Yes N = 1 CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 23
IN FAVOROFTAILORMADETEMPLATES AT ACTIVITY CODE LEVEL Activity Code Yes (60.3%) N = 111 Funding Opportunity Announcement Yes No (56.8%) N = 63 (43.2%) N = 48 Request For Applications Yes Yes No No (92.1%) N = 58 (22.9%) N = 11 (7.9%) N = 5 (77.1%) N = 37 Individual Applications Yes (5.4%) N = 2 No No Yes Yes (20%) N = 1 Yes No (80%) N = 4 No (94.6%) N = 35 (90.9%) N = 10 (9.1%) N = 1 (22.4%) N = 13 (77.6%) N = 45 CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 24
AGAINSTTAILORMADETEMPLATES AT ACTIVITY CODE LEVEL Activity Code No (39.7%) N = 73 Funding Opportunity Announcement Yes No (16.4%) N = 12 (83.6%) N = 61 Request For Applications Yes Yes No No (91.7%) N = 11 (32.8%) N = 20 (8.3%) N = 1 (67.2%) N = 41 Individual Applications Yes No No Yes (0%) N = 0 Yes (0%) N = 0 Yes (100%) N = 11 No No (0%) N = 0 (2.4%) N = 1 (97.6%) N = 40 (100%) N = 20 (100%) N = 1 CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 25
ANATOMYOF CUSTOMIZATION 2 There is overwhelming support for tailor made templates at the activity code level 60.9% of all respondents (and 60.3% of those who answered all four refinement questions) indicated that it would be useful. There is also a great deal of support for tailor made templates at the RFA level, with 54.6% of all respondents (and 54.3% of those who answered all four refinement questions) indicating this would be useful. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 26
ANATOMYOF CUSTOMIZATION 3 Less than 10% of all respondents (and 9.8% of those who answered all four refinement questions) indicated that it would be useful to have tailor made templates for each individual application reviewed. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 27
ANATOMYOF CUSTOMIZATION 4 Respondents seem to understand the increasing complexity that tailor made templates would require, with only 13 indicating that tailor made templates at each level of refinement would be useful. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 28
REPRESENTATIVE NEGATIVE COMMENTS This will create more confusion among reviewers. Standard template is always beneficial except for few FOAs. Reviewers don't always use the assigned template- sometimes just edit an old critique. Reviewers barely tolerate the template as it is. If there were more than one option it would be ignored or the wrong one would be used - further complicating the process of editing critiques afterwards. Uniformity and fairness of review cannot be maintained if different applications in the same meeting are reviewed using different criteria. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 29
REPRESENTATIVE POSITIVE COMMENTS Tailor made templates would be great but only needed when review criteria is different. Even though the standard criteria are the same for R01 and R21, for example, there should be reminders that focus the reviewers on the intent of the activity code. So, R21 is exploratory and should not include the consideration of preliminary data. This would ensure that those unique criteria are not missed. Different activities have different requirements in the grant submission; the same is true for the review criteria; thus the template should be different. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 30
OTHER COMMENTS Applications for non-R01 grants--fellowships or conferences, for example-- would benefit from having tailored explanations of how to judge overall impact. Fellowship research can be valuable for training and contributing to the field without necessarily involving studies that are likely to have a "sustained, powerful influence on the field. There should be an emphasis on what determines the impact for a given activity code. I see the current family of template as striking a good balance between specificity and convenience. I believe there should be a sub-section on RFA specific criteria in the FOA and there should be a hyperlink (specific criteria) in the Approach section of the template, immediately following the standard Approach criteria. So that SRO should be able to hyperlink the specific criteria with relevant URL that takes the reviewers to RFA specific criteria section in the FOA. Keep the templates consistent as much as possible. FOAs can be specific about what is desired, but needs to follow basic template format. Reviewers are more likely to follow the template than the FOA. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 31
ADDITIONAL SCORED CRITERIA 1 SROs were asked in cases where there are additional criteria receiving scores, how many additional scored criteria would be appropriate. 2.1% 8.5% 0 24.1% 1 53.9% 2 3 4 11.3% 41 (of 176, 20.3%) chose other with 23 (56.1%) saying it was dependent on the FOA. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 32
ADDITIONAL SCORED CRITERIA 2 eRA asked that this question be included, so they could get some input for the new system they are developing. This question is better directed at Program Directors and Program Officers, since they are the ones who develop and determine the additional criteria that applications will be judged on. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 33
ASSEMBLYOF PRELIMINARY DRAFT SUMMARY STATEMENTS SROs were asked whether they used Internet Assisted Review (IAR) to assemble preliminary draft summary statements. 21.5% Yes No 78.5% 17 of 42 (40.5%) who responded No and provided comments indicated that ESAs or support staff use IAR. 10 of 85 (11.8%) who responded Yes and provided comments also indicated that ESAs use IAR. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 34
ASSEMBLYOF PRELIMINARY DRAFT SUMMARY STATEMENTS Overall, respondents are pleased with IAR as a means to assemble preliminary draft summary statements. It would be nice if it were faster. It would be helpful if the macro could delete the extra blank lines and sections where they are irrelevant or the reviewers left blank. The spacing is a problem. It works fine - but would prefer it to have the final output similar to the output after the CSR Macro.. There are often problems using the CSR Macro - especially with modified templates. Since everyone needs to clean up the SS in a similar way - this should be part of the final output from IAR. CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 35
CONCLUSIONS AND PRIORITY SETTING CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 36
CONSTRAINTS Reviewers Discussion with reviewers may be warranted regarding Work Habits Universal Platform Tablets Attitude and ability to adapt to changes CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 37
SHORT-TERM Review the Critique Template process for roles and ownership There are several cross-functional players in the process; process ownership is fragmented Stakeholders SRO and Reviewer Community Policy OEP Technology eRA, CSR, OEP The focus groups and survey indicate areas for quick-hit improvements: Non-applicable sections Criterion Scores (how and where captured) Review Criteria (as presented in critique template) Formatting and Drop-Down Boxes Changes to improve template appearance What safeguards can be put in place to ensure that the overall impact section is completed? If changes are made to the template, ensure reviewers can accommodate the change What safeguards can be put in place to ensure that additional scored review criteria are addressed? To improve the connection between score and written critique a place should be provided for reviewers to enter scores while writing critiques. What changes need to be made so the review criteria can appear directly on template, with the exact wording used in the FOA? Make Training materials more readily accessible CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 38
LONG-TERM eRA has proposed changes to the summary statement process. Many of the issues raised in this assessment are addressed by the proposed changes. eRA is also very interested in the results of the current study. Will New Technology: Facilitate the functionality for tailor made templates? Reduce system interface issues? Allow the process to proceed more smoothly, with fewer intermediate steps introducing more opportunity for delay? CRITIQUE TEMPLATE NEEDS ASSESSMENT SUBMITTED BY: CONSULTANT CONCIERGE CORPORATION June 26, 2013 39
