Overview of NSF CAREER Program and Conflicts of Interest
The National Science Foundation's Directorate for Computer & Information Science & Engineering (CISE) panel charges CAREER proposals, promoting transformative research and supporting junior faculty. The CAREER program emphasizes outstanding research, education integration, and offers at least $400K over 5 years to recipients. Additionally, the importance of managing conflicts of interest in the peer review process is highlighted to ensure integrity and fairness in evaluating proposals.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
National Science Foundation Directorate for Computer & Information Science & Engineering (CISE) Panel Charge CAREER Proposals
Overview of the CAREER Program Faculty Early Career Development (CAREER) Program An NSF-wide program to support junior faculty. Emphasizes outstanding research, excellent education, and the integration of education and research. Each award is at least $400K spent over 5 years Presidential Early Career Awards for Scientists and Engineers (PECASE) Nominations (citizens and green-card holders) are made by participating federal agencies NSF nominates, about 20/year, from among the most meritorious new CAREER awardees Final selection made by White House Office of Science & Tech. Policy. NSF selections are strongly influenced by Reviews/Panel Outcomes and Broader Impact activities post-award
Emphasizing Transformative Research Transformative research involves ideas, discoveries, or tools that radically change our understanding of an important existing scientific or engineering concept or educational practice or, leads to the creation of a new paradigm or field of science, engineering, or education. Such research challenges current understanding or provides pathways to new frontiers or, allow development of new practices in cyberinfrastructure (for ACI)
Conflicts of Interest Primary purpose is to remove or limit the influence (or appearance of influence) of ties to an applicant institution or investigator that could affect reviewer advice. Second purpose is to preserve the trust of the scientific community, Congress, and the general public in the integrity, effectiveness, and evenhandedness of NSF s peer review process.
Conflicts of Interest Sign and turn in Conflict-of-Interest form Typical relationships that could lead to a conflict: INSTITUTIONAL current or previous employment (12 months) or seeking employment award, honorarium, or travel payment (past 12 months) officer or governing board any financial interest PERSONAL co-author of paper or project collaborator (48 months) co-edited journal or proceedings (24 months) Ph.D. thesis advisor or student (life- long) family member or close friend You must not participate in the discussion of any proposal for which you have a conflict. Please discuss any actual or perceived conflicts with PDs now. Any relationship with an institution or person involved in a proposal that might look questionable to a third party should be discussed with a PD 5
Confidentiality Process and results are confidential! Do not disclose identities of your fellow reviewers. Do not discuss proposals out-of-band (in the absence of a PD) including email, Skype or text etc. Do not disclose identities of people associated with proposals (PI, Co- PIs, Consultants, etc.) Do not discuss results or recommendations with other people. Do not use names of other reviewers in your review or Panel Summary (if you are the Scribe). Proposals contain sensitive information and are not in the public domain -- do not copy, distribute or quote from them. You can indicate (e.g., on a resume) that you served NSF on a review panel just don t identify which panel(s).
Merit Review Principles All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge. NSF projects, in the aggregate, should contribute more broadly to achieving societal goals. Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects.
Proposal Review Criteria The Intellectual Merit criterion encompasses the potential to advance knowledge; and The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.
Proposal Review Criteria The following elements should be considered in the review for both criteria: What is the potential for the proposed activity to: advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and benefit society or advance desired societal outcomes (Broader Impacts)? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? Is the plan for carrying out the proposed activities well-reasoned, well- organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? How well qualified is the individual, team, or institution to conduct the proposed activities? Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?
Proposal Review Criteria NSF staff will also give careful consideration to the following in recommending funding decisions: Integration of Research and Education: level of engagement in joint efforts that infuse education with the excitement of discovery and enrich research through the diversity of learning perspectives. This is particularly emphasized in the CAREER solicitation Integrating Diversity into NSF Programs, Projects, and Activities: broadening opportunities and enabling the participation of all citizens women and men, underrepresented minorities, and persons with disabilities is essential to the health and vitality of science and engineering.
Proposal Review Criteria Integration of Research and Education: All CAREER proposals must have an integrated research and education plan at their core. Does the PI think creatively about how his or her research will impact their education goals and, Conversely, how does his or her education activities feed back into his or her research. (These plans should reflect both the proposer's own disciplinary and educational interests and goals, as well as the needs and context of his or her organization.)
Panel Discussion Protocol (for some programs within CISE) For Each Proposal L (Lead) opens the discussion by presenting the objectives of the research, the strengths of the proposed research, and weaknesses of the proposed research R1, R2 (Reviewers 1 and 2) add to the comments or express any disagreements with the lead reviewer if necessary S (Scribe) makes notes on the panel discussion. The scribe may also add comments or express any disagreements if necessary. The summary should be written in third person The floor is open for discussion. Majority vote will be taken if there is no unanimous agreement, but panel summary will note the disagreement. After all proposals are discussed and placed into categories, the Scribe will draft the Panel Summary, and read the panel summary for comments, editing and approval.
The CAREER Program and Your Reviews Panel Outputs Individual Reviews in FastLane for each proposal OK to modify reviews, including change of rating. Ensure individual reviews for each proposal are on electronic panel system and are correct . Be sure any modifications to reviews are recorded in FastLane! These MUST be made BEFORE leaving your panel. Panel summary for each proposal Initially framed by one reviewer who serves as scribe using the provided template. Should reflect discussion (not just restate individual reviews). Includes short, clear comments to help unsuccessful PIs improve their proposals in the next competition. Add Justification for Recommendation" heading at the end of the summary and write an informative, concise justification (1-2 sentences). Should be written in 3rd-person and proof-read by all assigned panelists.
The CAREER Program and Your Reviews Panel Outputs Panelist grades: E, V, G, F, P Avoid being overly harsh ( I never give an E ) or overly generous. Be discriminative & use the entire spectrum P .. E if appropriate Panel recommendations: Highly Competitive (HC): Solid proposal, worth working on. Competitive (C): Good proposal, but some portions unconvincing. Low Competitive (LC): As is the proposal is weak, but contains good ideas. Not Competitive (NC): there are major flaws and PI is discouraged from resubmitting Panel recommendation is based on insights gained during discussion Funding a project with F or P rating and declining one with E rating requires explanation by PD. And remember . CISE will compete for PECASE nominations. Identify potential PECASE candidates i.e. this proposal is especially meritorious on both Intellectual Merit and Broader Impact. Assign your ratings accordingly.
Panel Summary Outline Please follow the format in writing panel summaries. Description of project (brief): Intellectual Merit: Strengths: Weaknesses: Do not merely list reviewers comments. Please address the educational components and activities as well as the research activities. Broader Impacts: Strengths: Weaknesses: Should reflect the whole discussion to be useful to NSF and PI Constructive suggestions for improvement: Crisp comments to help unsuccessful PIs improve their proposals for the next competition. HC should not have excessively negative comments. Data management plan Postdoc mentoring plan (if applicable) Justification(s) for panel s recommendation, including key strengths and critical weaknesses : The panel placed this proposal in the following category: ____ Highly Competitive ____ Competitive ____ Low Competitive ____ Not Competitive Comments in reviews and panel summaries should be constructive, informative, non- inflammatory, and non- discriminatory The summary was read by the panel, and the panel concurred that the summary accurately reflects the panel discussion. Only letters of collaboration that prescribe to the GPG format can be considered.
Please Remember! Reviews and panel summaries are sent to Principal Investigators feedback, laudatory or critical, is important comments should be constructive, informative, and non-inflammatory Results are advisory and confidential do not discuss proposals or results proposals may contain sensitive information and are not in the public domain -- do not copy, distribute or quote from them PLEASE DELETE THE PROPOSALS AND NOTES FROM ALL ELECTRONIC DEVICES AND SHRED PAPER COPIES!
Minimizing Bias in Evaluation Implicit bias toward a group ( schemas ) Non-conscious hypotheses/stereotypes, often about competence Lack of critical mass greater reliance on schemas Few women and minorities in sciences Accumulation of disadvantage Small bias in same direction has large effect over time Very small differences in treatment can have major consequences in salary, promotion and prestige [Valian (1998)]
Schemas are Widely culturally shared All people, even members of under-represented groups, hold schemas about these groups People are often not aware of them Applied more under circumstances of: Lack of information Stress from competing tasks Time pressure Lack of critical mass Fiske (2002). Current Directions in Psychological Science, 11, 123-128.
Example: Impact of Blind-Auditions Based on audition records of 14,000 individuals & rosters of orchestras from 1970-1996: The audition data show the use of a screen increases the probability that a woman will advance from preliminary rounds by 50% The roster data show the switch to blind auditions accounts for 30% of the increase in the proportion of women among new hires Goldin & Rouse (2000) The American Economic Review, 90, 4, 715-741.
Evaluation of Identical CVs: Race Jamal had to send 15 resumes to get a callback, compared to 10 needed by Greg. Jamal Greg yielded as many more callbacks as an additional eight years of experience for Jamal. Greg The higher the resume quality, the higher the gap between callbacks for Greg and Jamal. Bertrand & Mullainathan (2004) Poverty Action Lab, 3, 1-27.
Evaluation of Fellowship Applications Average rating of applicants as a function of their scientific productivity the success rate of female scientists applying for postdoctoral fellowships at the [Swedish Medical Research Council] during the 1990s has been less than half that of male applicants. Wenneras & Wold (1997) Nature, 387, p. 341 Women had to be 2.5 times more productive to receive the same competence score. 3.00 Competence Score males females 2.75 2.50 2.25 2.00 0-19 20-39 40-59 60-99 >99 Total impact* One impact point = one paper in a journal with an impact factor of one. Similar findings: GAO report on Peer Review in Federal Agency Grant Selection (1994); and European Molecular Biology Organization Reports (2001) *Cited by Richard Zare, Stanford chemistry professor and former NSB chair, editorial in 5/15/06 Chemistry and Engineering News
Ways to Mitigate Evaluation Bias 1. 2. 3. 4. Increase awareness of how schemas might bias evaluation. Decrease time pressure and distractions in evaluation process. Rate on explicit criteria rather than global judgments. Point to specific evidence supporting judgments. Bauer & Baltes, 2002, Sex Roles, 47 (9/10), 465-476 Please incorporate (3) & (4) in your discussions
