Merit Review Process at NSF for STEM Education

 
NSF
Merit Review
Proposal creation
Mike Erlinger
Program Director:  July 2014 to July 2016
NSF Division of Undergraduate Education (DUE)
Education and Human Resources Directorate
(EHR
)
PW Commandments
 
 
11/23/2024
1
•
All information provided here represents the
opinions of individuals
•
T
h
e
 
o
n
l
y
 
o
f
f
i
c
i
a
l
 
s
o
u
r
c
e
 
f
o
r
 
N
S
F
 
p
o
l
i
c
y
 
i
s
p
u
b
l
i
s
h
e
d
 
m
a
t
e
r
i
a
l
s
h
t
t
p
:
/
/
n
s
f
.
g
o
v
/
Important!
11/23/2024
PW Commandments
2
M
e
r
i
t
 
R
e
v
i
e
w
o
f
P
r
o
p
o
s
a
l
s
11/23/2024
PW Commandments
3
 Merit Review
1)
Two Criteria Specified
1)
Intellection Merit
2)
Broader Impacts
2)
Others in Play
1)
Ed Research in DUE
2)
Solicitation specific Requirements
3)
Well Defined Evaluation Process
1)
Diverse Panels
2)
PO Comments
3)
Review by management
11/23/2024
PW Commandments
4
•
NSF: 
www.nsf.gov
•
Guide to Program:
www.nsf.gov/funding/browse_all_funding.jsp
•
Award Information: 
www.nsf.gov/awardsearch
•
FastLane: 
www.fastlane.nsf.gov
•
Broader Impacts: 
www.nsf.gov/pubs/gpg/broaderimpacts.pdf
•
Data Management Plan: 
www.nsf.gov/bfa/dias/policy/dmp.jsp
•
Funding Opportunities: 
www.nsf.gov/funding
•
Education Research: 
www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf
•
Program Web Page
U
s
e
f
u
l
 
R
e
s
o
u
r
c
e
s
2013
PW Commandments
NSF/IES guidelines “intended to
improve the quality, coherence,
and pace of knowledge
development in STEM education”
11/23/2024
6
•
The 
Common Guidelines
 describe the roles of different
types of R & D projects in generating 
evidence
 about
strategies and interventions for enhancing student
learning
•
For each type of R & D, the 
Common Guidelines
 describe:
Purpose
Empirical and theoretical justifications
Types of project outcomes
Quality of evidence
Missing ??
Common Guidelines
PW Commandments
11/23/2024
7
T
h
e
 
R
e
v
i
e
w
 
P
r
o
c
e
s
s
11/23/2024
PW Commandments
8
NSF Merit Review Process
11/23/2024
PW Commandments
9
NSF Peer Review Process
•
Reviewers are invited by program directors
–
Number of proposals determines the number of panels
–
Panels types:  Face to face or virtual
–
5-6 reviewers/panel
–
12-14 proposals/panel
•
Reviewers write individual reviews and
assign ratings for 
all
 assigned proposals
before
 the panel convenes
•
Panels typically meet over a 2 day period
11/23/2024
PW Commandments
10
Overview of the Review
•
The review expresses the reviewers 
expert
opinions
 about the quality of the proposed projects
•
Use the rating to indicate overall evaluation of the
proposal
•
Organize review into sections for IM and BI
•
For each section (IM and BI) use the text to
–
Describe the evidence supporting the rating
–
Identify concerns
–
Provide suggestions for improvement
11/23/2024
PW Commandments
11
Review Material
•
The entire proposal is used to inform reviewers
–
Project Summary Project Description
–
Biographical sketches
–
Budget
–
Supplementary documentation, etc
•
But Summary & Description are focus of Review
11/23/2024
PW Commandments
12
Parts of the Review
•
Rating
–
Excellent, Very Good, Good, Fair, and Poor
•
Text - evidence supporting the rating presented
in terms of
–
Intellectual Merit (IM)
–
Broader Impacts (BI)
11/23/2024
PW Commandments
13
Panel Review Meeting
•
Program officer facilitates the panel
•
Proposals are discussed individually
•
Each proposal will have at least 3, typically 4,
reviews
•
A “scribe” is designated to capture all of the
points brought up in discussion and produce a
summary review – called the “Panel Summary”
•
The scribe will often start the discussion
•
All reviewers return on day 2 to review and
approve the Panel Summaries
11/23/2024
PW Commandments
14
Panel Summary
•
Purpose
–
to capture thoughts not expressed in
individual reviews
–
to indicate areas of general agreement and
disagreement
•
Summary does not contain a rating
11/23/2024
PW Commandments
15
Practical Aspects of Review
Process
Reviewers have:
•
Many proposals  (ten or more from several
areas)
•
Limited time for each proposal
•
Different experiences in review process
–
Veterans to novices
•
Different levels of knowledge in proposal area
–
Experts to outsiders
•
Discussions of proposals’ merits at panel
meeting
–
Share expertise and experience
11/23/2024
PW Commandments
16
Audience for Reviews
•
NSF program directors
–
Informs recommendations relative to funding
–
Guides pre-award negotiations
•
Applicants
–
If proposal is funded:
•
Provides suggestions for improving project
–
If proposal is not funded:
•
Provides information to guide a revision of the
proposal
11/23/2024
PW Commandments
17
The Proposer Receives…
R
e
v
i
e
w
s
P
a
n
e
l
 
S
u
m
m
a
r
y
(
i
f
 
a
p
p
l
i
c
a
b
l
e
)
C
o
n
t
e
x
t
 
s
t
a
t
e
m
e
n
t
&
A
w
a
r
d
/
D
e
c
l
i
n
a
t
i
o
n
 
l
e
t
t
e
r
 
&
11/23/2024
PW Commandments
18
Confidentiality
•
What you read and discuss as panelists
–
do not quote or use
–
leave all proposals behind
–
do not discuss outside of panel
•
Don’t reveal your identity to the PIs
–
they  receive anonymous copies of  all reviews
–
don’t discuss  reviews once you leave
11/23/2024
PW Commandments
19
P
r
o
p
o
s
a
l
 
C
r
e
a
t
i
o
n
O
b
v
i
o
u
s
 
C
o
m
m
a
n
d
m
e
n
t
s
11/23/2024
PW Commandments
20
Don’t Wait!!
11/23/2024
PW Commandments
21
Remember  This!!!
Grant writing is not an exercise in creative
writing!!
11/23/2024
PW Commandments
MERGE Solicitation Specs with Your Project
    Change your Project NOT Solicitation!!!
22
Guidelines: Proposal Structure
•
Follow the solicitation and PPAPG Adhere
to page, font size, and margin limitations
•
Use allotted space but don’t pad the proposal
•
NOW Auto Checked!!!
–
Follow suggested (or implied) organization
–
Use appendices sparingly (check solicitation
to see if allowed)
–
Include letters showing 
commitments
 from
others
•
Avoid form letters
11/23/2024
PW Commandments
23
Guidelines: Writing
•
Use good style (clarity, organization, etc.)
–
Be concise, but complete
–
Write simply but professionally
–
Avoid jargon and acronyms
–
Check grammar and spelling - Fastlane
–
Use sections, headings, short paragraphs &
bullets (Avoid dense, compact text)
–
Provide appropriate level of detail
–
Poorly formatted proposals are hard to read
11/23/2024
PW Commandments
24
Guidelines: Proposal Writing
•
Reinforce your ideas
•
Summarize; highlight with judicious use of bolding,
italics
•
Give examples
•
Use tables, figures – where it makes
sense – Write to the Figure!!!
•
Reference your referencs
11/23/2024
PW Commandments
25
Guidelines: Your Audience
•
Community: About 5 reviewers
–
Usually in your discipline (not always your
field)
–
Do not assume they are experts in what you
are proposing to do
–
You need to provide enough detail for them to
understand what you want to do
•
NSF: Program Officer
•
Public: Funded Proposals
11/23/2024
PW Commandments
26
Guidelines: First (Second?) Page
•
The first page of your proposal is the most
important
•
Must sell your audience on your idea and get
them excited about it
•
If you lose them they will spend their time
looking for ways to 
ding
 you
11/23/2024
PW Commandments
27
Project Summary
•
P
a
y
 
s
p
e
c
i
a
l
 
a
t
t
e
n
t
i
o
n
 
t
o
 
P
r
o
j
e
c
t
 
S
u
m
m
a
r
y
–
Plainly state what you plan to do
–
Summarize goals, rationale, methods, and
evaluation and dissemination plans
–
Address intellectual merit and broader impacts
•
Explicitly and independently
–
Three paragraphs with headings:
•
“Summary”
•
“Intellectual Merit”
•
“Broader Impacts”
11/23/2024
PW Commandments
28
M
i
k
e
’
s
 
C
o
m
m
a
n
d
m
e
n
t
s
11/23/2024
PW Commandments
29
Me:
11/23/2024
PW Commandments
MANY  Solicitations have specific REQUIREMENTS
 HIDDEN!
30
M
E
:
 
 
R
e
v
i
e
w
 
F
u
n
d
e
d
 
P
r
o
j
e
c
t
s
A
b
s
t
r
a
c
t
s
 
o
f
 
R
e
c
e
n
t
 
A
w
a
r
d
s
11/23/2024
PW Commandments
31
ME: Build on Past Projects
Use Google!!!
•
You have done the literature search, make sure to use it.
Most good ideas are based on past ideas and maybe
even the same idea. Find the references.
•
Make sure you project reflects any Best Practices, e.g.,
any K12 in service PD requires X hours to be of value.
•
Look for references with proven approaches and
evidence.
•
If your approach is new or better, explain why. Ed
Research is the new emphasis of DUE. Figure out what it
means for what you want to do. Find and READ,
“Common Guidelines for Education Research and
Development”
11/23/2024
PW Commandments
32
ME: It Is All Research
•
NSF Motto says:
–
Supporting Education and Research across all the
fields of Science, Mathematics and Technology.
•
The NSF has made it clear that Education must
including validation of learning based on
Educational Research. This requires a good Ed
Research justification for any new proposal,
i.e., no more anecdotal ‘good idea’
11/23/2024
PW Commandments
33
ME: Education Research Question
•
In DUE Always required
. You cannot just gather data,
rather you need to have a purpose for that data and
then be able to show some learning progress.
•
This is probably the most important feature of new
proposals. Creating such a question is something that
probably goes beyond the technical PI.
•
Ed Research is the 
primary
 concern of EHER/DUE
(Division of Education).
–
Figure out what it means for what you want to do.
–
Find and READ, “Common Guidelines for Education
Research and Development”
11/23/2024
PW Commandments
34
Common Guidelines for Ed Research
•
Developed by the Department of Education
and the NSF
•
Search
–
Common Guidelines for Education Research and
Development
•
Defines 6 types of research
•
Reading will put you in the proper “mindset”
11/23/2024
PW Commandments
35
ME: Second Reader
•
The largest mistake that I have noticed in my ‘rejected’
proposals is the lack of a internal ‘2nd Reader’. Proposal
writers do NOT get someone in their department or
field to read their proposal nor do they get someone
outside their particular emphasis to read their proposal.
•
Panel summaries of rejected proposals almost always
refer to the lack of clarity.
–
The classic is: “very interesting concept, no idea what is
actually being proposed”
•
Also, someone needs to catch the grammar and spelling
issues, etc.  - FastLane HATES smart quotes!!!
11/23/2024
PW Commandments
36
ME: This is Research So Treat it as Such
•
Read the Literature, Determine Adequate
References
•
Define Hypotheses based on literature and your
ideas
•
Design an experiment to test Hypothesis, i.e.,
your Proposal
•
Independently, Determine assessment questions
•
Gather Data
•
Analyze Outcomes
•
Tell the world
11/23/2024
PW Commandments
37
ME: Turn in Reports from Prior Awards
•
You cannot have new money if you owe
reports on old money. True of the PI and any
CoPI, and now institutions.
•
Realize it may take some time for a program
officer to approve a report, so last minute
Reports are an issue.
11/23/2024
PW Commandments
38
ME: Write to the Figure
•
Figures and tables are powerful, but the text
needs to match the figure. After you insert the
figure make sure that the text really does go
along with the figure.
11/23/2024
PW Commandments
39
ME: Commitment Letter 
not
 Support
Letters
•
The classic prewritten letter for numerous
people to say they like your proposal is
useless. Letters need to show some level of
commitment. Usually such letters will show
program sustainment after NSF support,
institution support, etc. NSF NOW HAS
SPECIFIC WORDING
11/23/2024
PW Commandments
40
ME: Project Goals, Objectives,
Management and Sustainability
•
Make CLEAR. Leave no doubt about what will be
accomplished; who will accomplish what;  and
what happens when NSF goes away
•
The schedule needs to be defined and detailed
enough that reviewers can follow your plan.
Tables are good, but be sure to reference and
clarify them in the text.
•
Reviews like a table showing responsibilities and
schedule
11/23/2024
PW Commandments
41
ME: Support for Sustainability,
Institution, Department, Others…
•
Are you alone? Is your department and
institution going to support you? Are there
others who will assist in your project. Saying
the institution will sustain a project without
some letter from the institution to that effect
is an issue.
•
Sustainability (what happens when NSF $ is
gone) is important. Institution support
reflecting sustainability is important.
11/23/2024
PW Commandments
42
ME: Details of Prior Support
•
If there is prior support, make sure to provide
details of success or failure. If this is a
continuation, make sure to show how it
continues the previous effort. Sentence saying
“Our previous project was fantastic” does not
cut it.
11/23/2024
PW Commandments
43
ME: Broader Impacts should be Broad
•
...should contribute more broadly to achieving
societal goals...
•
May be accomplished through the research itself,
through activities directly related to the research,
or through activities that are supported by and
complementary to the research.
•
Historically in Computer Science, Broader Impacts
have meant underrepresented groups, but while
necessary that idea is limiting. Consider, Citizen
Science, Workforce Development, etc
. Think
Globally
11/23/2024
PW Commandments
44
ME: Volunteer for Panel Reviews
•
Panels are an important way to learn how the
review process works. Also they are a way to
learn about NSF. You get paid. More panels
are moving to Virtual which reduces travel,
but also panel interaction.
11/23/2024
PW Commandments
45
ME: Evaluation
•
The NSF has moved quickly through the steps:
–
from: no evaluation requirement
–
to : you need to have an evaluation or assessment
–
to : you need an evaluator outside your department
–
to: you should (must) have external evaluation which
includes both evaluation and an Ed Research question.
•
For many of the large continuing Solicitations, e.g., STEM+C, NSF
supports an outside firm to act as a source of evaluation and
data management activities. So far these companies are
‘voluntary’ for PIs, but the future is clearly non-voluntary. In
some cases, NSF may soon tell you the evaluation instruments
(at a minimum) to be used.
11/23/2024
PW Commandments
46
ME: Data Management Plan Needs to
be Real
•
Long term use of data should be part of
preparation, e.g., someone may want to look at
an evaluation of all the CS Principles courses. NSF
wants to get your data for future use.
•
This could be accomplished in a number of ways,
but you need to know that data management is
extremely important to be able to justify the $s
being spent, e.g.,
 
–
http://www5.hmc.edu/ITNews/?p=2322
–
http://libguides.libraries.claremont.edu/dmp
11/23/2024
PW Commandments
47
ME: No Tricky Titles
•
Congress is ever present with features like:
Waste Book.
•
Catchy titles like ‘penetration testing’ for a
security proposal are not going to fly.
11/23/2024
PW Commandments
48
ME: Who is the PI?
•
A technical proposal requires technical
expertise from the start. Think about who
should be on the cover as PI and CoPi.
11/23/2024
PW Commandments
49
ME: Dissemination
•
More than a web site!!! NSF has supported
the development of a vast number of teaching
improvements, ed research, etc. How will the
world know what you have done?
•
eBooks are a new approach gaining traction.
11/23/2024
PW Commandments
50
ME: Common Review Panel
Comments…
•
Failure to reference existing ed research and learning
references
•
Failure to reference existing similar technical projects
•
Failure to adequately address evaluation and assessment
•
Over 50% of my declines said: 
“Prior to submitting any
future proposal, you might consult the NSF documents,
‘Common Guidelines for Education Research and
Development’ and ‘Grant Proposal Guide’”
.
•
Many declines also said
: “It is not clear why the approach
advocated here is preferable or fundamentally different
(more learning) than similar existing projects.”
11/23/2024
PW Commandments
51
S
u
m
m
a
r
y
11/23/2024
PW Commandments
52
What makes a good proposal?
•
A good idea that is a significant improvement
•
A capable person or team to do the project
•
Time, equipment, technical support to carry out
the project
•
Preliminary work helps
•
Knowing the literature -- what has been done
elsewhere -- and building on it
11/23/2024
PW Commandments
53
What makes a good proposal?
•
Put the project in context -- local and national
•
Be specific about what is to be done and who
will do it
•
Describe what the project will accomplish -
products
•
Describe evaluation and dissemination
appropriate to the project
•
Relate the budget to what is to be done
11/23/2024
PW Commandments
54
To Become a Reviewer
The best way to learn how to write a great
proposal is review proposals, i.e.,  serve on a
review panel
–
Let NSF PO know you are interested in reviewing
–
Mention the specific program(s) you wish to review for
–
Each program maintains its own list of interested
reviewers
11/23/2024
PW Commandments
55
•
Submit Proposals
•
Serve as Reviewers and Panelists
•
Be Active as Workshop Participants
and Organizers
•
Consider Being a Rotator
http://www.nsf.gov/about/career_opps/rotators/index.jsp
For information on a particular EHR division and program, go to the EHR website
and choose a division.
 
http://www.nsf.gov/dir/index.jsp?org=EHR
  
Contact NSF Program Directors for questions and suggestions.
S
t
a
y
 
C
o
n
n
e
c
t
e
d
Additional Information
•
NSF Proposal Processing and Review
(includes Review Criteria)
http://www.nsf.gov/pubs/policydocs/pappgui
de/nsf11001/gpg_3.jsp
•
Merit Review
http://nsf.gov/bfa/dias/policy/meritreview/
•
Merit Review Facts
http://nsf.gov/bfa/dias/policy/meritreview/fac
ts.jsp
11/23/2024
PW Commandments
57
Questions?
11/23/2024
PW Commandments
58
Most Critical Concern for ALL EHR
Programs
 
11/23/2024
PW Commandments
59
P
r
o
p
o
s
a
l
 
D
e
v
e
l
o
p
m
e
n
t
 
a
n
d
 
P
r
o
c
e
s
s
i
n
g
 
T
i
m
e
l
i
n
e
Div. Dir. Concur
Award
DGA Review &
Processing of  Award
Proposal Preparation Time
Proposal received by NSF
NSF
Review of Proposal
PO Recommend
STEM education R & D projects that contribute to
core knowledge
Type #1: 
Foundational
 test, develop, or refine models of learning or teaching
 generate fundamental understandings related to learning
and education
Type #2: 
Early stage/exploratory
 examine relationships among constructs to establish
logical connections that may form the basis for future
interventions to improve education outcomes
Common Guidelines
PW Commandments
11/23/2024
61
Type #3: 
Design and development
 draw on existing models and theories
 iteratively design, develop, and test interventions (new
curricula, approaches to teaching and learning,
applications of technology, practices, policies)
Common Guidelines
PW Commandments
11/23/2024
62
Impact Studies: assess impacts of 
fully-developed
intervention on education-related outcome(s)
Type #4: 
Efficacy studies
: estimate the impacts
under “ideal” conditions (e.g., developer
involvement, higher levels of support than typical)
Type #5: 
Effectiveness studies
: examine
implementation and estimate impacts similar to
routine practice but on a limited scale
Type #6: 
Scale-up studies
: explore implementation
and estimate impacts under conditions that prevail
under wide-scale adoption
Common Guidelines
PW Commandments
11/23/2024
63
Helpful Hints
1.
Read the Program Solicitation (several
times)
2.
Work on projects you care deeply
about
3.
Build on what others have done
64
Helpful Hints (continued)
4.
Think global - act local and global
5.
Have measurable goals and
objectives
6.
Think teamwork
65
Helpful Hints (continued)
7.
Use Good Management Skills
8.
Evaluation is impact and effectiveness
9.
Spread the word
10.
Pay back time
66
10 Fatal Flaws
1.
Assume deadlines are not enforced
2.
Assume page limits and font size restrictions
don’t matter
3.
Substitute flowery rhetoric for good examples
4.
Don’t check your speeling nore you’re
grammer
5.
Assume program guidelines have not changed,
or better yet, ignore th
em.
67
10 Fatal Flaws (continued)
6.
Assert evaluation will be ongoing and consist
of a variety of methods
7.
Assume a website is sufficient for
dissemination
8.
Assume your past accomplishments are well
known
9.
Provide “support” rather than “commitment”
letters
10.
Inflate your budget to allow for negotiations
68
Last words...
•
Think beyond the material,  to 
affect
 and the
broader implications
 of what we do.
•
Study whether our innovations are effective,
and share them with others
69

								

								

								

										

												
Slide Note

												

										

								


								
						

							
						


							

								

									
										
Embed
									
									
										
Share
									

									
									
									

								

								
							


							

							 
 							 
 							 
						  


							
							

								
This presentation provides valuable insights into the Merit Review Process at NSF, focusing on STEM education initiatives. It covers key guidelines, criteria for proposal evaluation, important resources, and the overall review process. The content emphasizes the importance of intellectual merit and broader impacts, as well as the role of diverse panels in ensuring a well-defined evaluation process.


								
    
																		
  •   NSF
    • 
																	
  •    Merit Review
    • 
																	
  •    STEM Education
    • 
																	
  •    Proposal Evaluation
    • 
																	
  •    Research Process
    • 
																

								

								

									

										nale_nea
										

											 nale_nea Follow
											

										

									

									
Uploaded on Nov 23, 2024 | 0 Views

								

								


								

									
Download Presentation

									
								

								
Please find below an Image/Link to download the presentation.

								
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.


									
									

									


										 
										 
										 
										 

									

									

									

								
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

								

									 
									
Download Presentation
								

								
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.


								

									
 E N D 

								


							
Presentation Transcript

							

							
  1. NSF Merit Review Proposal creation Mike Erlinger Program Director:  July 2014 to July 2016 NSF Division of Undergraduate Education (DUE) Education and Human Resources Directorate  (EHR) PW Commandments 11/23/2024 1
    2. 
            
            
            
  2. Important!     All information provided here represents the  opinions of individuals     The only official source for NSF policy is  published materials http://nsf.gov/ 11/23/2024 PW Commandments 2
    3. 
            
            
            
  3. Merit Review of  Proposals 11/23/2024 PW Commandments 3
    4. 
            
            
            
  4. Merit Review  1)Two Criteria Specified 1)Intellection Merit 2)Broader Impacts 2)Others in Play 1)Ed Research in DUE 2)Solicitation specific Requirements 3)Well Defined Evaluation Process 1)Diverse Panels 2)PO Comments 3)Review by management 11/23/2024 PW Commandments 4
    5. 
            
            
            
  5. Useful Resources     NSF: www.nsf.gov     Guide to Program:  www.nsf.gov/funding/browse_all_funding.jsp      Award Information: www.nsf.gov/awardsearch     FastLane: www.fastlane.nsf.gov     Broader Impacts: www.nsf.gov/pubs/gpg/broaderimpacts.pdf     Data Management Plan: www.nsf.gov/bfa/dias/policy/dmp.jsp     Funding Opportunities: www.nsf.gov/funding      Education Research: www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf     Program Web Page
    6. 
            
            
            
  6. NSF/IES guidelines intended to  improve the quality, coherence,  and pace of knowledge  development in STEM education    2013 PW Commandments 11/23/2024 6
    7. 
            
            
            
  7. Common Guidelines     The Common Guidelines describe the roles of different  types of R & D projects in generating evidence about  strategies and interventions for enhancing student  learning     For each type of R & D, the Common Guidelines describe:     Purpose     Empirical and theoretical justifications     Types of project outcomes     Quality of evidence     Missing ?? PW Commandments 11/23/2024 7
    8. 
            
            
            
  8. The Review Process 11/23/2024 PW Commandments 8
    9. 
            
            
            
  9. NSF Merit Review Process 11/23/2024 PW Commandments 9
    10. 
            
            
            
  10. NSF Peer Review Process     Reviewers are invited by program directors     Number of proposals determines the number of panels     Panels types:  Face to face or virtual     5-6 reviewers/panel     12-14 proposals/panel     Reviewers write individual reviews and  assign ratings for all assigned proposals  before the panel convenes     Panels typically meet over a 2 day period 11/23/2024 PW Commandments 10
    11. 
            
            
            
  11. Overview of the Review     The review expresses the reviewers expert  opinions about the quality of the proposed projects     Use the rating to indicate overall evaluation of the  proposal     Organize review into sections for IM and BI     For each section (IM and BI) use the text to     Describe the evidence supporting the rating     Identify concerns      Provide suggestions for improvement 11/23/2024 PW Commandments 11
    12. 
            
            
            
  12. Review Material     The entire proposal is used to inform reviewers     Project Summary Project Description     Biographical sketches     Budget     Supplementary documentation, etc     But Summary & Description are focus of Review 11/23/2024 PW Commandments 12
    13. 
            
            
            
  13. Parts of the Review     Rating     Excellent, Very Good, Good, Fair, and Poor     Text - evidence supporting the rating presented  in terms of     Intellectual Merit (IM)     Broader Impacts (BI) 11/23/2024 PW Commandments 13
    14. 
            
            
            
  14. Panel Review Meeting     Program officer facilitates the panel     Proposals are discussed individually     Each proposal will have at least 3, typically 4,  reviews     A    scribe    is designated to capture all of the  points brought up in discussion and produce a  summary review    called the    Panel Summary        The scribe will often start the discussion     All reviewers return on day 2 to review and  approve the Panel Summaries 11/23/2024 PW Commandments 14
    15. 
            
            
            
  15. Panel Summary     Purpose     to capture thoughts not expressed in  individual reviews     to indicate areas of general agreement and  disagreement     Summary does not contain a rating 11/23/2024 PW Commandments 15
    16. 
            
            
            
  16. Practical Aspects of Review  Process Reviewers have:     Many proposals  (ten or more from several  areas)     Limited time for each proposal     Different experiences in review process     Veterans to novices     Different levels of knowledge in proposal area      Experts to outsiders      Discussions of proposals    merits at panel  meeting     Share expertise and experience 11/23/2024 PW Commandments 16
    17. 
            
            
            
  17. Audience for Reviews     NSF program directors     Informs recommendations relative to funding     Guides pre-award negotiations     Applicants     If proposal is funded:     Provides suggestions for improving project     If proposal is not funded:     Provides information to guide a revision of the  proposal 11/23/2024 PW Commandments 17
    18. 
            
            
            
  18. The Proposer Receives NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia  22230 Dear Dr. Doe, NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia  22230 The National Science Foundation hereby  awards a grant of...  Dear Dr. Doe, & I regret to inform you that the National  Science Foundation is unable to support your  proposal referenced above... $ Context statement  &  Award/Declination letter Reviews Panel Summary (if applicable) 11/23/2024 PW Commandments 18
    19. 
            
            
            
  19. Confidentiality     What you read and discuss as panelists     do not quote or use     leave all proposals behind     do not discuss outside of panel     Don   t reveal your identity to the PIs     they  receive anonymous copies of  all reviews     don   t discuss  reviews once you leave 11/23/2024 PW Commandments 19
    20. 
            
            
            
  20. Proposal Creation Obvious Commandments 11/23/2024 PW Commandments 20
    21. 
            
            
            
  21. Dont Wait!! 11/23/2024 PW Commandments 21
    22. 
            
            
            
  22. Remember  This!!! Grant writing is not an exercise in creative  writing!! MERGE Solicitation Specs with Your Project     Change your Project NOT Solicitation!!! 11/23/2024 PW Commandments 22
    23. 
            
            
            
  23. Guidelines: Proposal Structure     Follow the solicitation and PPAPG Adhere  to page, font size, and margin limitations     Use allotted space but don   t pad the proposal     NOW Auto Checked!!!     Follow suggested (or implied) organization     Use appendices sparingly (check solicitation  to see if allowed)     Include letters showing commitments from  others     Avoid form letters 11/23/2024 PW Commandments 23
    24. 
            
            
            
  24. Guidelines: Writing     Use good style (clarity, organization, etc.)     Be concise, but complete     Write simply but professionally     Avoid jargon and acronyms     Check grammar and spelling - Fastlane     Use sections, headings, short paragraphs &  bullets (Avoid dense, compact text)     Provide appropriate level of detail     Poorly formatted proposals are hard to read 11/23/2024 PW Commandments 24
    25. 
            
            
            
  25. Guidelines: Proposal Writing     Reinforce your ideas     Summarize; highlight with judicious use of bolding,  italics     Give examples     Use tables, figures     where it makes  sense     Write to the Figure!!!     Reference your referencs 11/23/2024 PW Commandments 25
    26. 
            
            
            
  26. Guidelines: Your Audience     Community: About 5 reviewers     Usually in your discipline (not always your  field)     Do not assume they are experts in what you  are proposing to do     You need to provide enough detail for them to  understand what you want to do     NSF: Program Officer     Public: Funded Proposals  11/23/2024 PW Commandments 26
    27. 
            
            
            
  27. Guidelines: First (Second?) Page     The first page of your proposal is the most  important     Must sell your audience on your idea and get  them excited about it     If you lose them they will spend their time  looking for ways to ding you 11/23/2024 PW Commandments 27
    28. 
            
            
            
  28. Project Summary     Pay special attention to Project Summary      Plainly state what you plan to do     Summarize goals, rationale, methods, and  evaluation and dissemination plans      Address intellectual merit and broader impacts      Explicitly and independently     Three paragraphs with headings:        Summary            Intellectual Merit           Broader Impacts    11/23/2024 PW Commandments 28
    29. 
            
            
            
  29. Mikes Commandments 11/23/2024 PW Commandments 29
    30. 
            
            
            
  30. Me: MANY  Solicitations have specific REQUIREMENTS  HIDDEN! 11/23/2024 PW Commandments 30
    31. 
            
            
            
  31. ME:  Review Funded Projects Abstracts of Recent Awards 11/23/2024 PW Commandments 31
    32. 
            
            
            
  32. ME: Build on Past Projects Use Google!!!     You have done the literature search, make sure to use it.  Most good ideas are based on past ideas and maybe  even the same idea. Find the references.      Make sure you project reflects any Best Practices, e.g.,  any K12 in service PD requires X hours to be of value.     Look for references with proven approaches and  evidence.      If your approach is new or better, explain why. Ed  Research is the new emphasis of DUE. Figure out what it  means for what you want to do. Find and READ,     Common Guidelines for Education Research and  Development    11/23/2024 PW Commandments 32
    33. 
            
            
            
  33. ME: It Is All Research     NSF Motto says:     Supporting Education and Research across all the  fields of Science, Mathematics and Technology.     The NSF has made it clear that Education must  including validation of learning based on  Educational Research. This requires a good Ed  Research justification for any new proposal,  i.e., no more anecdotal    good idea    11/23/2024 PW Commandments 33
    34. 
            
            
            
  34. ME: Education Research Question     In DUE Always required. You cannot just gather data,  rather you need to have a purpose for that data and  then be able to show some learning progress.     This is probably the most important feature of new  proposals. Creating such a question is something that  probably goes beyond the technical PI.     Ed Research is the primary concern of EHER/DUE  (Division of Education).      Figure out what it means for what you want to do.      Find and READ,    Common Guidelines for Education  Research and Development    11/23/2024 PW Commandments 34
    35. 
            
            
            
  35. Common Guidelines for Ed Research     Developed by the Department of Education  and the NSF     Search     Common Guidelines for Education Research and  Development     Defines 6 types of research     Reading will put you in the proper    mindset    11/23/2024 PW Commandments 35
    36. 
            
            
            
  36. ME: Second Reader     The largest mistake that I have noticed in my    rejected     proposals is the lack of a internal    2nd Reader   . Proposal  writers do NOT get someone in their department or  field to read their proposal nor do they get someone  outside their particular emphasis to read their proposal.      Panel summaries of rejected proposals almost always  refer to the lack of clarity.     The classic is:    very interesting concept, no idea what is  actually being proposed         Also, someone needs to catch the grammar and spelling  issues, etc.  - FastLane HATES smart quotes!!! 11/23/2024 PW Commandments 36
    37. 
            
            
            
  37. ME: This is Research So Treat it as Such     Read the Literature, Determine Adequate  References     Define Hypotheses based on literature and your  ideas     Design an experiment to test Hypothesis, i.e.,  your Proposal     Independently, Determine assessment questions     Gather Data     Analyze Outcomes     Tell the world 11/23/2024 PW Commandments 37
    38. 
            
            
            
  38. ME: Turn in Reports from Prior Awards     You cannot have new money if you owe  reports on old money. True of the PI and any  CoPI, and now institutions.     Realize it may take some time for a program  officer to approve a report, so last minute  Reports are an issue. 11/23/2024 PW Commandments 38
    39. 
            
            
            
  39. ME: Write to the Figure     Figures and tables are powerful, but the text  needs to match the figure. After you insert the  figure make sure that the text really does go  along with the figure. 11/23/2024 PW Commandments 39
    40. 
            
            
            
  40. ME: Commitment Letter not Support  Letters     The classic prewritten letter for numerous  people to say they like your proposal is  useless. Letters need to show some level of  commitment. Usually such letters will show  program sustainment after NSF support,  institution support, etc. NSF NOW HAS  SPECIFIC WORDING 11/23/2024 PW Commandments 40
    41. 
            
            
            
  41. ME: Project Goals, Objectives,  Management and Sustainability     Make CLEAR. Leave no doubt about what will be  accomplished; who will accomplish what;  and  what happens when NSF goes away     The schedule needs to be defined and detailed  enough that reviewers can follow your plan.  Tables are good, but be sure to reference and  clarify them in the text.     Reviews like a table showing responsibilities and  schedule 11/23/2024 PW Commandments 41
    42. 
            
            
            
  42. ME: Support for Sustainability,  Institution, Department, Others        Are you alone? Is your department and  institution going to support you? Are there  others who will assist in your project. Saying  the institution will sustain a project without  some letter from the institution to that effect  is an issue.     Sustainability (what happens when NSF $ is  gone) is important. Institution support  reflecting sustainability is important. 11/23/2024 PW Commandments 42
    43. 
            
            
            
  43. ME: Details of Prior Support     If there is prior support, make sure to provide  details of success or failure. If this is a  continuation, make sure to show how it  continues the previous effort. Sentence saying     Our previous project was fantastic    does not  cut it. 11/23/2024 PW Commandments 43
    44. 
            
            
            
  44. ME: Broader Impacts should be Broad      ...should contribute more broadly to achieving  societal goals...     May be accomplished through the research itself,  through activities directly related to the research,  or through activities that are supported by and  complementary to the research.     Historically in Computer Science, Broader Impacts  have meant underrepresented groups, but while  necessary that idea is limiting. Consider, Citizen  Science, Workforce Development, etc. Think  Globally 11/23/2024 PW Commandments 44
    45. 
            
            
            
  45. ME: Volunteer for Panel Reviews     Panels are an important way to learn how the  review process works. Also they are a way to  learn about NSF. You get paid. More panels  are moving to Virtual which reduces travel,  but also panel interaction. 11/23/2024 PW Commandments 45
    46. 
            
            
            
  46. ME: Evaluation     The NSF has moved quickly through the steps:      from: no evaluation requirement     to : you need to have an evaluation or assessment     to : you need an evaluator outside your department     to: you should (must) have external evaluation which  includes both evaluation and an Ed Research question.     For many of the large continuing Solicitations, e.g., STEM+C, NSF  supports an outside firm to act as a source of evaluation and  data management activities. So far these companies are     voluntary    for PIs, but the future is clearly non-voluntary. In  some cases, NSF may soon tell you the evaluation instruments  (at a minimum) to be used. 11/23/2024 PW Commandments 46
    47. 
            
            
            
  47. ME: Data Management Plan Needs to  be Real     Long term use of data should be part of  preparation, e.g., someone may want to look at  an evaluation of all the CS Principles courses. NSF  wants to get your data for future use.     This could be accomplished in a number of ways,  but you need to know that data management is  extremely important to be able to justify the $s  being spent, e.g.,      http://www5.hmc.edu/ITNews/?p=2322     http://libguides.libraries.claremont.edu/dmp 11/23/2024 PW Commandments 47
    48. 
            
            
            
  48. ME: No Tricky Titles     Congress is ever present with features like:  Waste Book.      Catchy titles like    penetration testing    for a  security proposal are not going to fly. 11/23/2024 PW Commandments 48
    49. 
            
            
            
  49. ME: Who is the PI?     A technical proposal requires technical  expertise from the start. Think about who  should be on the cover as PI and CoPi. 11/23/2024 PW Commandments 49
    50. 
            
            
            
  50. ME: Dissemination     More than a web site!!! NSF has supported  the development of a vast number of teaching  improvements, ed research, etc. How will the  world know what you have done?     eBooks are a new approach gaining traction. 11/23/2024 PW Commandments 50
    51. 
            
            
            
						

						

					

					
					
					

						
Related

						

						


							
							
							
							

						

							

									

      

        
          

            

              2023-24 Staff Performance Review and Merit
            

            

              
2023-24 Staff Performance Review and Merit

              
daire

            

          

        
      


     

      

        
          

            

              Enhancing STEM Education Through Diversity and Inclusion
            

            

              
Enhancing STEM Education Through Diversity and Inclusion

              
daniella

            

          

        
      


     

      

        
          

            

              JM Sethia Merit Scholarship Scheme
            

            

              
JM Sethia Merit Scholarship Scheme

              
OBCRIGHTS

            

          

        
      


     

      

        
          

            

              A Comprehensive Overview of ATS STEM Framework for Teachers
            

            

              
A Comprehensive Overview of ATS STEM Framework for Teachers

              
fern

            

          

        
      


     

      

        
          

            

              Guide to Scout Advancement and Merit Badges
            

            

              
Guide to Scout Advancement and Merit Badges

              
barack

            

          

        
      


     

      

        
          

            

              STEM STARS Institute - Empowering Future STEM Educators
            

            

              
STEM STARS Institute - Empowering Future STEM Educators

              
midge

            

          

        
      


     

      

        
          

            

              Introduction to STEM Training for Nova and Supernova Awards
            

            

              
Introduction to STEM Training for Nova and Supernova Awards

              
clodagh

            

          

        
      


     

      

        
          

            

              Empowering Future STEM Educators for a Brighter Tomorrow
            

            

              
Empowering Future STEM Educators for a Brighter Tomorrow

              
jasper

            

          

        
      


     

      

        
          

            

              Insights into Aquatics Merit Badges Maintenance and Review Process
            

            

              
Insights into Aquatics Merit Badges Maintenance and Review Process

              
pendragon

            

          

        
      


     

      

        
          

            

              Building Bridges Grant Program for STEM Education
            

            

              
Building Bridges Grant Program for STEM Education

              
rdunk

            

          

        
      


     

      

        
          

            

              NSF Programs Supporting Undergraduate Research Overview
            

            

              
NSF Programs Supporting Undergraduate Research Overview

              
wcro

            

          

        
      


     

      

        
          

            

              Transforming Your School into an Inclusive STEM Learning Center
            

            

              
Transforming Your School into an Inclusive STEM Learning Center

              
ckreu

            

          

        
      


     

      

        
          

            

              Insights from a Season at NSF: Program Director's Perspective
            

            

              
Insights from a Season at NSF: Program Director's Perspective

              
makenz

            

          

        
      


     

      

        
          

            

              Comprehensive Guide to Setting Up a K-6 STEM Laboratory
            

            

              
Comprehensive Guide to Setting Up a K-6 STEM Laboratory

              
gmae

            

          

        
      


     

      

        
          

            

              Transforming STEM Education in Scotland: Key Findings and Recommendations
            

            

              
Transforming STEM Education in Scotland: Key Findings and Recommendations

              
kios888

            

          

        
      


     

      

        
          

            

              Promoting STEM Diversity in Education with STEM Person of the Week
            

            

              
Promoting STEM Diversity in Education with STEM Person of the Week

              
aar_h

            

          

        
      


     

      

        
          

            

              NSF Rejections in Banner System
            

            

              
NSF Rejections in Banner System

              
aalee

            

          

        
      


     

      

        
          

            

              National Science Foundation Division of Undergraduate Education (DUE) Programs
            

            

              
National Science Foundation Division of Undergraduate Education (DUE) Programs

              
sport

            

          

        
      


     

      

        
          

            

              Stem Cells and Cell Potency in Animal Cells
            

            

              
Stem Cells and Cell Potency in Animal Cells

              
amadr

            

          

        
      


     

      

        
          

            

              Recommendations for Advancing STEM Education
            

            

              
Recommendations for Advancing STEM Education

              
isabellar

            

          

        
      


     

      

        
          

            

              The Impact of STEM Ambassadors in Education
            

            

              
The Impact of STEM Ambassadors in Education

              
khigh

            

          

        
      


     

      

        
          

            

              Creating a Diverse STEM Center: Overcoming Challenges and Promoting Inclusion
            

            

              
Creating a Diverse STEM Center: Overcoming Challenges and Promoting Inclusion

              
asha_146

            

          

        
      


     							

						

					

					

				


				

			

		
		

			

				
				

					
More Related Content

					

						

														

					

				

				
			

		












 





  
  
  



giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#