Insights from Serving on a Mental Health Research Committee

 
Lessons learned on a study
section
 
My background
 
Biostatistician by training
Served a three-year term (2016-2019) on the NIMH Mental Health
Services Research Committee (SERV)
SERV “will review applications focused on research topics in mental
health services, including delivery and financing of services;
accessibility, use, quality, cost, and outcomes of services; the impact
of health and insurance policy changes; and dissemination and
implementation of evidence-based and promising new preventive,
treatment, and services interventions.”
Have also reviewed on other NIH study sections and PCORI too
 
SERV
 
Observational studies
Administrative data
Sensor data
Social network data
Disease areas: depression, autism, schizophrenia, suicide
R01s, R21s, Ks
I usually don’t get assigned to R21s
 
Timeline
 
SERV meets 3 times per year
About 1.5 months prior to a meeting, the SRO sends a list of all
applications that will be reviewed to identify conflicts of interest:
Members of your institution
People you have published with, etc.
A week after that, receive a list of applications to review
SERV meets for 1 day. I get about 6 applications
Assigned as either 1
st
, 2
nd
, 3
rd
, or 4
th
 reviewer
 
Timeline cont’d
 
Even though I have 1 month to review 6 applications, I always wait
until the last week
Spend 4-5 hours per application. Applications are around 200 pages
Comments are due (uploaded on ERA Commons) 4-5 days before in-
person meeting in DC.
Reviewers are expected to read the comments of other reviewers
who reviewed the same grants they did
In DC, the applications scoring in the top half are discussed. The rest
are “triaged.” Applicants get scores and comments only.
 
How I review a grant
 
Scoring sheet. Need to fill out strengths and weakness based on 5
criteria
Significance
Investigator(s)
Innovation
Approach
Environment
For each criterion assign a score from 1 (best) to 9 (worst)
Provide overall impact score which is a weighted (your own weights)
score of the 5 individual scores
 
How I review a grant cont’d
 
Summary: I’m trying to fill out the sheet. Looking for strengths and
weakness. Make it easy for the reviewer to find strengths:
Say, “The strengths of this application are….”
Make the strengths (bolded) sub-headings under each of the criteria
Avoid weakness or maybe mention them and then say why they are less
important or not fatal flaws
 
My grant review steps, cont’d
 
My order of review
Abstract:
Project narrative
Facilities
Biosketches, budget justification:
Budget
Then I print out the Specific Aims and Research strategy and read
carefully, making note of strengths and weaknesses, flipping back to
other sections of proposal as needed
Rest: Human subjects, women and minorities, children, resource
sharing plan
 
What I’m looking for:
 
Significance
Usually give this a high score, but not a lot of weight in overall impact score
SERV topics tend to be important. Also, I don’t have the subject-matter
expertise to really assess this
Investigator(s)
Has the PI run large projects before? Do they have previous grants? Does the
PI have a good publication record? Have Investigators worked together and
published together before? Do they have sufficient time allocated to the
project? Are they at the same institution or spread out all over the US? Is
there a Biostatistician on the project for an adequate amount of time?
 
Innovation
 
From NIH:
“Does the application challenge and seek to shift current research or
clinical practice paradigms by utilizing novel theoretical concepts,
approaches or methodologies, instrumentation, or interventions? Are
the concepts, approaches or methodologies, instrumentation, or
interventions novel to one field of research or novel in a broad sense?
Is a refinement, improvement, or new application of theoretical
concepts, approaches or methodologies, instrumentation, or
interventions proposed?”
 
Innovation cont’d
 
If innovation is not required to answer the research question then (I
think) that is okay and a low innovation score will not hurt the overall
impact score
But one certainly wants to use up to date methods, designs,
technologies, etc.
 
Approach
 
Meat of the proposal
As a Biostatistician, I focus on design and analyses methods
Figures are good
White space is good
Headers are good
 
Environment
 
Usually ok for most applications
Should be 
targeted
 to the proposal
Annoying when applications just dump 40 pages of boilerplate
 
In person meeting
 
Each grant is given around 15 minutes
There are around 30 people in study section but only 4 have been
assigned to each application
For each grant, we begin by each reviewer providing their overall
summary score.
Then 1
st
 reviewer introduces the application, providing a summary of
investigators and project and then their impressions of the grant,
Remaining reviewers then give their impressions also noting where
they agree and disagree with other reviewers
 
In person meeting cont’d
 
Often, when it is the 4
th
 reviewers turn, they might just say, “I have
nothing to add.” This is encouraged in order to keep things on
schedule
Reviewers then discuss the application, asking each other questions,
getting clarification etc.
Then rest of the panel has opportunity to ask questions, give
comments etc. Other panel members often look at specific aims page,
some might read a grant not assigned to them if it looks interesting
Then the Chair will provide a summary of application, strengths and
weakness. And will ask 4 reviewers for the final scores
 
In person meeting cont’d
 
Finally, everyone on panel records their final overall impact score
NOTE: all final overall impact scores must be within the range
provided by the 4 reviewers. So if reviewers score 2’s and 3’s, panel
members cannot give a score outside of this range
If they want to they can, but most provide some verbal justification
Corollary: Over the course of the day, as it gets harder to pay
attention, panel members may just score the average of the 4
reviewers. So one bad reviewer score can bring down an application
 
3 tips for providing statistical
support to grant proposals
 
Tip 1
 
Read the whole grant including the statistical analysis plan. Need to
understand the study in order to provide appropriate methods. Really
important that the methods can answer the research question. It is
hard to know what the research question is if you haven’t read the
grant.  Some things you can learn by reading the grant:
The PI does not know what the research question is either
The effect size is likely to be very small and you need to take account of this in
the power analysis or suggest a different study design
The trial is better framed as a non-inferiority trial
 
Tip 2
 
If a clinical trial, make use of the statistical analysis plan
No page limit. A detailed plan is sign of diligence and reflects well on the
study team. The statistical reviewer (probably no one else) will flip to it if they
are unclear on something in the main grant and it is great if they find the
answer to their question. Don’t just copy and paste what is already in the
grant.
Power analyses based on a range of effect sizes
Detailed mediation plan
Detailed plan for missing data
Equations ok too.
 
Tip 3
 
Avoid boilerplate language. This is easy to identify, is not always
appropriate, and is boring to read. Also a sign of laziness.
I see boilerplate language a lot in grants with machine learning aims. Reads
like a laundry list of different methods that have not been modified to the
current aims
Mediation analyses too.
 
Questions?
 
Slide Note
Embed
Share

Reflecting on my experience serving on the NIMH Mental Health Services Research Committee (SERV) and reviewing applications focused on mental health services, this piece discusses the process, timelines, types of studies reviewed, and grant reviewing criteria followed. The narrative provides valuable insights into the review process and the responsibilities of committee members in evaluating research proposals in the mental health field.


Uploaded on Jul 14, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Lessons learned on a study section

  2. My background Biostatistician by training Served a three-year term (2016-2019) on the NIMH Mental Health Services Research Committee (SERV) SERV will review applications focused on research topics in mental health services, including delivery and financing of services; accessibility, use, quality, cost, and outcomes of services; the impact of health and insurance policy changes; and dissemination and implementation of evidence-based and promising new preventive, treatment, and services interventions. Have also reviewed on other NIH study sections and PCORI too

  3. SERV Observational studies Administrative data Sensor data Social network data Disease areas: depression, autism, schizophrenia, suicide R01s, R21s, Ks I usually don t get assigned to R21s

  4. Timeline SERV meets 3 times per year About 1.5 months prior to a meeting, the SRO sends a list of all applications that will be reviewed to identify conflicts of interest: Members of your institution People you have published with, etc. A week after that, receive a list of applications to review SERV meets for 1 day. I get about 6 applications Assigned as either 1st, 2nd, 3rd, or 4threviewer

  5. Timeline contd Even though I have 1 month to review 6 applications, I always wait until the last week Spend 4-5 hours per application. Applications are around 200 pages Comments are due (uploaded on ERA Commons) 4-5 days before in- person meeting in DC. Reviewers are expected to read the comments of other reviewers who reviewed the same grants they did In DC, the applications scoring in the top half are discussed. The rest are triaged. Applicants get scores and comments only.

  6. How I review a grant Scoring sheet. Need to fill out strengths and weakness based on 5 criteria Significance Investigator(s) Innovation Approach Environment For each criterion assign a score from 1 (best) to 9 (worst) Provide overall impact score which is a weighted (your own weights) score of the 5 individual scores

  7. How I review a grant contd Summary: I m trying to fill out the sheet. Looking for strengths and weakness. Make it easy for the reviewer to find strengths: Say, The strengths of this application are . Make the strengths (bolded) sub-headings under each of the criteria Avoid weakness or maybe mention them and then say why they are less important or not fatal flaws

  8. My grant review steps, contd My order of review Abstract: Project narrative Facilities Biosketches, budget justification: Budget Then I print out the Specific Aims and Research strategy and read carefully, making note of strengths and weaknesses, flipping back to other sections of proposal as needed Rest: Human subjects, women and minorities, children, resource sharing plan

  9. What Im looking for: Significance Usually give this a high score, but not a lot of weight in overall impact score SERV topics tend to be important. Also, I don t have the subject-matter expertise to really assess this Investigator(s) Has the PI run large projects before? Do they have previous grants? Does the PI have a good publication record? Have Investigators worked together and published together before? Do they have sufficient time allocated to the project? Are they at the same institution or spread out all over the US? Is there a Biostatistician on the project for an adequate amount of time?

  10. Innovation From NIH: Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

  11. Innovation contd If innovation is not required to answer the research question then (I think) that is okay and a low innovation score will not hurt the overall impact score But one certainly wants to use up to date methods, designs, technologies, etc.

  12. Approach Meat of the proposal As a Biostatistician, I focus on design and analyses methods Figures are good White space is good Headers are good

  13. Environment Usually ok for most applications Should be targeted to the proposal Annoying when applications just dump 40 pages of boilerplate

  14. In person meeting Each grant is given around 15 minutes There are around 30 people in study section but only 4 have been assigned to each application For each grant, we begin by each reviewer providing their overall summary score. Then 1streviewer introduces the application, providing a summary of investigators and project and then their impressions of the grant, Remaining reviewers then give their impressions also noting where they agree and disagree with other reviewers

  15. In person meeting contd Often, when it is the 4threviewers turn, they might just say, I have nothing to add. This is encouraged in order to keep things on schedule Reviewers then discuss the application, asking each other questions, getting clarification etc. Then rest of the panel has opportunity to ask questions, give comments etc. Other panel members often look at specific aims page, some might read a grant not assigned to them if it looks interesting Then the Chair will provide a summary of application, strengths and weakness. And will ask 4 reviewers for the final scores

  16. In person meeting contd Finally, everyone on panel records their final overall impact score NOTE: all final overall impact scores must be within the range provided by the 4 reviewers. So if reviewers score 2 s and 3 s, panel members cannot give a score outside of this range If they want to they can, but most provide some verbal justification Corollary: Over the course of the day, as it gets harder to pay attention, panel members may just score the average of the 4 reviewers. So one bad reviewer score can bring down an application

  17. 3 tips for providing statistical support to grant proposals

  18. Tip 1 Read the whole grant including the statistical analysis plan. Need to understand the study in order to provide appropriate methods. Really important that the methods can answer the research question. It is hard to know what the research question is if you haven t read the grant. Some things you can learn by reading the grant: The PI does not know what the research question is either The effect size is likely to be very small and you need to take account of this in the power analysis or suggest a different study design The trial is better framed as a non-inferiority trial

  19. Tip 2 If a clinical trial, make use of the statistical analysis plan No page limit. A detailed plan is sign of diligence and reflects well on the study team. The statistical reviewer (probably no one else) will flip to it if they are unclear on something in the main grant and it is great if they find the answer to their question. Don t just copy and paste what is already in the grant. Power analyses based on a range of effect sizes Detailed mediation plan Detailed plan for missing data Equations ok too.

  20. Tip 3 Avoid boilerplate language. This is easy to identify, is not always appropriate, and is boring to read. Also a sign of laziness. I see boilerplate language a lot in grants with machine learning aims. Reads like a laundry list of different methods that have not been modified to the current aims Mediation analyses too.

  21. Questions?

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#