Understanding Metrics for Research Impact Assessment

Metrics: a game of
hide and seek
Eleanor Colla
Researcher Services Librarian
To cover
Metrics overview
Generating
Tracking
Measuring
Part 1- theory
Part 2- doing
Seeking
Presenting
Examples
Resources
Questions &
Discussions
Part 3- casual chats
Metrics overview
Metrics assist in assessing the academic impact of research outputs
Metrics are an indicator that needs to be interpreted
Metrics are numbers that can assist your research narrative
Generating metrics
Publishing
Articles
Books & Book chapters
Conference proceedings
Policy reports
Patents
Public and Community engagement
Consulting (industry)
Radio and television appearances
Community forums and public lectures
Online engagement
Blogs (academic, research, general)
Social media
Library can assist
Strategic publishing
Advice on online engagement
Metrics consultations
Tracking metrics
M
a
i
n
 
t
o
o
l
s
ORCiD
Scopus
Web of Science
Google Scholar
OCLC WorldCat
Altmetric.com
O
t
h
e
r
 
a
n
d
 
e
m
e
r
g
i
n
g
 
t
o
o
l
s
lens.org
PlumX Analytics
Publons
Kudos
Impact Story
Dimensions Data
Measuring- types of metrics
Traditional
Scholarly publications in books and journals
Track citation counts in other 
books and journals
Track quality of the journal/book and publisher
Formulates an h-index
Non-traditional
Scholarly-related outputs in different publication types (ie. grey literatu
re)
Track wider influence in scholarly areas and outputs
Alternative
Scholarly/ general outputs in non-scholarly publications
Track the quantity- not the quality- of your impact and online presence
Good for early career researchers and when there is a short turnaround between publication
and metrics
Main index & citation tools for traditional metrics
Google Scholar
Scopus
Web of
Science
Your
research
outputs
Indexing scope
Measuring- H-index
h-index = number of papers with citation number greater than
or equal to h
Eg. The researcher would have an h-index of 8, as 8 articles have been cited at
least 8 or more times, and the remaining articles have each been cited 8 times or
less.
Use with care:
Citation patterns vary across disciplines, thus
researchers in different disciplines cannot be
compared
Researchers in the same discipline at different
career stages will have different h-index score
Measuring- journal quality
Journal Impact Factor
Based on articles, reports, and proceedings indexed in Web of Science
Calculated by citations received in the year from articles, reviews and proceedings
published in the previous 2 years
SCImago Journal Ranking
Based on articles, reports, and proceedings indexed in Scopus
Calculated by citations received in the year from articles, reviews and proceedings
published in the previous 3 years
Australian Business Deans Council (ABDC) Journal Quality List
Arts and Humanities Citation Index Journal List
European Reference Index for the Humanities
Measuring- article level
Many databases and publisher websites now show
article metrics
Number of views, downloads, and citations
(PlumX, Altmetric.com, Mendeley downloads)
Assigned to papers in Web of Science
Assigned to papers in Scopus
Non-traditional metrics
Government papers and policies
Conferences and presentations
Panels
Scholarly websites and academic blogs (eg, The Conversation, LSE Blog)
News articles
Books in libraries (WorldCat.org, The European Library, COPAC)
Awards received
Patents (lens.org)
Engagement through media (eg. radio programmes, podcasts)
Alternative metrics
Social media (mentions, shares)
Downloads
Publons
Altmetric.com (e-publications@UNE, ProQuest)
PlumX analytic (Ebsco, Scopus)
Seeking the metrics
a list of publications
ORCiD
ResearcherID (if applicable)
Scopus Author ID (if applicable)
Google Scholar profile (if applicable)
Any other author IDs you are using (ResearchGate, Mendeley, Publons, etc)
Any websites/projects/online presence you may have (Kudos, Impact Story, The Conversation, blog)
Presenting- Where
Academic promotions
CV/Resume
Grant funding applications
Two important points
1.
Always cite your metrics
Source
Date
1.
Always be able to provide proof of your claims
Keep a folder of screencaptures
Where applicable
-
C
i
t
a
t
i
o
n
 
m
e
t
r
i
c
s
 
t
a
k
e
n
 
f
r
o
m
 
G
o
o
g
l
e
 
S
c
h
o
l
a
r
 
(
G
S
)
,
 
0
3
/
0
4
/
2
0
1
8
.
 
C
i
t
a
t
i
o
n
s
 
i
n
c
l
u
d
e
 
a
c
a
d
e
m
i
c
a
n
d
 
g
r
e
y
 
l
i
t
e
r
a
t
u
r
e
 
a
n
d
 
m
a
y
 
i
n
c
l
u
d
e
 
d
u
p
l
i
c
a
t
i
o
n
s
.
-
L
i
b
r
a
r
y
 
h
o
l
d
i
n
g
 
m
e
t
r
i
c
s
 
t
a
k
e
n
 
f
r
o
m
 
O
C
L
C
 
W
o
r
l
d
C
a
t
 
(
W
C
)
,
 
0
3
/
0
4
/
2
0
1
8
,
 
i
n
c
l
u
s
i
v
e
 
o
f
 
a
l
l
e
d
i
t
i
o
n
s
 
a
n
d
 
t
y
p
e
 
o
f
 
r
e
s
o
u
r
c
e
-
P
l
e
a
s
e
 
s
e
e
 
g
l
o
s
s
a
r
y
 
f
o
r
 
t
e
r
m
s
 
w
i
t
h
 
a
n
 
a
s
t
e
r
i
x
 
(
*
)
Scholarly book chapters
Author. (year). Title of chapter. In Ed Author (Eds.), 
Book title
 (pp). Place of publication:
Publisher.
GS citation = 10, WC = 357 libraries in 23 countries
Author. (year). Title of chapter. In Ed Author (Eds.), 
Book title
 (pp). Place of publication:
Publisher.
 
GS citation = 7, WC = 250 libraries in 30 countries, indexed in Web of
Science
Refereed journal articles
Author. (year). Title of article, 
Title of Journal
, V(Iss), pp. Doi:
 
GS citation = 8, indexed in Scopus, Journal SJR (2017)- Q2 (Education)
Author. (year). Title of article, 
Title of Journal
, V(Iss), pp. Doi:
G
S
 
c
i
t
a
t
i
o
n
 
=
 
1
5
,
 
i
n
d
e
x
e
d
 
i
n
 
S
c
o
p
u
s
,
 
F
W
C
I
*
 
=
 
2
.
0
0
Glossary
Field Weighted Citation Impact (FWCI): A Scopus metric
where a  FWCI greater than 1.00 means the resource is
more cited than expected based on: 
 
 
       ·  The year of publication
·  Document type, and
·  The disciplines associated with its source
Each discipline makes an equal contribution to the
metric, which eliminates differences in research citation
behaviour.
Scimago Journal Rank (SJR): Calculated on articles,
reports, and proceedings indexed in Scopus being cited
in the previous 3 years
Metrics in grant applications and CVs
Metrics in grant applications and CVs
 
O
v
e
r
a
l
l
 
R
e
s
e
a
r
c
h
 
P
r
o
f
i
l
e
Within my research field, citations, a journal’s impact factor, peer reviewed publications, and book
holdings in libraries are all considered to be good indicators of research impact.
Outputs (as of 03/04/2018)
Total outputs
:11: book chapters (5), books (3), refereed journal articles (3)
Total citations
: 45 (Google Scholar), 15 (Scopus), 7 (Web of Science)
Field of Research Codes
: FOR13 (7 publications), FOR22 (3 publications), FOR08 (2 publications)
Overall, my research has a FWCI* of 1.75. I publish consistently in FOR13 where my
FWCI is 2.00. This indicates that my research in FOR13 is performing twice above the
world average for FOR13 indexed outputs between 2012-2017.
(source:
SciVal)
All data were sourced
on 03/04/2018. Where
applicable the source
has been named.
Metrics in grant applications and CVs
J
o
u
r
n
a
l
 
Q
u
a
l
i
t
y
My three refereed journal articles are all focused on the field of Education and all are
indexed in Scopus. These journals are:
E
d
u
c
a
t
i
o
n
a
l
 
P
o
l
i
c
y
 
(
S
J
R
*
=
 
1
.
6
9
,
 
Q
u
a
r
t
i
l
e
 
1
 
i
n
 
E
d
u
c
a
t
i
o
n
)
Urban Education
 (SJR= 1.43, Quartile 1 in Education, Quartile 1 in Urban Studies)
Quality in Higher Education
 (SJR= 0.61, Quartile 2 in Education).
(source: Scimago)
B
o
o
k
 
i
n
d
e
x
i
n
g
Cumulatively, my three published books and three published book chapters, are held in 850 libraries
across 53 countries (WC). 
Title A
 and 
Title B
 are indexed in Scopus with 
Title A
 having a citation count
of 4 (Scopus) and 3 (Web of Science). 
Chapter Title A
 has been cited by 5 resources indexed in Web
of Science.
      
   
(data sourced on 03/04/2018)
All data were sourced
on 03/04/2018. Where
applicable the source
has been named.
Resources
Leiden Manifesto for Research Metrics: 
http://www.leidenmanifesto.org/
 
Leiden Manifesto Video (4’30’’): 
https://vimeo.com/133683418
REF2014 Impact Case Studies: 
http://impact.ref.ac.uk/CaseStudies/Search1.aspx
Metrics Toolkit: 
http://www.metrics-toolkit.org/
 
“Metrics should support, not supplant, expert judgement”
 
-
The Metric Tide, 2015
http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/The,Metric,Ti
de/2015_metric_tide.pdf
Three key messages
Metrics: a game of hide and seek
Cite your metrics!
Questions
 & Discussions
Book an appointment with us:
libraryresearch@une.edu.au
Slide Note
Embed
Share

Metrics play a crucial role in assessing the academic impact of research outputs. They provide valuable indicators that need to be interpreted to enhance your research narrative. This overview covers generating, tracking, and measuring metrics with examples of tools and strategies for traditional and non-traditional scholarly publications. Learn about key metrics like h-index, citation counts, and journal impact factor, along with tools such as ORCiD, Scopus, and Altmetric.com for effective research evaluation.


Uploaded on Sep 11, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Metrics: a game of hide and seek Eleanor Colla Researcher Services Librarian

  2. To cover Metrics overview Generating Tracking Measuring Part 1- theory Seeking Presenting Examples Resources Part 2- doing Part 3- casual chats Questions & Discussions

  3. Metrics overview Metrics assist in assessing the academic impact of research outputs Metrics are an indicator that needs to be interpreted Metrics are numbers that can assist your research narrative

  4. Generating metrics Publishing Public and Community engagement Consulting (industry) Radio and television appearances Community forums and public lectures Online engagement Blogs (academic, research, general) Social media Library can assist Strategic publishing Advice on online engagement Metrics consultations Articles Books & Book chapters Conference proceedings Policy reports Patents

  5. Tracking metrics Main tools Other and emerging tools ORCiD lens.org Scopus PlumX Analytics Web of Science Publons Google Scholar Kudos OCLC WorldCat Impact Story Altmetric.com Dimensions Data

  6. Measuring- types of metrics Traditional Scholarly publications in books and journals Track citation counts in other books and journals Track quality of the journal/book and publisher Formulates an h-index Non-traditional Scholarly-related outputs in different publication types (ie. grey literature) Track wider influence in scholarly areas and outputs Alternative Scholarly/ general outputs in non-scholarly publications Track the quantity- not the quality- of your impact and online presence Good for early career researchers and when there is a short turnaround between publication and metrics

  7. Main index & citation tools for traditional metrics Benchmarking in InCites Reliable - Doesn t cover all disciplines 66 million+ records 18,000+ journals, books, conference proceedings Strongest coverage of natural sciences, health sciences, engineering, computer science, materials sciences Time period: Sciences: 1900- present Social Sciences: 1900- present A&H: 1975- present Proceedings: 1990- present Books: 2005- present Journal Impact Factor in Journal Citation Reports Web of Science Clarivate Analytics - Strong in the sciences ResearcherID Benchmarking in SciVal Reliable - Doesn t cover all disciplines 69 million records 22,000+ serials 14,000+ journals 150,000+ books Coverage: health sciences (32%), physical sciences (29%), social sciences (24%), life sciences (15%) Time period: SJR in SCImago Journal Ranking Scopus Back dated to 1970 Elsevier - Strong in the sciences Scopus AuthorID - Broader than WoS Analysis in Harzing s Publish or Perish Unreliable Covers most disciplines ??? ??? N/A Google Scholar Google Scholar Profile

  8. Indexing scope Your research outputs Google Scholar Scopus Web of Science

  9. Measuring- H-index h-index = number of papers with citation number greater than or equal to h Eg. The researcher would have an h-index of 8, as 8 articles have been cited at least 8 or more times, and the remaining articles have each been cited 8 times or less. Use with care: Citation patterns vary across disciplines, thus researchers in different disciplines cannot be compared Researchers in the same discipline at different career stages will have different h-index score

  10. Measuring- journal quality Journal Impact Factor Based on articles, reports, and proceedings indexed in Web of Science Calculated by citations received in the year from articles, reviews and proceedings published in the previous 2 years SCImago Journal Ranking Based on articles, reports, and proceedings indexed in Scopus Calculated by citations received in the year from articles, reviews and proceedings published in the previous 3 years Australian Business Deans Council (ABDC) Journal Quality List Arts and Humanities Citation Index Journal List European Reference Index for the Humanities

  11. Measuring- article level Assigned to papers in Scopus Many databases and publisher websites now show article metrics Number of views, downloads, and citations (PlumX, Altmetric.com, Mendeley downloads) Assigned to papers in Web of Science

  12. Non-traditional metrics Government papers and policies Conferences and presentations Panels Scholarly websites and academic blogs (eg, The Conversation, LSE Blog) News articles Books in libraries (WorldCat.org, The European Library, COPAC) Awards received Patents (lens.org) Engagement through media (eg. radio programmes, podcasts)

  13. Alternative metrics Social media (mentions, shares) Downloads Publons Altmetric.com (e-publications@UNE, ProQuest) PlumX analytic (Ebsco, Scopus)

  14. Seeking the metrics a list of publications ORCiD ResearcherID (if applicable) Scopus Author ID (if applicable) Google Scholar profile (if applicable) Any other author IDs you are using (ResearchGate, Mendeley, Publons, etc) Any websites/projects/online presence you may have (Kudos, Impact Story, The Conversation, blog)

  15. Presenting- Where Academic promotions CV/Resume Grant funding applications Two important points 1. Always cite your metrics Source Date 1. Always be able to provide proof of your claims Keep a folder of screencaptures

  16. Metrics in grant applications and CVs Where applicable -Citation metrics taken from Google Scholar (GS), 03/04/2018. Citations include academic and grey literature and may include duplications. -Library holding metrics taken from OCLC WorldCat (WC), 03/04/2018, inclusive of all editions and type of resource -Please see glossary for terms with an asterix (*) Glossary Field Weighted Citation Impact (FWCI): A Scopus metric where a FWCI greater than 1.00 means the resource is more cited than expected based on: The year of publication Scholarly book chapters Author. (year). Title of chapter. In Ed Author (Eds.), Book title (pp). Place of publication: Publisher. GS citation = 10, WC = 357 libraries in 23 countries Document type, and The disciplines associated with its source Each discipline makes an equal contribution to the metric, which eliminates differences in research citation behaviour. Author. (year). Title of chapter. In Ed Author (Eds.), Book title (pp). Place of publication: Publisher. GS citation = 7, WC = 250 libraries in 30 countries, indexed in Web of Science Scimago Journal Rank (SJR): Calculated on articles, reports, and proceedings indexed in Scopus being cited in the previous 3 years Refereed journal articles Author. (year). Title of article, Title of Journal, V(Iss), pp. Doi: GS citation = 8, indexed in Scopus, Journal SJR (2017)- Q2 (Education) Author. (year). Title of article, Title of Journal, V(Iss), pp. Doi: GS citation = 15, indexed in Scopus, FWCI* = 2.00

  17. Metrics in grant applications and CVs Overall Research Profile Within my research field, citations, a journal s impact factor, peer reviewed publications, and book holdings in libraries are all considered to be good indicators of research impact. All data were sourced on 03/04/2018. Where applicable the source has been named. Outputs (as of 03/04/2018) Total outputs:11: book chapters (5), books (3), refereed journal articles (3) Total citations: 45 (Google Scholar), 15 (Scopus), 7 (Web of Science) Field of Research Codes: FOR13 (7 publications), FOR22 (3 publications), FOR08 (2 publications) Overall, my research has a FWCI* of 1.75. I publish consistently in FOR13 where my FWCI is 2.00. This indicates that my research in FOR13 is performing twice above the world average for FOR13 indexed outputs between 2012-2017. (source: SciVal)

  18. Metrics in grant applications and CVs Journal Quality My three refereed journal articles are all focused on the field of Education and all are indexed in Scopus. These journals are: Educational Policy (SJR*= 1.69, Quartile 1 in Education) Urban Education (SJR= 1.43, Quartile 1 in Education, Quartile 1 in Urban Studies) Quality in Higher Education (SJR= 0.61, Quartile 2 in Education). All data were sourced on 03/04/2018. Where applicable the source has been named. (source: Scimago) Book indexing Cumulatively, my three published books and three published book chapters, are held in 850 libraries across 53 countries (WC). Title A and Title B are indexed in Scopus with Title A having a citation count of 4 (Scopus) and 3 (Web of Science). Chapter Title A has been cited by 5 resources indexed in Web of Science. (data sourced on 03/04/2018)

  19. Resources Leiden Manifesto for Research Metrics: http://www.leidenmanifesto.org/ Leiden Manifesto Video (4 30 ): https://vimeo.com/133683418 REF2014 Impact Case Studies: http://impact.ref.ac.uk/CaseStudies/Search1.aspx Metrics Toolkit: http://www.metrics-toolkit.org/

  20. Three key messages Metrics should support, not supplant, expert judgement -The Metric Tide, 2015 http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/The,Metric,Ti de/2015_metric_tide.pdf Metrics: a game of hide and seek Cite your metrics!

  21. Questions & Discussions Book an appointment with us: libraryresearch@une.edu.au

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#