The Importance of Open Science and Reproducibility in Scientific Research

 
Open Science & Reproducibility
 
Jess Butler
University of Aberdeen
Centre for Health Data Science
 
Open Science: 
what is it?
 
Open Science 
means that scientific
knowledge, research data, and publications
should be openly shared.
 
This includes:
measured and observational data,
experimental setups and workflows,
analysis scripts and tools,
analysed data,
metadata and provenance data,
published manuscripts
 
https://en.wikipedia.org/wiki/Open_science
 
https://www.fosteropenscience.eu/content/
what-open-science-introduction
 
Open Science: 
why bother?
 
Research data created
through public funds
should be freely available
for public exploitation
 
Open Science: 
why bother?
 
 
 
Open Science: 
why bother?
 
 
Generate
and specify
hypotheses
 
Design
study
 
Collect data
 
Analyse data
 Test hypotheses
 
Interpret
data
 
Publish or
Conduct next
experiment
 
Chris Chambers
osf.io/dfr85
 
The Scientific Research Cycle
 
The results
 
The results
 
The Results
 
The Results
Don’t touch THIS
But make sure THIS is
amazing
 
Chris Chambers
osf.io/svf98
 
The Scientific Research Cycle: 
why it goes wrong
P
-hacking
 
 
Generate
and specify
hypotheses
 
Design
study
 
Collect data
 
Analyse data
 Test hypotheses
 
Interpret
data
 
Publish or
Conduct next
experiment
 
Publication bias
Lack of data sharing
Low statistical power
Changing the hypothesis
 
Failure to control for bias
Lack of replication
Poor measurement
Poor quality control
 
Chris Chambers
osf.io/dfr85
 
The Scientific Research Cycle: 
where it goes wrong
P
-hacking
 
 
Generate
and specify
hypotheses
 
Design
study
 
Collect data
 
Analyse data
 Test hypotheses
 
Interpret
data
 
Publish or
Conduct next
experiment
1 in 1000 papers
Makel et al (2012)
~50% chance to detect
medium effects
Cohen (1962); Sedlmeier
and Gigerenzer (1989);
Bezeau and Graves
(2001)
~50-100% prevalence
John et al (2012)
~50-90% prevalence
John et al (2012)
Kerr (1998)
~92% positive
 
Fanelli
(2010)
~70% failure
Wicherts et al (2006)
 
Publication bias
Lack of data sharing
Low statistical power
Changing the hypothesis
 
Failure to control for bias
Lack of replication
Poor measurement
Poor quality control
 
Chris Chambers
osf.io/dfr85
 
Imagine you had to show a complete trail of the provenance of
the figures and tables in one of your papers
 
Where is the research?
 
Why might it be hard to replicate?
 
What resources would make this easier?
 
Open Science: 
how?
 
Imagine you have to prove to reviewers/funders that you didn’t
torture your data and get a random “significant” result
 
How would you convince them?
 
What would it look like if you had?
 
What resources would make this easier to prove?
 
Open Science: 
how?
 
Imagine you want a big promotion or a new job
 
How would you show you worked to a high standard?
 
Is this way of working rewarded?
 
What resources would make this easier?
 
Open Science: 
how?
 
Preregistration
Researcher b
li
nding
Open code
Open metadata
O
pen data
R
eporting guidelines
Journal blinding (registered reports)
Preprints
Validation studies
Replication studies
 
abdn.ac.uk/iahs/research/profiles/jessicabutler
 
Open Science: 
how?
 
Imagine you want to work this way but find it terrifying and/or
wildly unlikely to succeed in the academic system as it is?
 
What resources would make this easier?
 
Open Science: 
how?
 
Build a Community
To share experiences about open science, including opportunities, challenges, and tools
 
Raise Awareness
To raise awareness about the benefits of open science practices for research and
researchers
 
Education
To generate tools for helping researchers use open practices
 
Internal Promotion
To lobby for policy reforms within our departments, colleges and universities
 
External Promotion
To drive forward open science policies in journals, funders, public bodies and learned
societies
 
UK Open Research Working Groups Objectives
 
osf.io/vgt3x
 
Open Science: 
help!
 
Join the 
UK Reproducibility Network
 Open Science Foundation 
UK ORWG
 
Survey department/school/university on interests and needs in open science
 
Talk by mentor from UKRN
 
Organise workshop on pre-registration and registered reports
 
Organise training for working reproducibly in 
R
 
Advertise/join collaborative research projects
 
Start a 
ReproducibiliTea
 club
 
Identify resource needs for teaching 
open science practices
 
Get open practices rewarded in recruitment and promotion
 
Open Science: 
organise
 
Jess Butler
University of Aberdeen
Centre for Health Data Science
Slides at: 
https://osf.io/q86y2/
 
@jebs284
jessicabutler@abdn.ac.uk
Slide Note
Embed
Share

Open Science emphasizes the open sharing of scientific knowledge, research data, and publications. It promotes transparency by making data, methodologies, and results accessible to all. This approach helps in reproducibility, ensuring the reliability and integrity of research findings. By addressing issues like bias, lack of replication, and publication bias, Open Science aims to enhance the quality and trustworthiness of scientific research.


Uploaded on Jul 18, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Open Science & Reproducibility Jess Butler University of Aberdeen Centre for Health Data Science

  2. Open Science: what is it? Open Science means that scientific knowledge, research data, and publications should be openly shared. This includes: measured and observational data, experimental setups and workflows, analysis scripts and tools, analysed data, metadata and provenance data, published manuscripts https://www.fosteropenscience.eu/content/ what-open-science-introduction https://en.wikipedia.org/wiki/Open_science

  3. Open Science: why bother? Research data created through public funds should be freely available for public exploitation

  4. Open Science: why bother?

  5. Open Science: why bother?

  6. The Scientific Research Cycle Publish or Conduct next experiment Generate and specify hypotheses Design study Interpret data Analyse data Test hypotheses Collect data Chris Chambers osf.io/dfr85

  7. The Scientific Research Cycle: why it goes wrong Don t touch THIS The results The Results But make sure THIS is amazing The results The Results Chris Chambers osf.io/svf98

  8. The Scientific Research Cycle: where it goes wrong Failure to control for bias Lack of replication Publication bias Lack of data sharing Publish or Conduct next experiment Generate and specify hypotheses Design study Interpret data Low statistical power P-hacking Analyse data Test hypotheses Collect data Poor measurement Poor quality control Chris Chambers osf.io/dfr85

  9. ~92% positive Fanelli (2010) Failure to control for bias Lack of replication Publication bias Lack of data sharing Publish or Conduct next experiment Generate and specify hypotheses 1 in 1000 papers Makel et al (2012) ~70% failure Wicherts et al (2006) ~50-90% prevalence John et al (2012) Kerr (1998) Design study Interpret data Low statistical power ~50% chance to detect medium effects Cohen (1962); Sedlmeier and Gigerenzer (1989); Bezeau and Graves (2001) P-hacking ~50-100% prevalence John et al (2012) Analyse data Test hypotheses Collect data Poor measurement Poor quality control Chris Chambers osf.io/dfr85

  10. Open Science: how? Imagine you had to show a complete trail of the provenance of the figures and tables in one of your papers Where is the research? Why might it be hard to replicate? What resources would make this easier?

  11. Open Science: how? Imagine you have to prove to reviewers/funders that you didn t torture your data and get a random significant result How would you convince them? What would it look like if you had? What resources would make this easier to prove?

  12. Open Science: how? Imagine you want a big promotion or a new job How would you show you worked to a high standard? Is this way of working rewarded? What resources would make this easier?

  13. Open Science: how? Preregistration Researcher blinding Open code Open metadata Open data Reporting guidelines Journal blinding (registered reports) Preprints Validation studies Replication studies abdn.ac.uk/iahs/research/profiles/jessicabutler

  14. Open Science: how? Imagine you want to work this way but find it terrifying and/or wildly unlikely to succeed in the academic system as it is? What resources would make this easier?

  15. Open Science: help! UK Open Research Working Groups Objectives Build a Community To share experiences about open science, including opportunities, challenges, and tools Raise Awareness To raise awareness about the benefits of open science practices for research and researchers Education To generate tools for helping researchers use open practices Internal Promotion To lobby for policy reforms within our departments, colleges and universities External Promotion To drive forward open science policies in journals, funders, public bodies and learned societies osf.io/vgt3x

  16. Open Science: organise Join the UK Reproducibility Network Open Science Foundation UK ORWG Survey department/school/university on interests and needs in open science Talk by mentor from UKRN Organise workshop on pre-registration and registered reports Organise training for working reproducibly in R Advertise/join collaborative research projects Start a ReproducibiliTea club Identify resource needs for teaching open science practices Get open practices rewarded in recruitment and promotion

  17. Jess Butler University of Aberdeen Centre for Health Data Science Slides at: https://osf.io/q86y2/ @jebs284 jessicabutler@abdn.ac.uk

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#