Evaluating Genomic Science: Insight into Genome Canada's Mission and Impact
This conversation with Dr. Samantha Evans, Director of Evaluation at Genome Canada, delves into the importance of program evaluation in genomic science. Genome Canada aims to connect ideas, invest in science and technology, and maximize impacts across sectors. The evaluation process involves collecting evidence on program outcomes to assess performance and inform decision-making, ultimately striving to build a thriving genomics enterprise in Canada.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
EVALUATING GENOMIC SCIENCE: A CONVERSATION WITH DR. SAMANTHA EVANS DIRECTOR OF EVALUATION, GENOME CANADA May 2014 August 25, 2024
SESSION OUTLINE Genomic science, performance measurement and evaluation: Genome Canada overview of mission and current programs Program Evaluation what is it, why is it important and how it is changing Genome Canada s approach to evaluation Outcomes and benefits how is this defined by GC 2
GENOME CANADAS MISSION We are driven by a mission to: connect ideas and people across public and private sectors to find new uses for genomics. investing in large-scale science and technology (S&T) to fuel innovation, and translate discoveries into applications to maximize impacts across all sectors. 3
CANADA HAS AN UNPRECEDENTED OPPORTUNITY TO PLAY A WORLD LEADERSHIP ROLE IN THE NEW BIOECONOMY 5 Given the excellence of our knowledge base in the life science sectors, our vibrant genomics enterprise, and our nation's footprint in natural resources, agriculture and other bio-based resources, Canada can claim a disproportionately-large share of the $1 trillion emerging global bioeconomy. How we know if we are successful in achieving our mission?
WE NEED TO SHOW WE ACT AS A CATALYST BUILDING A THRIVING GENOMICS ENTERPRISE IN CANADA THAT DELIVERS TANGIBLE IMPACTS 6
EVALUATION 7 Evaluation Program design and delivery is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results . TBS, Policy on Evaluation, April 2009 Collection of performance data & information on outputs & outcomes Feedback into program development Evaluation/ analysis of evidence
THE OLD APPROACH Performance monitoring and evaluation in S&T environments has traditionally focused on the quality and quantity of the science produced, the outputs. For example, metrics you may be familiar with: Number of publications Number of citations Number of highly qualified personnel trained This happens because: It s relatively easy to collect, It s what funders seem to ask for, It s what everyone does. BUT IT DOESN T ANSWER THE SO WHAT? QUESTION 8
THINGS ARE CHANGING Why is this important for your research? Funders need to ensure programs & projects delivering on the promises. If we can t demonstrate that what we fund is relevant; well administered; and has impact Policy makers increasingly need to know their program investments provide them with the results they need. Funding opportunities for your research Federal /Provincial Funding GC Programs 9
HOW IS IT SHIFTING? 10 There is impetus for S&T organizations to demonstrate: Quality of the science (checked by peer review/ due diligence, ongoing monitoring etc etc) BUT ALSO Success along the innovation continuum; Translational benefits (i.e. commercialization, tech transfer, & value-added impacts to society beyond science); and Longitudinal outcomes.
THE PRESSURE TO DEMONSTRATE OUTCOMES AND IMPACTS IS BEING FELT BY ALL FUNDERS 11 Industry Canada s S&T portfolio funds the following partner organizations: 1) Canadian Institute for Advanced Research; 2) Canada Foundation for Innovation; 3) CANARIE; 4) Genome Canada; 5) Perimeter Institute for Theoretical Physics; 6) Institute for Quantum Computing; and; 7) Mitacs. WE ALL STRUGGLE WITH HOW BEST TO DEMONSTRATE OUTCOMES AND IMPACT. THERE IS NO QUICK OR EASY FIX OR CLEAR DIRECTION FORWARD GC is working with these organisations and IC to: Develop common performance indicators; Develop common definitional understandings of Develop a community of practice for evaluation
THEORY OF CHANGE Where S&T funders have traditionally focused Intermediate Outcomes or Immediate Outcomes Outcomes Ultimate Activities Outputs Impact Inputs Where we need to now focus
THEORY OF CHANGE 13 It s a road map for an initiative/program/intervention: Defines all the building blocks required to bring about a given long-term goal Usually depicted as a graphic representation of the change process. (logic model) Articulates the assumptions that stakeholders use to explain the change process Assumptions explain both the connections between early, intermediate and long term outcomes and the expectations about how and why proposed interventions will bring them about Provides a commonly understood vision of the long-term goals, how they will be reached, and what will be used to measure progress along the way.
A SYSTEMS APPROACH TO OUTCOME AND IMPACT EVALUATION We are trying to establish an evaluation function based on systems approach: Understanding the role and relationship of components with the system as a whole Holistic, a change in part of a system affects the whole system The whole is more than the sum of the parts. A common starting-point and/or the use of different roads may lead to different results. Circular, a causal connection is not linear
A SYSTEMS APPROACH In practice that means: Using frameworks to identify the complexity of component and their relationships; Paying careful attention to how boundaries are drawn (does not mean including everything); Use of a balanced approach to collecting performance data, before, during and after program design; Considers processes as well as outcomes and impacts; Uses evaluation for both learning and accountability; Quantitative and qualitative needed to properly support decision-making 15
HOW GC HAS BEEN SETTING UP A SYSTEMATIC EVALUATION FRAMEWORK 16 Genome Canada Strategy Mission and Objectives Logic Model Performance Measurement Framework Evaluation Work Plan Development Performance Measurement Strategy Data collection and analysis Application Inform Communication Societal Perspective Accountability Stakeholder Perspective Management Organizational Perspective Risk mitigation Informing decision making Assessment of performance Monitoring social and economic outcomes Inform Public, Genomic Enterprise, & Media on Genome Canada s performance Accountability to the Board Accountability to Government Accountability to Canadian public
WHAT GC IS WORKING TOWARDS 17 As a funder GC has an opportunity to lead the way to be innovative in the way it evaluates the research it funds We look at evaluation requirements at the different stages of a Program lifecycle Program development Program life Program closure
WHAT GC IS WORKING ON 18 Program Life Program Development Program Closure Collecting performance metrics with the Centres- ongoing through project life cycle Using final reports to undertake program or thematic reviews Mapping expected program outcomes & impacts while developing RFA s Feeding data/information on outcomes into the development of new competitions Mining interim reviews & Research Oversight Committee work for outcomes information Developing better definitions for social and economic benefits upfront Exploring initiatives to capture outcome data post funding Annual reporting of metrics to Industry Canada Evaluating the science through peer review Exploring ways of working with Comms and Centres in collecting great outcomes stories of research Considering separate RFA s for evaluation research concurrent with funded programs.
WHAT ARE THE CHALLENGES TO THIS NEW APPROACH TO EVALUATION? 19 We want to find ways to measure: This will need: new ideas, new methods, new ways of working together. Success along the innovation continuum; Translational benefits (i.e. commercialization, tech transfer, & value-added impacts to society beyond science). Longitudinal outcomes There are plenty of challenges.
SUCCESS ALONG THE INNOVATION CONTINIUM
WEVE DESIGNED A PORTFOLIO OF PROGRAMS ALONG THE INNOVATION CONTINUUM 21
MULTIPLE STAKEHOLDERS MULTIPLE SECTORS MULTIPLE NEEDS 22 Multiple Stakeholders Multiple Needs Accountability Genome Centres, Industry Strategic development Canada, GC Board of Directors, Engagement Research Community, GC Staff, Communication Media, Policy Makers, Evaluation End-user Multiple Sectors Forestry Fishery and aquaculture Mining Agriculture Health Environment
MULTIPLE FOCI OVER TIME Evaluating long term Social, Economic & Environmental Impact Capturing Intangible Translation & Diffusion Pathways Complexity of measurement Capturing Tangible Scientific and Social, Economic & Environmental Outcomes Detailing Scientific Outputs Time 23
ATTRIBUTION VERSUS CONTRIBUTION Attribution is relatively clear with measuring outputs, more tricky with both intermediary and long term outcomes and impacts; Observed change in outcomes may be the result of many other factors other than GC funding- other programs- economic factors-social trends etc etc; Need to shift thinking less about precision of attribution and more about increasing understanding and knowledge of our contribution
THEN: THE TRUTH BEHIND THE HYPE Evaluation is about measuring reality
METHODOLOGICAL CHALLENGES Substantive evaluation of the scientific/innovation process relatively new to researchers Accountability has been mainly administrative rather than substantial Introducing metrics/data collection and evaluation causes changes in practise consequences not always known Lack of evaluation budgets often limit risk-taking in data collection and experimenting with cutting edge methodologies. Capturing significant impacts and outcomes outside academia and unintended outcomes. 26
BENEFITS: SOCIAL AND ECONOMIC 27 Challenge is we can t give a quick and easy definition of S & E benefits because every program and every project is different Even though it affects: How objectives are set for RFA How applications are prepared How progress is monitored through the life of the project How reports are done at the end of the project and How outcomes and impacts are demonstrated that often come to fruition post funding Our approach has been to keep it broad to allow researchers flexibility to interpret and apply to their research. Is this still sufficient???
ABC: BENEFITS FOR CANADA IN 28 THE RFA To be realized within five (5) years from the end of the project potential benefits could include: job creation and economic growth in Canada, development of a product or service, (a clear commercialization process, which includes IP management and ownership, technology transfer and benefit sharing, must be described) an impact on society, quality of life, better health, and a cleaner environment, knowledge generation or translation, potential licenses and/or new start-ups or the creation of new policies.
ABC: DIRECTION ON OUTCOMES IN 29 THE FINAL REPORT We tried to provide direction with some questions. Do they help??? Explain how the project has (or will in the future) transfer, disseminate, use, and/or apply the potential deliverables from the research to realize the anticipated benefits. For example, key new products and/or services developed, commercialization processes put in place (including IP management and ownership, technology transfer and benefit sharing). How far along the innovation pipeline would you describe the project? What role or function has been played by the co-funders or end-users involved in the project and what impact has this had?
ABC: DIRECTION ON OUTCOMES IN 30 THE FINAL REPORT Describe the integration between the GE3LS research and the overall goals of the project. What opportunities and challenges has the project faced in this area? What were the principal means of knowledge dissemination or translation into policy and practice and how successful were these? If the project had a chance to do it over again , what other or different GE3LS research would the project team undertake? Briefly describe further research or development work which could be undertaken as a direct result of this project and the possible outcome(s). Describe future collaborations that the project team may establish. Are there any anticipated next steps for the research?
ABC- WHAT WE ASKED FOR IN THE 31 BENEFITS TO CANADA SECTION OF FINAL REPORT Potential benefits could include: job creation and economic growth in Canada; development of a product or service; an impact on society, quality of life, better health, and a cleaner environment; knowledge generation or translation; or potential licenses and/or new start-ups or the creation of new policies that have advanced the following key areas: securing Canada s future bio-based economy feedstock optimization sustainable processing value-added bioproducts crops for a healthy population agriculture and food production sustainability
LSARP 2012 PERSONALISED HEALTH 32 In our recent Personalised Heath Competition we stated: lead to social and/or economic benefits for Canada in terms of contributing to a more evidence-based approach to health and a more cost-effective health-care system. These benefits could include, for example, adoption of a new technology, a change in clinical practice guidelines, an application of an existing drug to a new indication, or a reduction in the number of adverse drug reactions. In addition, there could be other positive impacts on society, the economy (e.g., development of products with commercial potential), quality of life, or the environment. Still a lot of emphasis on the economic benefits in the applications though!!!!!!
LSARP 2012 PERSONALISED HEALTH CRITERIA FOR ASSESSING SEB 33 Deliverables To what extent have the applicants identified appropriate deliverables in terms of their potential to have eventual clinical utility and/or practical applicability? What is the probability that the deliverables will be achieved by the end of the funding period? Expected Benefits Significance of anticipated benefits contributing to a more evidence-based approach to health and improved cost- effectiveness of the health-care system? Is the expected timeframe of the anticipated benefits reasonable?
LSARP 2012 PERSONALISED HEALTH 34 CRITERIA FOR ASSESSING SEB Strategy for Realizing Benefits How persuasive is the strategy set out for realizing benefits from their research? How strong is the plan for knowledge translation and development of benefits, i.e., how well does the plan explain the next steps of how the deliverables from the research will be transferred, disseminated, used, and/or applied to realize the socio-economic benefits? How convincing is the rationale (including an economic component) explaining how the outcomes of the project will potentially contribute to a more evidence-based approach to health and a more cost-effective health-care system?
LSARP 2012 PERSONALISED HEALTH 35 CRITERIA FOR ASSESSING SEB Expertise for Realizing Benefits How appropriate is the expertise of the team that will further develop and implement the strategy for realizing benefits? To what extent are likely end-users involved in the project and the strategy to realize benefits? If the strategy includes commercialization, to what extent has appropriate technology transfer expertise been included? How strong is the expertise assembled to ensure appropriate attention to ethical, environmental, economic, legal and social factors in the process of realizing benefits?
WORKING TOGETHER 36 We are all trying to figure this out together We can be adaptive It has to work for all stakeholders, the projects, the funders etc etc There will never be a clear model/ process or system to follow We are open to idea, thoughts, feedback as we work our way through this. Start anticipating the potential benefits of your next application early in the process- outcome mapping
