Evaluating Policy Tools for Grand Challenges Framework
Special features of Grand Challenges and trans-national public partnerships in research and innovation are discussed, along with resulting evaluation challenges. A new framework for assessing policy tools aimed at addressing Grand Challenges is proposed, supported by evidence from ERA-LEARN analysis of Public-Private Partnerships.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Towards a new rationale and framework for evaluating policy tools oriented towards dealing with Grand Challenges Effie Amanatidou, Deborah Cox, Dimitri Gagliardi, Chiara Marzocchi, and Mickael Benaim Manchester Institute of Innovation Research AMBS / The Univeristy of Manchester EU-SPRI Conference Paris, 6-8 June 2018
Outline Special features of Grand Challenges Special features of trans-national public partnerships in R&I Resulting challenges in evaluation Supporting evidence from ERA-LEARN analysis of P2Ps A suggested new framework for evaluating policy tools oriented towards dealing with Grand Challenges
Special features of Grand Challenges EU-SPRI Conference Paris, 6-8 June 2018
Special features of Grand Challenges wicked problems : difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize / a problem whose social complexity means that it has no determinable stopping point because of complex interdependencies, the effort to solve one aspect of a wicked problem may reveal or create other problems. (C. West Churchman, 1967) Boundary spanning in several respects: Inter-disciplinarity, multi-disciplinarity, multi-stakeholder relevance, multi-level governance (from regional to international), cross-sectoral / cross-ministerial collaboration
Special features of trans-national Public Partnerships (P2Ps) in R&I EU-SPRI Conference Paris, 6-8 June 2018
What are P2Ps? Public Partnerships (P2Ps) in research and innovation are networks of public organisations (Ministries, funding agencies, programme managers) that join forces to support research activities under an agreed vision or research and innovation agenda. P2Ps aim at aligning national strategies, helping to overcome fragmentation of public research effort. P2Ps include partnerships supported by the European Commission such as ERA-NETs and Art 185s as well as Member State-led initiatives, the Joint Programming Initiatives.
What are P2Ps? As networks they have numerous players, some of whom may leave after a given period of time (e.g. after a Cofund Action ends) while others may join in during the life-time of a network or when a follow-up phase is planned; not all members need to join and commit resources to realise the planned, joint activities; thus there are diverse types and level of membership, engagement and commitment; their success depends on the degree to which the network establishes connections among its members building trust, and long-term commitment. (*) A network is a decentralized member-driven platform of relationships that evolves its capabilities and underlying structure of connectivity. (Network Impact and Centre for Evaluation Innovation, 2014)
Resulting challenges in evaluation EU-SPRI Conference Paris, 6-8 June 2018
Resulting challenges in evaluation Source: Amanatidou et al. 2014
Additional challenges due to P2P network nature Variety in national interests, motivations and expectations Variety in national R&I systems Different types of agencies involved Different types of funding (block or competitive, to either public or private organisations) Different rules of participation and funding inability for funding to cross national borders Different timing and duration of funding and policy-making cycles Different levels of coordination/communication among ministries and across other public organisations and agencies Different levels of national R&I budgets => Different starting points
Additional challenges due to the network nature Variety of interest and conditions when joining P2Ps Research capacities Degree of alignment among national structures, policies and programmes Different levels of available financial contributions and administrative capacities Different readiness levels
Additional challenges due to the network nature Variety in P2P governance structures, mechanisms and processes Governing and Management Boards, Scientific Boards, Stakeholder Boards, etc. Different decision-making processes Fluidity in membership, resources, capacities, and commitments Importance of network health(*): a P2P s ability to engage its members, sustain their engagement, and adapt as needed. May involve issues of trust building and management effectiveness, and network connectivity(*): the extent to which the members ties to each other are resulting in efficient and effective pathways for shared learning and action. (*) Amended from Network Impact Centre for Evaluation, 2014
Evidence from ERA-LEARN analysis of P2Ps EU-SPRI Conference Paris, 6-8 June 2018
Evidence from ERA-LEARN analysis of P2Ps Three annual assessment exercises under ERA-LEARN 2020: 2015: statistical elaboration of existing data for FP7 ERA-NET and targeted interviews of JPI members: ERA-LEARN 2020 Policy Brief on impact assessment of networks 2015 , https://www.era-learn.eu/publications/other- publications/policy-brief-on-impact-assessment-of-networks-2013-2015 2016: two large-scale surveys (ERA-NET Cofund participants and overall ERA- NET participants): ERA-LEARN 2020 Updated policy brief on the impacts of networks 2016 , https://www.era-learn.eu/publications/other- publications/updated-policy-brief-on-the-impacts-of-networks-2016 2017: pilot survey of project participations of bio-economy related P2Ps + interviews at network and project level
A variety of motivations (2015 & 2016 exercises) Motivations that reflect a recognition that certain challenges are better dealt with at the international rather than the national level. Associated to this is the opportunity to access complementary research expertise to achieve critical mass in the areas addressed. More internal motivations reflecting own interests such as the opportunity to improve the international experience of the national research communities and to collaborate with other funding agencies. Rationale for participation shifts from being primarily based on serving national interests, competences and strengths (as in national programme design) to joining efforts trans-nationally for the common good
Multi-level governance EU/International policy context Trans-national policy context National policy context P2P specific objectives
The importance of alignment (2015 exercise ERA-NETs) Operational alignment: a) common evaluation procedures, b) common funding rules and c) joint project monitoring were key factors for achieving benefits in relation to supporting transnational research in an area requiring transnational cooperation. Strategic alignment: the benefits in relation of opening up national programmes to transnational research areas complementarity between the national programme and the ERA- NETs and the existence of cooperation agreements between the participating national programmes. relied upon the level of Importance of both operational and strategic alignment as pre- conditions forperceived benefits from participation in P2Ps (ERA-NETs in the specific case)
The importance of alignment (2015 exercise JPIs) Operational alignment a key factor of success National coordination both a condition and an impact Strategic alignment both a condition and an impact Alignment and national coordination basic aspects of the readiness level (apart from available resources)
A portfolio of joint activities (2015 exercise ERA-NETs) Benefits from opening up of national programmes to transnational cooperation are enhanced when organisations participated in schemes for joint training activities, supervising theses or common PhD schemes and specific cooperation agreements or arrangements between participating programmes. Achieving higher quality projects at national level is facilitated both when knowledge exchange is encouraged (through personnel exchange schemes) as well as when joint research activities are enabled (through common project funding rules). Importance of a portfolio of activities, i.e. both actual transnational research as well as capacity building and knowledge transfer, and other than research, activities
Excellence is not enough (2016 exercise ERA-NETs) Interdependence across certain types of impacts at project level: Increasing research skills is not a matter of international research collaboration alone, but also of adequate policy attention and support at national level Quality of research is a matter of skills and resources and of international collaboration; however, collaboration at the national level among different agencies also plays a role Solving societal challenges is not a matter of research alone, but also of influencing European / international agendas To achieve their ultimate goal, ERA-NETs need to focus on excellence, but at the same time, they need to pursue objectives beyond that, by increasing efforts to attract policy attention and support primarily at the national level so that they become influential actors at the European / international arena. (2016 exercise ERA-NET survey of network members)
A variety of project impacts (2017 pilot exercise) What are the expected impacts on your organisation from participating in the specific transnational project? Improved access to networks, consortia, etc. Higher European profile Improved competences and skills Increased interest in European partnerships Max score = 3.0 Additional research income Better evidence to make policy/strategy decisions Higher level of influence on third parties Increased interest in collaborating outside Europe Increased interest in seeking European commercial partnerships Improved environmental performance Better access to external investment Additional commercial income Reduced operating costs Increased European/global market share 0 0.5 1 1.5 2 2.5 3 (2017 pilot exercise ERA-NET project beneficiaries survey) 3 = High impact, 2 = Moderate impact, 1 = Low impact, 0 = Not applicable
A variety of project impacts (2017 pilot exercise) The greater involvement of end users generates greater impact on industry/sector practices Improved inter-disciplinarity across research domains so useful to tackle networks formation for Grand Challenges proposals Societal impact: long tail impact on wider society that was unexpected and now will be fully embedded in new calls (2017 pilot exercise - project beneficiaries and network member interviews)
A variety of impact types at network level (2015 exercise - JPIs) within and across nations; for JPI partners and JPI beneficiaries (Enduring) connectivity in science, strategic thinking, project management multidisciplinarity Capacity-building Attitude towards collaboration; level of ministries and agencies; Attitude/Culture Change visibility increased for certain areas influences in national and international level agendas Conceptual/policy changes to organisation for better national coordination development of strategies in new areas Structural ultimate impact: finding solutions to deal with the challenges addressed Instrumental
Confirmed by the 2016 exercise (ERA-NETs) Attitudinal/cultural type strengthening international collaboration among research communities across countries, Capacity building type both in relation to science and technology capacities and skills, and in international project management, as well as in terms of increased quality of research projects at national / regional level, and adopting interdisciplinary approaches in research (collaboration between multiple academic disciplines), and Conceptual type of benefits in terms of adopting transdisciplinary approaches in research (collaboration between academics and non- academics), and in terms of increasing awareness /visibility of specific research topics / issues at both national and cross-national level.
A suggested new framework for evaluating trans-national public partnerships in R&I EU-SPRI Conference Paris, 6-8 June 2018
A new evaluation framework Challenges New rationale Objectives Multi-level gov. Multi-stakeholder governance Inputs Readiness level: Alignment User engagement Activities National coordination A portfolio of activities; Multi-disciplinarity Outputs Network level Inter-dependence of impacts Inter/ate impacts Project level Variety of types Global impacts
P2P evaluation issues ( old and new ones) Utility Challenges Relevance Objectives Network health & connectivity Effectiveness Inputs Efficiency Readiness level Activities Added value examined at national, transnational &European/inter al level Outputs Outcomes Global impacts
References Amanatidou, E., Cunningham, P., G k, A. Garefi, I., (2014), Using evaluation research as a means for policy analysis in a new mission-oriented policy context . MINERVA. Volume 52, Issue 4 (2014), Page 419-438 DOI: 10.1007/s11024-014-9258-x Cagnin, C., Amanatidou, E., and Keenan, M. 2012. Orienting European Innovation Systems towards Grand Challenges and the Roles that FTA Can Play. Science and Public Policy 39: 140 152. ERA-LEARN 2020. 2015. Policy Brief on impact assessment of networks 2015. https://www.era- learn.eu/publications/other-publications/policy-brief-on-impact-assessment-of-networks-2013-2015 ERA-LEARN 2020. 2016 Updated Policy Brief on the Impacts of Networks 2016. https://www.era- learn.eu/publications/other-publications/updated-policy-brief-on-the-impacts-of-networks-2016 ERA-LEARN 2020 Guide for P2P impact assessment, https://www.era-learn.eu/publications/other- publications/guide-for-p2p-impact-assessment-1 Kallerud, E., Amanatidou, E., Upham, P., Nieminen, M., Klitkou, A., Olsen, D. S., ... & Scordato, L. (2013). Dimensions of research and innovation policies to address grand and global challenges. EU-SPRI Exploratory Initiative Paper. http://www.euspri-forum.eu/key_missions/exploratory_initiatives/ Lepori, B., Reale, E., & Lar do, P. (2014). Logics of integration and actors strategies in European joint programs. Research Policy, 43(2), 391-402. Mazzucato, M. (2018). Mission-Oriented Research & Innovation in the European Union - A problem-solving approach to fuel innovation-led growth. European Commission. ISBN 978-92-79-79832-0 doi:10.2777/360325 Meagher, L., Lyall, C., & Nutley, S., (2008), Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research , Research Evaluation 17(3): 163-173 Nedeva, M., 2013. Between the global and the national: Organising European science. Research Policy 42 (2013) 220 230. Weber, K. M., & Rohracher, H. (2012). Legitimizing research, technology and innovation policies for transformative change: Combining insights from innovation systems and multi-level perspective in a comprehensive failures framework. Research Policy, 41(6), 1037-1047. http://www.networkimpact.org/wp-content/uploads/2014/09/NetworkEvalGuidePt1_FramingPaper.pdf
https://www.era-learn.eu/ Thank you! Effie.Amanatidou@manchester.ac.uk Debbie.Cox@manchester.ac.uk Dimitri.Gagliardi@manchester.ac.uk Chiara.Marzocchi@manchester.ac.uk Mickael.Benaim@manchester.ac.uk
Meaghers framework and typology of impacts Typology of impacts Instrumental Conceptual Capacity-building Attitude/Culture Change Enduring Connectivity Source: Meagher, L., Lyall, C., & Nutley, S., (2008), Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research , Research Evaluation 17(3): 163-173 Meagher, R. (2013) . Research impact on practice. Case Study Analysis Report to the ESRC.
Meaghers framework and typology of impacts
Theoretical framework International research collaborations may be strengthened through institutional support via large-scale international research programmes (Wagner, 2002). A political process involving power and influence, more complex in the case of trans-national collaboration where several different national contexts need to co-exist. (Reale, et. al. 2013) Important prerequisite: coordination and alignment of national research systems in terms of organisation, structures and funding (Nedeva, 2013). Persisting challenges: different degrees of openness of national research systems (Lepori et al, 2014, Primeri et al 2014 and Cunz & Peuckert, 2015), the levels of financial support for research and innovation, the types of funding provided, the accountability and evaluation systems to inform research funding decisions. (Whitley and Glaser, 2007) Social science impacts on policy and practice. Conceptual, connectivity, cultural, and instrumental impacts important (Meagher, et al. 2008). Evaluation of transnational networks of collaboration usually follows Logic Model frameworks but this is not enough. Specificities as the network structure and governance in the success of P2Ps, issues such as alignment, or structural impacts are rather overlooked.
The ERA-LEARN 2020 Guide for P2P impact assessment Step 1: Defining the intervention logic of the P2P Step 2: Linking the challenges and objectives Step 3: What are the inputs and activities Step 4: What are the outputs (programme, projects)? Step 5: What are intermediate and global impacts? Step 6: Setting up a monitoring system of inputs, outputs and outcomes Step 7: Defining output, outcome and impact indicators Step 8: Data collection and analysis methods Step 9: From planning to actually conducting a P2P evaluation https://www.era-learn.eu/publications/other-publications/guide-for-p2p-impact- assessment-1
What is ERA-LEARN 2020? https://www.era-learn.eu/ ERA-LEARN 2020 is a support service for transnational public-to- public partnerships (P2P) in research and innovation - a H2020 funded project (1/2015 12/2017). Why? To support the P2P community, the Joint Programming Process, and the on-going optimisation of P2P networks via the learning and support platform including a database of P2P networks, organisations and countries toolbox of practices, training guides, case studies, annual monitoring reports, analysis reports and policy briefs, newsletters monitoring and impact assessment of P2P networks & projects benchmarking of current approaches and practices in relation to alignment and other important P2P issues Organisation of training workshops, and conferences,
Materials and Methods 2015 exercise NETwatch Survey (Doussineau et al, 2014). Integrated with information on 1) Networks, 2) Organisations, and 3) Calls launched by each Network (obtained through the EC). Analysis limited ONLY to ERANET under FP 7th (hence excluding previous experiences i.e. FP 6th and top-up funding ERANET+)
Materials and Methods 2016 exercise First survey of ERA-NET Cofund community 245 unique responses (50.3% response rate) Second survey sent to the rest of the ERA-NET community, i.e. ERA-NET Partners (covering FP6, FP7 and Horizon 2020) that did not participate in any Cofund Actions as well as to those Cofund Partners/Coordinators that had not replied to the first survey. The respective questionnaire was adjusted to refer to the overall ERA-NET experience rather than that of participating in a single Action. 342 responses (response rate of 21.8%).
Materials and Methods 2017 exercise (still ongoing) Common framework questionnaire Survey in April/May 2017 76 responses received From 18 countries More than 80% from academic and other research organisations 27% response rate overall One network achieved nearly 50% interviews of network members and project participants in 3 networks Government Body 6% Assocations 2% Research Organisation 48% Higher Education Academia 34% Private 9% Non-Profit Organisation 1%
Indications and way forward as an impact case Until now: Early adoption of the impact typology already at the Annual Joint Programming Conference 2016 Presentations of the Guide under specific invitations (JPI UE, Platform IA workshop) Invitations for membership in special groups (JPI Chairs Task Force, EUROSTARS IA expert panel,, ERA-NET Cofund Expert Group) Presentations of results at the EU-SPRI Conference 2017 Immediate future two peer-reviewed publications Continuation of ERA-LEARN 2020 for the next 4 years Opportunities to trace impact of our work through the planned dissemination and training activities in the next contract via personal testimonies (P2P community), level of interest in and evaluation of training activities, invitations to present our work and assist, citations of published papers, etc.)
How important are P2Ps? Evolution of national investments in P2P calls ( million)
How important are P2Ps? Cumulative Investment in Joint Calls, including EU Contribution, 2004-2017 ( m) Source: ERA-LEARN 2020 database, Sept. 2017
What are P2Ps? Experiments not fully institutionalised yet but with strong potential in creating critical mass and coordinating national efforts to tackle certain challenges that cross national borders. Despite differences in technicalities and level of reference (ministry level, agency level, etc.) P2Ps are: Networks where memberships are bound by a partnership agreement to join forces in pursuing commonly agreed targets. As such they are formed for a specific time, and might be slightly or significantly changed over time in terms of membership and thus capacities and committed resources.
The importance of alignment (2015 exercise ERA-NETs) Increased benefits for new researchers, i.e. without any prior experience in transitional research, accrue when organisations have prior ERA-NET experience, go a step further to setting up and implementing action plans for joint activities and monitoring of projects, and enjoy synergies between their national programmes and the ERA-NET research areas.
A variety of project impacts (2017 pilot exercise) How do you judge the level of achievement of the impacts on your organisation until now compared with your original expectations? Science/innovation related impacts Max score = 3.0 Behavioural impacts Policy related impacts Economic impacts Environmental impacts 0 0.5 1 1.5 2 2.5 3 (2017 pilot exercise ERA-NET project beneficiaries survey) 3 = More than expected, 2 = More or less as expected, 1 = Less than expected, 0 = Not applicable
