Cultures of Evaluation and Policy Analysis: Uncovering Common Ground and Divergent Paths
This paper explores the separate yet interconnected cultures of evaluation and policy analysis, revealing how practitioners in these fields share common objectives while drawing upon distinct literatures and methodologies. The analysis uncovers discrepancies in the sources cited and publication focuses between the Journal of Policy Analysis and Management (JPAM) and the American Journal of Evaluation (AJE), shedding light on the different outlooks and approaches taken by evaluation and policy practitioners.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
The Separate but Related Cultures of Evaluation & Policy Analysis AES 2011 Conference David Turner
Overview This paper started with the observation that evaluation and policy practitioners read different journals, cite different authors, used different terminology but cover common ground I acknowledge that policy and evaluation have wide and diverse literatures; more than one person will usually follow closely
Common Ground Evaluation and policy analysis have a common focus on how policies are developed and implemented, and a general goal of public interest Evaluation practitioners have to consider policy needs and stakeholders for evaluation work
Different Outlooks Evaluation practitioners may treat policy goals and designs as fixed Evaluators may not follow debates within policy community on policy & implementation
Separate Literatures Analysis of journal publications shows differences in sources Reviewed 2010 publications in Journal of Policy Analysis and Management (JPAM) and American Journal of Evaluation Looked at content of the articles and the sources cited
JPAM Articles 50 research articles in 4 issues, on range of policy areas such as education, tax policies, health, employment, and housing Focus often on evaluative questions concerning policy implementation and impacts Public administration, economic journals cited (rarely evaluation)
AJE Articles 18 research articles in 4 issues, most often focused on evaluation methods or approaches (less focus on policy substance) A few references to public administration or management journals, or journals in policy areas such as criminal justice or education
Book Buying Patterns Looked for reading patterns among policy and evaluation books by using Amazon data Identified lists of evaluation and policy texts Looked for links under customers who bought this book also bought Mapped linkages between books (pointing from one to another bought by same people)
Evaluation Books Patton, Utilization-Focused Evaluation Morris, Evaluation Ethics for Best Practice Chelimsky and Shadish, Evaluation for the 21stCentury Rossi, Lipsey & Freeman, Evaluation Scriven, Evaluation Thesaurus Posavac, Program Evaluation Stufflebeam, Evaluation Theory, Models, & Applications Weiss, Evaluation
More Evaluation Books Shadish, Foundations of Program Evaluation Mathison, Encyclopedia of Evaluation Wholey, Handbook of Practical Program Evaluation Owen, Program Evaluation Davidson, Evaluation Methodology Basics Yarbrough, Program Evaluation Standards Fitzpatrick, Program Evaluation: Alternative Approaches Bamberger, Real World Evaluation
Policy Books Bardach, A Practical Guide for Policy Analysis Weimer & Vining, Policy Analysis Birkland, An Introduction to the Policy Process Stone, Policy Paradox: The Art of Political Decision Making Kraft & Furlong, Public Policy Sabatier, Theories of the Policy Process Kingdon, Agendas, Alternatives, and Public Policies
More Policy Books Stokey & Zeckhauser, A Primer for Policy Analysis Pressman & Wildavsky, Implementation Hill & Hupe, Implementing Public Policy Smith, Writing Public Policy Dye, Understanding Public Policy Wilson, Bureaucracy
Book Buying Networks (all links) P11 E12 E13 P1 0 P12 E14 E11 P9 E15 P13 E10 E16 E9 P8 P1 E1 E8 P7 P2 E2 E7 P6 P3 E3 E6 P5 E4 P4 E5
Book Buying Networks (reciprocal) P11 E13 E12 P1 0 P12 E14 E11 P9 E15 P13 E10 E16 E9 P8 P1 E1 E8 P7 P2 E2 E7 P6 P3 E3 E6 P5 E4 P4 E5
Learning from Policy Evaluation practice could draw on developments in policy and public management How the policy process works What makes for successful policy implementation
Policy Process Models Descriptions of how the policy process works have advanced from the stages heuristic over time Programs emerge over time, decisions are part of a continuous process Different models of policy provide a range of insights
Policy Model: Example 1 Richard Matland described policy implementation as a function of ambiguity and conflict A policy can be ambiguous in either means or ends A policy is subject to conflict when parties with a stake in an issue have different views over its means or ends
Ambiguity-Conflict Model Conflict Low High Political implementation Issue: power Example: busing Administrative implementation Issue: resources Example: smallpox eradication Low Ambiguity Experimental implementation Issue: Context Symbolic implementation Issue: coalition strength Example: community action agencies High wide range of policy areas such as education, tax policies, health, employment, and housing Example: Headstart for children
Implications for Evaluation The model describes a policy in ways that suggest what evaluation can achieve Administrative: focus on technical questions Political: consider whose values and needs are being met by evaluation Experimental: scope for contribution to new ideas and approaches Symbolic: people may need to be seen doing something, may not expect impacts!
Another Policy Model Inside Government Agencies In the Real World Revised hypothesis and next action Note anomalies Ask questions Hypothesis Action All available technical, practitioner, and user information about what is known and unknown informs policy hypothesis Collaborative learning in and between networks of government and non-government actors Source: Institute of Policy Studies Working Paper 11/04, Experimentation and Learning in Policy Implementation , Eppel, Turner, Wolf
Implications for Evaluation Program theory: need to adapt to emerging issues, anomalies; may not have a stable theory of action Outcome evaluation: question of when (if ever) program goals stabilise for measurable results
Grounding Formative Evaluation Implementation has been widely studied since Pressman & Wildavsky, but may not be considered in formative evaluation Wide range of endogenous and exogenous factors, significance of key actors Substantial and varied literature
Learning from Evaluation It s not just evaluators who should look outside their own boundaries; policy analysts can learn from evaluation: Alternative approaches for gaining insight (not just gold standard ) Need to define goals, indicators, logic more precisely where possible
Evidence of Change Recent attention to issues of complexity reflects awareness of need to adapt evaluation approaches Developmental evaluation one example of a response to complex policy environments and emerging policy solutions Attention also given to need for knowledge exchange across boundaries
Discussion What do you think? Is there a cultural divide with policy analysis, and if so how significant is it?
