University Senate Meeting Summary - October 5, 2017
Senate meeting summary detailing key discussions on staff awards, upcoming meetings, establishment of certificate programs, and the web accessibility policy. The Joint President/Senate Inclusion & Respect Task Force aims to promote a respectful and inclusive campus community. Includes images and important resolutions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014
Agenda Quasi-evaluation studies Activity (if time allows)
Quasi-evaluation studies
Quasi-evaluation studies Address specific questions (often employing a wide range of methods) Advocate use a particular method Whether the questions or methods are appropriate for assessing merit and worth is a secondary consideration Both are narrow in scope and often deliver less than a full assessment of merit and worth
Approach 7: Objectives-based studies Advance organizers Statement of program objectives Purposes To determine to what extent a program achieved objectives Sources of questions Objectives as defined by staff, funder, or evaluator Questions To what extent were each of the stated objectives met?
Objectives-based evaluation results from a national research center Objective Relevance 60% Need 67% Quality of Design 62% Quality of Measurement 50% Quality of Analysis 75% Impact on Policy 90% Impact on Practice 70% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Methods Any relevant method for determining to which operationally defined objectives were met Pioneers Ralph Tyler Use considerations Must have clear, supportable objectives Strengths Ease of application Weaknesses Narrowness and inability to identify positive and negative side effects
Approach 7: The success case method Advance organizers Comparison between successful and unsuccessful instances Purposes To determine how well and in what respects a program is working Sources of questions Generally from program providers Questions What are the noteworthy successes? How were successes produced? What factors contributed to success/failure?
0 +1 +2 -1 -4 -3 -2 +3 +4
Methods 1. Focus and plan the study 2. Create an impact model 3. Survey all participants 4. Interview a sample of success and nonsuccess cases 5. Communicate findings, conclusions, and recommendations Pioneers Robert Brinkerhoff Use considerations Intended to assist service providers in increasing successes and decreasing nonsuccesses Strengths Ease of application Use for improvement Weaknesses Narrowness of scope
Approach 9: Outcome evaluation as value-added assessment Advance organizers System-wide measures of growth or gains Purposes Value added by a program and its constituent parts Sources of questions Oversight bodies Questions What parts of a program contribute most to growth or gains ?
Methods Gain score analysis, hierarchical linear modeling, etc. Pioneers Raudenbush, Sanders, Horn, Timms, etc. Use considerations Can be used to make and/or support policy decisions Strengths Longitudinal rather than cross-sectional Weaknesses Potential misuse by policy makers in placing blame
Approach 10: Experimental and quasi-experimental studies Advance organizers Cause-and-effect hypotheses, competing treatments, etc. Purposes To determine causal relationships between independent and dependent variables Sources of questions Researchers, developers, policy makers, etc. Questions To what extent is one treatment superior to another?
Recruitment Eligible Units Excluded Randomization Allocation Intervention Control Not Received Not Received Received Received Discontinued Discontinued Follow-Up Lost to Follow-Up Lost to Follow-Up Completed Completed
Methods Random or other method of assignment to conditions Pioneers Campbell, Cook, Shadish Use considerations Addresses only one particular type of question Strengths Strong causal conclusions (if assumptions are met) Weaknesses Requires substantial expertise, time, money, etc.
Approach 11: Cost studies Advance organizers Costs associated with program inputs, outputs, and outcomes Purposes The costs and outcomes of one more more alternatives Sources of questions Policy makers, planners, taxpayers, etc. Questions What are the costs of obtaining desired outcomes?
Methods Analysis of monetary and nonmonetary units Pioneers Levin, McEwin, Yates, etc. Use considerations Expertise required Strengths Bottom line conclusions of interest to most decision makers Weaknesses Often difficult to validly execute
Approach 12: Connoisseurship and criticism Advance organizers Specialized expertise, sensitivities, tacit knowledge, etc. Purposes To describe, appraise, and illuminate Sources of questions Determined by the connoisseurs or critics Questions What are a program s salient characteristics, strengths, and weaknesses?
Methods Perceptual sensitivity, prior experience, refined insights, etc. Pioneers Elliot Eisner Use considerations An audience willing to accept the approach Strengths Exploitation of refined expertise Weaknesses Objectivity and reliability
Approach 13: Theory-based evaluation Advance organizers A carefully specified theory of how a program is intended to operate Purposes To determine the extent to which a program is theoretically sound Sources of questions Determined by the guiding program theory Questions To what extent does the program theory work or not?
Program Process Theory Program Impact Theory Initial Intermediate Long-Term Initial Outcomes Outcomes Intermediate Outcomes Outcomes Long-Term Outcomes Outcomes Inputs Inputs Activties Activties Outputs Outputs
Resources Environment Action Model Implementing Implementing Organizations Organizations Associate Organizations and Intervention and Service Delivery Associate Organizations and Community Partners Community Partners Intervention and Service Delivery Protocols Protocols Ecological Ecological Context Context Implementors Implementors Target Populations Target Populations Change Model Program Program Implementation Implementation Intervention Intervention Determinants Determinants Outcomes Outcomes Figure 7.10
Methods Any method appropriate for testing the program theory Pioneers Chen, Donaldson, Weiss, Rogers, Rossi, etc. Use considerations Difficulty in applying the approach Strengths Useful for determining potential measurement variables Weaknesses Few programs are grounded by validated theories
Approach 14: Meta-analysis Advance organizers Sufficient studies of the same or similar programs Purposes To assemble and (statistically) integrate findings from multiple studies of the same or similar programs Sources of questions Policy makers, research repositories , etc. Questions What is the average effect of a particular type of program?
Sample Size Odds Ratio and 95% Confidence Interval 20% Equivalence Range Random-Effects Weight Study 0.80 1.20 17.66% Fombonne et al. (2006) 180 0.02 (0.01, 0.04) Mrozek-Budzyn et al. (2010) 11.82% 288 0.05 (0.01, 0.38) 15.76% Takahashi et al. (2003) 63 0.18 (0.05, 0.62) Aldridge-Sumner et al. (2006) 18.01% 114 0.51 (0.26, 1.02) 18.50% Fombonne et al. (2001) 194 0.61 (0.36, 1.02) Uchiyama et al. (2007) 18.24% 769 0.63 (0.34, 1.16) Random-Effects Model 1,608 0.20 (0.06, 0.63) Fixed-Effect Model 1,608 0.31 (0.23, 0.43) 1.00 Does Not Favor MMR Vaccine Favors MMR Vaccine Odds Ratio and 95% Confidence Interval
Methods Statistical methods for integrating study results (varies widely) Pioneers Glass Use considerations Major source of contemporary best practices across a variety of domains Strengths Evidence of effectiveness over units, treatments, observations, and settings Weaknesses Exclusive emphasis on program outcomes
Activity We will split the class into two sections (1 and 2) In each section, appoint one student to chair your appointed group Each member of section 1 should select one of the approaches discussed today and discuss why it is useful Members of section 2 should listen and take notes Members of section 2 should then outline weaknesses of the selected approaches Finally, the chair of each group should discuss the potential strengths, weaknesses, and utility of the selected approaches
Encyclopedia Entries Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence Evaluability Assessment Evaluation Use Fournier, Deborah Positivism Relativism Responsive evaluation Stake, Robert Thick Description Utilization of Evaluation Weiss, Carol Wholey, Joseph
