Understanding School Counseling Evaluation in Nigeria

Evaluation in Support of School Counseling in Nigerian Schools APROCOM 2018 John Carey Chair of the International Society for Policy Research and Evaluation in Scholl-Based Counseling and Emeritus Professor of Counseling University of Massachusetts, Amherst

Workshop Participants Today, Will: Know what evaluation is Understand the purpose of formative and summative evaluation Understand a framework for evaluation of school counseling See how the framework is applied to a particular context for formative evaluation Develop an interview protocol for your context

Goal of Evaluation To serve social betterment by providing balanced, defensible information collected by the counselor that can be acted on to improve services

Evaluation Evaluation is a systematic and intentional process of gathering and analyzing data (quantitative and qualitative), to inform learning, decision-making, and action. Gopa and Preskill (2014)

Evaluation Research Both Evaluation and Research use quantitative and/or qualitative research methods.

Differences

Involvement of stakeholders Evaluation is shaped directly or indirectly based on stakeholder perspectives. Research is not required to involve stakeholders.

Purpose Research works to understand the world. Evaluation seeks to determine the value, merit, or worth of a program.

Methods While evaluation uses social science research methods, Evaluation and research differ in how the methods are assembled, used based on purpose

Values Research works to avoid values Evaluation makes values explicit in order to make judgements against a set of standards.

Three scenarios where evaluation would be helpful A school counselor provides counseling and consultation services to students, teachers and parents. A school counselor working is delivering a prevention program in classrooms or groups. A school counseling program is providing preventative programs, teacher training and counseling and consultation services to students, teachers and parents.

Two Key Purposes of Evaluation for School Counseling To improve practice. To show to others (stakeholders, school decision-makers, funders) that counseling activities are effective.

Two Types of Evaluation

Formative Evaluation Program implementation What is working well What needs improvement and in what way

Summative Evaluation Beneficiaries Anticipated Benefits Unintended Negative/Positive Consequences

Framework for Conducting Evaluation Theory of action (and Logic Model) Evaluation questions Evaluation design, method, and data collection Data analysis and findings Evaluation reporting and use

A Helpful First Step: Creating a Logic Model to Describe The Theory of How Counseling Works 1) The Inputs Resources required 2) The Activities What is actually done 3) The Short-Term Changes Immediate client changes 4) The Long Term Benefits Long term benefits to clients (and stakeholders)

Counseling and consultation services to students, teachers and parents. Formative Evaluation Inputs Activities Short Term Changes Long Term Benefits Achievement 100 Hours of Counselor Time Counseling with Students Engagement in School Wellbeing Consultation with Parents Skill in Parenting

Counselor Implementing Prevention Program Inputs Activities Short Term Changes Long Term Benefits Students Attitudes Counselor Time Social Climate 12 Group Sessions Student Skills Anti- Bullying Curriculum Reduced Bullying Student Knowledge

School counseling program is providing preventative programs, teacher training and counseling and consultation services to students, teachers and parents. Inputs Activities Short Term Changes Long Term Benefits Parent Consultation Sessions 25 Hours of Supervision Parenting Skill Achievemen t 500 Hours of Contact Time Student Counseling Sessions Student Engagement in School Increased Student Wellbeing Teacher Consultation Sessions Management of Problem Students 20 Hours of Training 60 Hours of Instructional Time Student Empathy/So cial Skills Improved Social Climate 12 session in 5 classrooms

Counseling and consultation services to students, teachers and parents. Formative Evaluation Theory of Action Inputs Activities Short Term Changes Long Term Benefits Achievement 100 Hours of Counselor Time Counseling with Students Engagement in School Wellbeing Consultation with Parents Skill in Parenting

Counseling and consultation services to students, teachers and parents. Formative Evaluation Evaluation Questions

Counseling and consultation services to students, teachers and parents. Formative Evaluation Is counselor time being administered as planned? Is consultation being conducted as planned with each stakeholder group? What is going well? What needs improvement?

Counseling and consultation services to students, teachers and parents. Formative Evaluation Evaluation design and method

Counseling and consultation services to students, teachers and parents. Formative Evaluation Obtain hourly records Develop questionnaire questionnaire asks individuals whether or not they are satisfied with the consultation and ways to improve service Each individual the counselor provides consultation is asked to complete the questionnaire

Formative Evaluation (continued) Develop interview questions questions ask stakeholders such as school principal and teachers about their perception of the services

Design and Method Decisions Comprehensive Input from key stakeholders Makes some logical sense addresses key elements of listening center Straightforward to develop and implement Efficient

Counseling and consultation services to students, teachers and parents. Formative Evaluation Data analysis and findings

Counseling and consultation services to students, teachers and parents. Formative Evaluation Count hours Count number of hours with each group Count questionnaire responses by category within each item Aggregate responses by stakeholder group Record interview responses

Formative Evaluation - Continued Describe themes in the questionnaire and interview responses Consider outliers or unique responses Organize based on evaluation questions

Counseling and consultation services to students, teachers and parents. Formative Evaluation Evaluation reporting and use

Counseling and consultation services to students, teachers and parents. Formative Evaluation Develop report Specify purpose Describe the design, method, and data collection procedure Describe data analysis procedure Organize by evaluation question

Counseling and consultation services to students, teachers and parents. Formative Evaluation Provide copy of report to school principal perhaps have report available for others Present to school community Act on recommendations

This is a SAMPLE Formative Evaluation Different decisions could be made about the design and data to be collected

Data Sources Existing data relevant to the services Data about services collected by the teacher/counselor Observations Interviews Focus groups Questionnaires

Interviews

What are attributes of quality interview questions?

EXERCISE Based on your own needs, develop an interview protocol, that includes interview questions, how they would be administered, and to whom. Include how the responses would be recorded.

Dealing With Resource Constraints Strategy 1: Ask for support/assistance from administration Strategy 2: Ask for volunteer help from parents Strategy 3: Divide into component parts Focus on one component at a time Consider representative sampling when possible

Other Essential Aspects of Evaluation Effective communication Interpersonal skills Ability to work in groups Savvy in negotiating organizational politics

Non-technical skills are essential Counselors can use their counseling skills in a strategic and effective manner to address the non- technical aspects of evaluation Counselors are well-positioned to do evaluation work

Questions/Comments

International Society for Policy Research and Evaluation in School-Based Counseling https://qshimunoz.wixsite.com /ispresc