Overview of CalAPA Standard Setting Study and Passing Score Recommendations
This document provides updates on the California Administrator Performance Assessment (CalAPA) development and standard setting process for Commission examinations. It includes information on the implementation timeline and recommendations for revising the passing score standard for the CalAPA. The discussion covers the foundational aspects, leadership cycles, and steps involved in supporting Preliminary Administrative Services Credentialing. Key events from the pilot phase to full consequential implementation are outlined.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Agenda Item : 2D Revisiting CalAPAStandard Setting Study Findings and Passing Score Recommendations Amy Reising, Chief Deputy Director Gay Roby, Consultant, Professional Services Division Heather Klesch, Vice President, Educator Solutions for Licensing and Learning, Evaluation Systems group of Pearson
Executive Summary This agenda item provides (a) an update on the development and implementation of the California Administrator Performance Assessment (CalAPA) to date; (b) foundational information about the standard setting process for Commission examinations and assessments with recommendations for revising the passing score standard for the CalAPA; (c) information and timeline describing how the Commission and its technical contractor Evaluation Systems group of Pearson (ES) will continue supporting Preliminary Administrative Services Credentialing(PASC) in the fifth year of operational administration, 2022-23. 2
Section 1: CalAPA Development and First Years of Implementation Pilot (2016-2017) and Field-Testing Phases (2017-2018) CalAPA First Non-Consequential Operational/Non-Consequential Year (2018-2019) Data Leading to a 2019 Standard Setting Study Description of Three Leadership Cycles of the CalAPA Use of Analytic Rubrics Assessor Training and Calibration Commission Adoption of a Passing Score in 2019 3
Timeline of CalAPAImplementation DATES EVENT 2012 Commission approves new Administrative Services program standards ; also approves of the concept of administrator PA 2014-2016 2016-2017 Development of the Assessment Pilot 2017-2018 Field-test 2018-2019 First Operational Year (Non-consequential) 2018-2019 2019-2020 2020-2021 2021-2022 Full Consequential Implementation: Any candidate enrolled after June 1, 2018 must pass the CalAPA as part of their program requirements 4
CalAPA Leadership Cycles and Recursive Steps Leadership Cycle 1: Analyzing Data to Inform School Improvement and Promote Equity Investigate Leadership Cycle 2: Facilitating Communities of Practice Reflect Plan Act Leadership Cycle 3: Supporting Teacher Growth 5
Essential Questions Rubric 1.1 Based on the chosen California state indicator, how does the candidate select and analyze quantitative data sources across the three most recent years, identify patterns and/or trends related to equity, choose one student group, and relate their analysis to the school s vision, mission, and/or goals? Rubric 1.6 Are the strategies proposed for equitable school improvement for the student group well informed by the findings of the equity gap analysis, including contributing factors, and responsive to the problem statement? Rubric 1.4 How does the candidate determine contributing factors, including institutional and/or structural factors, that created or added to the identified equity gap affecting a student group and cite the research supporting their determination? Rubric 1.5 How does the candidate use the equity gap analysis and identification of potential contributing factors to develop a feasible problem statement related to student achievement and/or well- being? Rubric 1.2 How does the candidate collect and analyze relevant qualitative data and explain their relation to quantitative data findings and the student group equity issue? Rubric 1.3 How does the candidate conduct an equity gap analysis based on the chosen California state indicator to inform their understanding of the equity issues for a student group 6
Analytic Rubric Assessment Essential Question: How does the candidate ... Level 1 Level 2 Level 3 Level 4 Level 5 Candidate does not provide an explanation of the purpose or processes for coaching, observation, and/or instructional feedback practices at the school. Candidate lists the teacher coaching, observation, and/or instructional feedback practices at the school, providing limited detail about the processes used. The purpose for conducting these processes is unclear. Candidate describes and analyzes the current role of teacher coaching, observation, and/or instructional feedback, including the purpose and intended goals of the practices for teachers and students. All of Level 3, plus: Candidate describes how teachers participate in the selection and purpose of the instructional feedback practices at the school to meet the unique needs of the teachers and build trust. All of Levels 3 & 4, plus: Candidate discusses relevant evidence- based practices and/or adult learning theory to explain how school practices of teacher coaching, observation, and/or instructional feedback support ongoing teacher development and benefit students. Candidate outlines a plan for the coaching cycle with little or no details about implications for their approach or provides no plan. Candidate describes plans for and implications of each step of the teacher coaching cycle and observation based on the volunteer teacher s professional past experience. Candidate provides minimal description of how the volunteer teacher s professional experience influenced their choices about how to conduct coaching and/or observation. 7
Table 32: CalAPA Candidate Submissions Fall 2019 through April 2022 CalAPA Cycle N* Leadership Cycle 1 4211 Leadership Cycle 2 2848 Leadership Cycle 3 2873 Table 32 is located in Appendix #, page 32 8
CalAPAAssessors Must meet eligibility criteria for assessors Attend a two-day training, including the accurate assessment of a candidate submission which verifies their calibration within the system parameters Are employed on a year-to-year basis by Evaluation Systems Attend annual refresher training sessions on program updates Attend monthly validation training sessions to maintain calibration 9
Component 2 Standard Setting Process Recommendations regarding revised passing score standards for CalAPA 10
Standard Setting Process Common term used in the large-scale assessment industry to describe the process of determining a minimum passing score for new or revised assessments. The term standard as it is used in standard setting refers to a performance standard, or minimum level of acceptable performance on an assessment. 11
Contextual Statement and Guiding Question Think about an administrator candidate who is just at the level of knowledge and skills required to perform effectively the job of a new administrator in California public schools. What score (the sum of all the rubric scores in the cycle) represents the level of performance that would be achieved by this individual? 12
Table 1: Cycle Rubric Scores for CalAPA Cycle N* Mean SD Median Min Max Leadership Cycle 1 4211 22.8 3.92 23.0 8 39 Leadership Cycle 2 2848 19.4 3.54 19.0 8 35 Leadership Cycle 3 2873 19.7 3.13 20.0 7 31 Table 1 is located on page 7 of the agenda item 13
Table 2: Initial Passing Score Recommendations N Panelists Recommending Side Condition of 1 rubric score of 1 5 of 15 5 of 15 5 of 15 CalAPA Cycle (Total Points Possible) Total Score Panel Recommendation Leadership Cycle 1 (40) Leadership Cycle 2 (35) Leadership Cycle 3 (35) 16 14 14 Table 2 is located on page 7 in the agenda item 14
Appendix D: CalAPAImpact Data by various subgroups Ethnicity Tables Leadership Cycle 1 Table 15, page 21 Leadership Cycle 2 Table 21, page 25 Leadership Cycle 3 Table 27, page 28 Gender Leadership Cycle 16 Table 16, page 22 Leadership Cycle 22 Table 22, page 26 Leadership Cycle 28 Table 28, page 29 Language Leadership Cycle 1 Table 17, page 23 Leadership Cycle 2 Table 23, page 26 Leadership Cycle 3 Table 29, page 30 School Setting Leadership Cycle 1 Table 18, page 23 Leadership Cycle 2 Table 24, page 27 Leadership Cycle 3 Table 30, page 30 15
Table 6: Final Passing Score Recommendations (Panel Median) N Panelists Recommending Side Condition of 1 rubric score of 1 CalAPA Cycle (Total Points Possible) Total Score Panel Recommendation Leadership Cycle 1 (40) 15 2 of 15 Leadership Cycle 2 (35) 14 3 of 15 Leadership Cycle 3 of (35) 14 4 of 15 Table 6 is located on page 9 in the agenda item 16
Component 3 Table 10: Candidate/Program Support Type of Support Date(s) Candidate Weekly Office Hours PASC Weekly Office Hours CASC Weekly Office Hours (supporting COVID-19 candidates completing CalAPA during induction) Deep Dives (an in-depth walk-through of the cycles, guides, and templates) New Coordinator Orientation Meredith Fellows Implementation Conference (highlighting the sharing of best practices) Every Wednesday 5:00-5:30 pm 1st & 3rd Thursdays 11:00 11:30 am 1st & 3rd Thursdays 8:00 8:30 am September 2022 September 30, 2022 September 16-17, 2022 Fall/Winter 2022 Spring/Summer 2023 Fall 2022 Winter 2023 Spring 2023 Fall and Winter2022 Semi-Annual CalAPA Coordinator Meetings Digging Deeper Series for all CTC Performance Assessments (investigating a global component in each assessment) CalAPA Faculty Workshops Table 10 cis located on page 12 of the agenda item 17
CTC Staff Recommendations Staff recommends that the Commission o Discuss and determine passing scores for CalAPA Leadership Cycle 1, Leadership Cycle 2, and Leadership Cycle 3, with a consideration of side-conditions o Determine an enactment date for the new scores to go into effect o Direct staff to continue to collect and study candidate data and return in two years to consider future revisions to the passing standards and potential side conditions . 18
Next Steps Should the Commission adopt the recommended revised passing scores of XX for each leadership cycle, this information will be posted on the CalAPA websites (www.ctcexams.nesinc.com) and communicated to the field. In addition, the revised passing standard adopted by the Commission will be applied to all CalAPA candidates at the adopted date 19
