Performance Assessment for Leadership Development and Readiness

Slide Note

Explore considerations in determining leadership candidate skill development and readiness, including assessment methods, evaluation criteria, and foundational theories such as authentic learning and professional practices.

bjak Follow

Uploaded on Sep 28, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Performance Assessment for Performance Assessment for Leaders: Leaders: the Massachusetts PAL the Massachusetts PAL Assessments and Field Trial Results Assessments and Field Trial Results New York State Principal Project Advisory Team Meeting New York State Education Department Albany, New York January 25, 2017

Credits Authors Margaret Terry Orr, Bank Street College, (morr@bankstreet.edu) Ray Pecheone, Stanford University Liz Hollingworth, University of Iowa Rabia Karatoprak, University of Iowa Barbara Beaudin, Bank Street College Jon Snyder, Stanford University Joe Murphy, Vanderbilt University Funding and support Work made possible by the Massachusetts Department of Elementary and Secondary Education (ESE) Funded through federal Race to the Top grants Bank Street College, lead agency for this work ShowEvidence (now TumbleFeed, on-line assessment management

What are the considerations to determine leadership candidate skill development and readiness? What is ascertained? Leadership skills and indicators Skill level of initial leaders How candidates demonstrate skills? Authentic work Scoreable work products How skills are evaluated? Rubrics and scoring systems Trained, reliable skilled scorers

Conceptual foundation Performance assessment as a means of authentic learning (Kolb, Mezirow) Essential leadership practices Inquiry on practice (Leithwood, Anderson) Professional learning culture (DuFour, Seashore, Wenger) Supervision and feedback (Danielson, Supovitz) Family and community engagement (Bryk, Epstein, Henderson) Reflective practice in leadership development (Schon; Kottkamp and Osterman)

Theory of performance assessments for professional readiness Attributes: Authentic Constructivist, educative Scalable to reflect views of professional readiness Use in teacher preparation and medical preparation Efficacy depends upon the validity and reliability of the assessment system: Components Candidate supports scoring

Massachusetts policy context for assessment development Policy reform in educational leadership made possible by RTTT Preparation program redesign Multiple pathways to preparation and licensure Preparation programs (university and non-university) Administrative internship/apprenticeship Panel review Earlier commitment to performance assessment 800-1200 candidates licensed annually Contracted with Bank Street College and their expert team to develop performance assessments for leaders

Ensuring valid and reliable Assessments: Development Process Design Implementation and standards setting Development and readiness Fall and spring pilot Field trial

Components of the Performance Assessment for School Leaders Four assessment tasks Represent the primary educational improvement levers of school leaders Based on MA Leadership Standards and Indicators 3-4 artifacts and commentary produced per task 3-4 rubrics with 6-8 indicators for scoring each task

PAL Assessment Task Components Four parts to the work for each task Three types of products Artifacts Other documents Commentary Four-point rating scales Beginning Developing Meeting Exceeding Investigate Assess Prepare Act

Task 1: Leadership through a vision of high student achievement Steps Work products Artifact #1 Priority area and its context Artifact #2 Plan for action strategies Artifact #3 Findings, feedback and recommendations Commentary leadership skills developed and learning experienced Collect and analyze 3-5 years of quantitative student performance indicators, qualitative indicators of school culture and student learning. Identify a priority student learning area of interest with attention to federally-designated priority student groups Collect additional qualitative and quantitative information for the priority student learning area, Document existing school programs, services, and practices and ascertain the gaps and opportunities for improvement. Develop action strategies to improve student learning in the priority student learning area. Seek leadership feedback and support

Task 2: Leadership for a Professional Learning Culture Steps Work products Document how teacher teams and groups are used in the school. Form a professional learning group of teachers Select a priority academic area and focus for improvement Engage group in meetings and support teachers collectively and individually on improving practice Collect evidence on new or improved practices and analyze Solicit team feedback on their learning and improvement Artifact #1 identify priority area, professional group and professional learning plan Artifact #2 describe the learning process and work accomplished by the group, emphasizing candidate s role Artifact #3 present group member feedback on the process, learning and benefits Commentary leadership skills developed and learning experienced

Task 3: Leadership in observing, assessing and supporting individual teacher effectiveness Steps Conduct a pre-observation conference. Document a teacher observation using a district guide on effective teaching practices Describe teacher performance using the rubric indicator performance levels. Conduct a post-observation conference that facilitates teacher rapport and learning Provide constructive feedback and strategies for improvement in a post-observation conference Collect and analyze teacher feedback on the effectiveness of the observation, feedback and support Work products Artifact #1 Preobservation template Artifact #2 Teacher observation video recording Artifact #3 Post-observation meeting video recording Artifact #4 Analysis of observed teaching Artifact#5 Teacher feedback on observation and post-conference Commentary leadership skills developed and learning experienced

Task 4: Leadership for family engagement and community involvement Steps Identify a priority area for improving family and community engagement that would directly or indirectly enhance student learning in a priority area Engage staff, leaders and family and community leaders as a planning group to enhance family and community engagement Create a multi-strategy plan on how to improve family and community engagement. Implement one planned strategy Gather and analyze feedback and other evidence on the plan and strategy s effectiveness for improving family and community engagement Work products Artifact #1 Analysis of data, priority area and plan Artifact #2 Implementation of one strategy Artifact #3 Feedback on plan and strategy implementation Commentary leadership skills developed and learning experienced

Field trial questions Is each task a valid and reliable measure of a key aspect of leadership practices? Do the four tasks measure different aspects of leadership? Are the four tasks measuring complementary parts of a comprehensive leadership construct? Is scoring reliable? Are the tasks free of bias in scoring?

The field trial process One year (2014-15) 100% of eligible candidates seeking initial school leader licensure Complete all four tasks; No cut scores, only completion requirements Yielded: 422 candidates who performed all four tasks 82% from preparation programs 65% female; 74% indicated white 30 scorers 25% of submissions were double scored

Are the tasks valid? Number of indicators 6 6 8 6 Number of domains Minimu m Maxim um Std. Mean Type of Score Deviation Task 1 Task 2 Task 3 Task 4 3 3 4 3 1.00 1.17 1.00 1.00 4.00 4.00 4.00 4.00 2.76 2.97 2.86 2.66 .553 .529 .472 .685 Total average score 26 13 1.35 3.58 2.83 .379

Are the measures independent? Do they have coherence as one overall measure? Does each task have measurement integrity? When factor analyzed, the indicators loaded strongly and discretely as four independent factors the tasks. Are the tasks independent measures? (.27-.35) Do the task measures have coherence as an overall leadership score? (.64-.74) total score Average Task 1 Task 2 Task 3 Task 4 Measure Average Total Score Task 1 1.000 -- -- -- -- .696 1.000 -- -- -- Task 2 .710 .350 1.000 -- -- Task 3 .642 .320 .315 1.000 -- Task 4 .736 .284 .340 .267 1.000

Are the measures reliable? Are they bias free? Estimated reliability coefficients for each task shows strong reliability for Tasks 1, 2 and 4 and weaker reliability for Task 3 (due primarily to low variance among candidates). Number of scorers Task 1 2 1 0.728 0.842 Initial evaluation of results show no different by program type but some differences by gender. There was insufficient information to compare results by race/ethnicity. 2 0.656 0.792 3 0.208 0.345 4 0.581 0.735

Considerations and related effects Assessment development Strengthen instructions, scorer training and supervision Testing out use of cut-scores and fee payment on submission quality Early work shows greater quality, narrowing the performance range Fewer candidates are seeking licensure Irregularity problem Candidate and program benefits Learning and readiness benefits reported by most candidates Program alignment and improvement reported by many programs Gradual reshaping school and district expectations for leadership preparation Other states are interested in replication

Implementation Year Outcomes Implementation changes Added a fee ($500 for initial registration) Added cut-scores (on 4-point scale) 2.1 threshold for each task 2.5 total average cut-score for 2016-17 2.75 total average cut score for 2017-18 Implementation outcomes Fewer candidates completed all four tasks during the Program Year (n=153) Tasks continue to be rated by candidates and programs as relevant, feasible, educative and beneficial. Total average scores were slightly higher (12% scored below 2.5 and very few below 2.1 threshold) Task 4 remained most challenging Program candidates performed somewhat better than nonprogram candidates, especially for Task 1 and 4.

Conclusions and implications The four tasks are valid and reliable assessments of initial leadership readiness PAL has had positive intended outcomes: Educative (yielded expected learning benefits for candidates) Evaluative (identified candidates who are ready for initial leadership) Beneficial for school settings (lead to improvements based on authentic work completed) Informative (lead to positive changes in preparation programs content, organization and internships)