Understanding and Analyzing Performance Patterns in Educational Evaluation

Evaluating individuals Evaluating classes Noting unusual patterns or trends Identifying gaps in learning USE OF SCORES

HOW TO RECORD AND INTERPRET SCORES To show overall classroom trends with respect to the learning progression To show the status of each individual student with respect to the learning progression To identify unusual patterns in a particular item or a particular student To identify remaining gaps in individual or class performance

GRADE LEVEL LEARNING PROGRESSION K is highest LP level in the specific LP of interest

IN A PERFECT WORLD You might expect student performance by item to look like this (all dichotomous or right/wrong items) Students ordered by performance Items ordered by difficulty Student Student Easiest Easiest item item Hardest Hardest item item Lowest performing 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1 Highest performing

FOR RUBRIC SCORED ITEMS Student Student Easiest Easiest item item Hardest Hardest item item Lowest performing 2 2 3 4 5 1 2 2 4 5 1 2 2 3 4 1 1 2 2 4 0 1 1 2 3 Highest performing

ARRANGE CLASS DATA ORDERED BY ITEM LEVEL AND STUDENT PERFORMANCE

CODE STUDENT DATA BY LP LEVEL ACHIEVED Right/wrong items code to the matching LP level (for LP level 2, right is 2, wrong is 0 see item 4) Rubric items should be scored by building block matching each rubric level Results might look like this

INTERPRETING CLASS TRENDS Pretest Posttest Class has gone from at least some mastery of Level 1 for most students (some students are doing better) to mastery of Levels 1 and 2, and partial mastery of Level 3

INTERPRETING INDIVIDUAL STUDENT PERFORMANCE Student 1 is at the lowest level (lower anchor) with some partial mastery of LP level 1 Student 2 is at LP level 1; partial mastery of LP 2 Student 3 is near mastery of LP 3 Student 4 is at LP 3; perhaps some mastery of upper anchor

GAPS IN INDIVIDUAL PERFORMANCE Student 1 has several gaps in mastery of LP level 1 (items 2, and 7-9; note item 4 is dichotomous at LP 2) Student 2 has a gap in mastery of LP 2 (item 2, and items 6-9) Etc.

UNUSUAL ITEM PATTERNS Although the class is mostly at LP levels 2 and 3, notice that item 3 is unusually difficult May be due to a problem with the item May be due to a gap in class understanding of a concept

UNUSUAL STUDENT PATTERNS Note that Student 1 has an unusual pattern of responses. Although this student is doing somewhat poorly overall (and is the lowest scoring student), they also did very well on items 5, 6, and 9, achieving the maximum score on these items. It might be worth checking in with this student to learn more about their performance.

BIBLIOGRAPHY Wilson, M. (2005). Constructing measures: An item response modeling approach. New York: Psychology Press, Taylor & Francis Group. Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13 (2), pp. 181- 208. Wright, B., & Masters, G. (1982). Rating Scale Analysis. Chicago: MESA Press.

CREATIVE COMMONS LICENSE Use of Scores PPT by the Oregon Department of Education and Berkeley Evaluation and Assessment Research Center is licensed under a CC BY 4.0. You You are free to: are free to: Share Share copy and redistribute the material in any medium or format Adapt Adapt remix, transform, and build upon the material Under Under the following terms: the following terms: Attribution Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. NonCommercial NonCommercial You may not use the material for commercial purposes. ShareAlike ShareAlike If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. Oregon Department of Education welcomes editing of these resources and would greatly appreciate being able to learn from the changes made. To share an edited version of this resource, please contact Cristen McLean, cristen.mclean@state.or.us.