Istar Standard Setting Methodology and Workflow Overview

ISTAR Standard Setting

Overview of ISTAR Standard Setting Methodology Item Descriptor (ID) Matching was suggested by the Indiana TAC for finding cut scores on the ISTAR tests. ID matching process: Evaluative process using an Ordered Item Booklet (OIB) and Performance Level Descriptors (PLDs) over several rounds Based on the Performance Level Descriptors, panelists make judgment about each of the items in the OIB, classifying it as Developing, Meeting, or Exceeding proficiency (Rounds 1 and 2) Panelists then indicate a threshold region where the item descriptors change from mostly lower PLD to mostly higher PLD (Rounds 1 and 2) Within the threshold region, panelists are asked to find a transition point in the OIB where knowledge, skills, and cognitive processes required by items change from the lower PLD and begin to match more closely the knowledge, skills, and cognitive processes described in the higher level PLD (Rounds 3 and 4)

ISTAR Standard Setting Workflow Meeting and Logistics: General session overview of ISTAR, introduction to the process, meeting logistics, housekeeping, security protocols Experience the assessment panelists utilize the testing environment mirroring the student experience and interaction with items PLD discussion panelists review and discuss the PLDs for the upper grade what knowledge, skills, and cognitive processes expected of students classified in each performance level OIB discussion panelists review and discuss the OIB for the upper grade what knowledge, skills, and cognitive processes are required to answer it correctly, and why is each item more difficult than the previous ones in the OIB ID Matching practice round panelists are assigned sample (10) items from the OIB to practice the tasks of matching items to PLDs

ISTAR Standard Setting Workflow (cont.) Meeting and Logistics (cont.) Rounds of judgments and feedback 4 rounds of judgments (assign every item in the OIB to a PLD, identify threshold regions, and locate a cut score within each threshold region) After each round, facilitator provides panelists with data that they can use to evaluate and inform their judgments in the subsequent rounds. Feedback data varied by round but generally included summary statistics (i.e., descriptive statistics, such as the mean, median, minimum, and maximum page numbers) for the threshold region boundaries and recommended cut scores, agreement data, and impact data. Repeat process with the lower grade-level assessment Table Leaders continue to the Vertical Articulation meeting Panelists examine the reasonableness of the proposed cut score recommendations across the grade levels

ISTAR Standard Setting Logistics Standard Setting Meetings: June 19 21: ELA and Science June 26 28: Mathematics and Social Studies Panels consisted of multiple breakouts: Four grade band panels (3 4, 5 6, 7 8 and 10) ELA and Mathematics Two grade band panels (4 and 6, and 10) Science One grade band panel (5 and 7) Social Studies Each grade band panel had a facilitator and a content specialist Each panel consisted of 8 9 Indiana Educators in two tables; most panels had one general education teacher Two of the panelists were designated by IDOE as table leaders they led table discussions, took notes, and spoke on behalf of the table during the large-group discussion

ISTAR Performance Levels Two cuts were recommended for each grade and subject: The Meeting Proficiency cut The cut score that differentiates Developing Proficiency performance from Meeting Proficiency performance. The Exceeding Proficiency cut The cut score that differentiates Meeting Proficiency performance from Exceeding Proficiency performance.

ISTAR Standard Setting Results Tables and graphs in the coming slides present the impacts of cut scores Three impacts are given for all grades (except Grade 7 Mathematics) Impacts of cuts recommended in Round 4 by grade band panels Impacts of cuts adjusted in vertical articulation committees, if adjustments were made Impact of IDOE recommendation for Grade 7 Mathematics Impacts of cuts on the whole population which includes the No Mode of Communication students reported as Undetermined proficiency level

ISTAR Standard Setting Results ELA, Round 4 Impact Data for ISTAR ELA after Round 4 100 90 26 34 38 41 80 42 44 53 70 60 34 50 48 35 44 40 39 36 34 30 39 20 23 19 19 18 10 18 14 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results ELA, Vertical Articulation Impact Data for ISTAR ELA Vertical Articulation 100 90 26 34 38 41 80 42 44 46 70 60 40 50 48 35 44 40 39 36 40 30 20 34 23 19 19 18 18 10 14 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results ELA, all students Impact Data for ISTAR ELA, NMC Students Included 100 90 24 32 36 39 39 42 80 44 70 60 36 50 45 32 36 42 34 40 39 30 31 20 22 18 18 18 18 13 10 9 7 7 6 5 4 4 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ELA: Context for Results The complexity of passages may not be as high as expected. Development of passage expectations to ensure distribution of complexity is ongoing. Using supporting graphics in some items may make them less difficult. Specifically, the Vertical Articulation panel had the following thoughts about grade 3 impact data: PLDs were seen as being more difficult in this grade than in other grades Grade 3 is the first year the students take the assessment, so they might be struggling with the concept of test-taking Revisit the assessment of writing in terms of the overall blueprint and relationship to the general education assessment

ISTAR Standard Setting Results Mathematics, Round 4 Impact Data for ISTAR Math after Round 4 100 3 4 5 6 9 12 13 11 90 25 80 31 25 20 38 32 70 60 50 85 40 72 67 66 63 30 57 56 20 10 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Mathematics, all students Impact Data for ISTAR Math, NMC Students Included 100 3 4 5 6 9 11 12 10 90 23 23 29 19 80 36 29 70 60 50 82 68 40 62 61 60 51 55 30 20 10 9 7 7 6 5 4 4 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Mathematics Grade 7 IDOE Recommendation Impact Data for ISTAR Math after Round 4 100 3 4 5 6 9 12 13 90 25 25 80 31 25 20 38 32 70 60 50 40 72 71 67 66 63 30 57 56 20 10 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Mathematics Grade 7 IDOE Recommendation Impact data for ISTAR Math Grade 7, NMC Students Included; Grade 7 adjusted (p.6) 100 3 4 5 6 9 11 12 90 23 23 23 29 80 19 36 29 70 60 50 68 40 69 62 61 60 51 55 30 20 10 9 7 7 6 5 4 4 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Mathematics: Context for Results Continued focus on rigor given the transition to the new standards Middle school grades have been identified as being naturally harder than other grades in mathematics by the vertical articulation panel for the following reasons: The PLDs, even at the developing level require more complex functioning of students than the PLDs from other grades ( understand and predict vs. identify ). The content is different, calling for more abstract mathematical thinking (square roots and rational/irrational numbers). Students are asked to use prior knowledge to apply their skills. They may or may not have this prior knowledge. Grade 3 has been identified by the Vertical Articulation panel as possibly being easier for the students than other grades for, among others, the following reasons: There are fewer components in Content Connectors. Content is more concrete and applicable to everyday life and therefore more familiar to students.

ISTAR Standard Setting Results Science, Round 4 Impact Data for ISTAR Science after Round 4 100 17 90 31 80 50 70 38 60 50 41 40 31 30 45 20 28 19 10 0 Grade 4 Grade 6 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Science, Vertical Articulation Impact Data for ISTAR Science Vertical Articulation 100 17 90 31 41 80 70 42 60 41 50 40 40 30 41 20 28 19 10 0 Grade 4 Grade 6 Grade 10 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Science, all students Impact Data for ISTAR Science, NMC Students Included 100 16 90 30 39 80 70 39 60 38 50 38 40 30 38 20 27 19 10 7 5 4 0 Grade 4 Grade 6 Grade 10 No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Social Studies, Round 4 Impact Data for ISTAR Social Studies after Round 4 100 20 21 90 80 70 34 60 48 50 40 30 44 20 32 10 0 Grade 5 Grade 7 Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results Social Studies, all students Impact Data for ISTAR Social Studies, NMC Students Included 100 18 20 90 80 70 33 45 60 50 40 30 43 30 20 10 7 4 0 Grade 5 Grade 7 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency