Mastering Standard Setting in Assessment Development
Delve into the intricacies of setting standards in assessment development through Module 21 of the Michigan Assessment Consortium's Development Series. Learn about different ways to set standards, including terminology, preliminary steps, performance levels, and descriptors. Gain insights into the crucial aspects of creating a well-developed test and defining proficiency levels to ensure accurate classification of students. Narrated by Jim Gullen from Oakland Schools, this module offers a comprehensive guide to standard setting in the realm of assessments.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Michigan Assessment Michigan Assessment Consortium Consortium Common Assessment Common Assessment Development Series Development Series Module 21 Module 21 Standard Setting
Narrated By: Narrated By: Jim Gullen Oakland Schools 2
Standard Setting Standard Setting In this module you will learn two ways to set standards Your test must be well developed before you set a standard If a purpose of your test is to classify students into categories, you need to set a standard 3
Some Terminology Some Terminology Standards Proficient Not Proficient The standard separates these two groups The standard is defined by a cut score a score on the test that separates the group into proficient not proficient 4
Preliminary Steps Preliminary Steps Before setting standards, several things need to be decided: Number of levels of performance to divide scores into The labels to be applied to each performance level e.g., advanced, proficient A few sentences to describe what each level means what is proficient student? 5
Number of Performance Levels Number of Performance Levels The number of performance levels and the labels you give them will depend on the purpose of your common assessment Interim benchmark assessments used at the end of instructional units 2 to 3 levels End-of-course assessments may want to use more levels (4- 5) More levels require more technically sound assessments, though! 6
Performance Level Descriptors Performance Level Descriptors Performance level descriptors (PLDs) are the written descriptions of what it means to score at each performance level The entire common assessment development team should help to write them, so they are internalized 7
Performance Level Descriptors Performance Level Descriptors Initially, different teachers will have different definitions of what proficient means, for example Through discussion and examples, the team will come to a common understanding for assessment and for instruction 8
Performance Level Descriptors Performance Level Descriptors The common assessment team should try to describe each performance level in a few sentences It should focus on the underlying skills that students need to have to score at that level, not just be a re- hash of the test items students pass at that level 9
Determining the Cut Scores Determining the Cut Scores The next step is for the common assessment team to determine the cut scores that divide the score scale into the various performance levels If you decided to use four performance levels, your team would need to determine three cut scores to place students into four performance categories 10
Setting Standards Setting Standards There are a number of ways of setting standards some item centered and others that are student centered In either case, best professional judgment is used; there is no right or wrong answer statistically It is important that you set standards that are meaningful to you 11
Setting Standards Setting Standards If there is no right way to do this, how should you proceed? The easiest way is to pick a number out of thin air say 75% This isn t a good way to set standards why 75% instead of 85% or 95% 12
Setting Standards Setting Standards There are two standard setting techniques that will be described Modified Angoff, an item-centered method Body of Work, a student-centered method Each will be described in simple terms that you can use; each has much more complex versions that you could use if you have someone skilled to help you 13
Modified Modified Angoff Angoff This procedure involves the identification of the qualities of a minimally-proficient student, as defined by the performance level descriptors What sorts of items should this student be able to answer correctly? Which types of items will give this student trouble? 14
Modified Modified Angoff Angoff Now, look at the test itself: Which items would this minimally-proficient student pass? Which items would this student not pass? Add up the total number of items this student would pass. This becomes the participant s estimate of the cut score Each member of the team performs this task 15
Modified Modified Angoff Angoff After each member has their cut score, the group should discuss Do you agree on the designations for each test item? Do you agree on the total number to be passed? Have the members repeat the process of setting their cut score Take the median value of all the participants cut score estimates and use that as the cut score 16
Modified Modified Angoff Angoff Finally, introduce impact data a summary of how many students would be placed in the proficient category based on the students actually tested (could be field test or operational test data) Do the rating one last time Select the median value for use for each performance level 17
Body of Work Body of Work The other standard setting technique is called Body of Work First, assemble all of the work of the students the multiple-choice, constructed-responses and any performance assessments for each student separately Use all of the students responses 18
Body of Work Body of Work The task of the standard setter is to examine all of the work of the student, and then to place the student in what you consider to be the appropriate performance level (as defined by the PLDs) There are several steps in this, again to be carried out by the entire team 19
Body of Work Body of Work First, determine the overall score scale and how the different types of items are summarized Second, select samples of students work all along the score scale every few points Example: 5, 10, 15, 20, 25, 30, 35 20
Body of Work Body of Work Students actual work for the constructed-response and performance assessments should be shown Each member of the team should place each student into one the performance levels defined by the PLDs The team members should compare notes where do they agree? Disagree? 21
Body of Work Body of Work Identify areas along the score scale where there is the most disagreement At these areas, introduce additional samples of student work and repeat the rating process Place each student in the higher or the lower performance level Discuss ratings as a group 22
Body of Work Body of Work The team may need to do this process once or twice, depending on whether agreement is reached This process would need to be repeated for each performance level that the team decided to use The final step is to show impact data to panelists 23
Standard Setting Procedures Standard Setting Procedures There are advantages and disadvantages of each of these standard setting methods See the handout for the list of these There are several other methods that could be used, all involving professional judgment 24
Standard Setting Procedures Standard Setting Procedures If you wish to use more sophisticated techniques, find someone knowledgeable to help you Remember, there is no correct standard Our standards are based on our judgments The techniques described will help assure thoughtful standards are set 25
Lets give it a try Let s give it a try Here s our performance level descriptor: Students who are proficient in arithmetic are able to calculate routine arithmetic operations without the use of a calculator. 26
Series Developers Kathy Dewsbury-White, Ingham ISD Bruce Fay, Wayne RESA Jim Gullen, Oakland Schools Julie McDaniel, Oakland Schools Edward Roeber, MSU Ellen Vorenkamp, Wayne RESA Kim Young, Ionia County ISD/MDE 27
Development Support for the Assessment Series The MAC Common Assessment Development Series is funded in part by the Michigan Association of Intermediate School Administrators In cooperation with Michigan Department of Education Ingham and Ionia ISDs, Oakland Schools, and Wayne RESA Michigan State University 28
