Constructing Items & Tasks
The intricacies of choosing the right assessment strategies, including selected response and performance assessments. Learn about designing and developing selected response assessments, creating effective distractors, and writing rules for assessment items. Enhance your understanding of how to ensure validity, authenticity, and cognitive complexity in educational assessments.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
Presentation Transcript
Choosing the right assessment strategy The 1999 Standards for educational and psychological testing recommends the use of multi-modal assessments. However, there are general principles that must guide our choice of methods in the classroom and in large-scale assessment
Choosing the right assessment strategy Selected Response like MCQs allow computer scoring and therefore more efficient for large numbers of items and examinees. Thus these items usually ensure higher content validity. However, authenticity and cognitive complexity might be lacking and the overall assessments may be low inference. Constructed Response often facilitate cognitive complexity and assessment of critical writing skills. However, efficient scoring systems are required.
Choosing the right assessment strategy Performance assessments usually but not always ensure authenticity and cognitive complexity. However, overall content validity [Generalizability] may be low and scaffolding may be required if equity is to be maintained. Without adequate rubrics, scoring issues will arise. Training of assessors is critical.
Designing & Developing Selected Response Assessments 1) Focus on a single topic & important objective 2) Select relevant stimulus material for context-dependent items 3) Formulate a clearly stated problem based on the material- Use question or incomplete statement 4) Identify the answer (keyed response) 5) Identify plausible alternatives (distractors) 6) Use checklist to remove cues
How to create distractors A term used in the unit but referring to a different concept An everyday term that is close but misleading in meaning A common misconception or confusion that students have A plausible word or phrase that is meaningless but derives from something in the stem
Writing Rules 1) Every item should reflect specific content and a specific mental behaviour as derived from a table of specifications Base each item on important content to learn; avoid trivial content Use novel material to test higher order learning. Paraphrase textbook language when used in a test item Keep the content of each item independent from content of other items on the test Avoid over specific and over general content when writing items Avoid opinion-based items. Avoid trick items Keep vocabulary simple for the group of students tested 2) 3) 4) 5) 6) 7) 8)
Writing Rules 1) 2) 3) 4) Avoid complex MC formats (& the MTF) Format the Item Vertically Edit and proof the items Use correct grammar, punctuation, capitalization, and spelling Minimize the amount of reading in each item Ensure that the directions in the stem are very clear Include the central item in the stem instead of the choices Avoid window dressing (excessive verbiage) Word the stem positively, avoid the use of negatives such as NOT or EXCEPT. If negative words are used, use cautiously and always capitalize or put in boldface. 5) 6) 7) 8) 9)
Writing Rules 1) 2) 3) Write as many effective choices (distractors) as possible. Make sure that only of these choices is the right answer. Vary the location of the right answer according to the number of choices. Place the choices in logical or numerical order. Keep choices independent; choices should not be overlapping. Keep choices homogenous in content and grammatical structure. Keep the length of the choices about equal. None of the above should be used carefully. Avoid All of the above . 10) Phrase choices positively; avoid negatives such as NOT. 4) 5) 6) 7) 8) 9)
More Writing Rules 1) Avoid giving clues to the right answer a) Specific determiners (Always, never, completely & absolutely). b) Clang associations (choices identical to or resembling words in the stem) c) Grammatical inconsistencies d) Conspicuous correct choice e) Pairs of triplets of options that provide a clue to the correct choice f) Blatantly absurd, ridiculous options Make all distractors plausible. Use typical errors of students to write distractors Use humor only if it is compatible with the stakes involved and the learning environment 2) 3) 4)
Designing & Developing Constructed Response Assessments 1) Create a stimulus 2) Attach prompts that elicit the cognitive behaviour you desire 3) Match the keyword to the expected student response 4) Include high order keywords to elicit critical thinking
Designing & Developing Performance Assessments 1) Decide on the skills and competencies you want to assess (or develop) 2) Identify or Choose assessment tasks that are authentic, real-life, practical, and/or holistic 3) Develop instructions for the tasks 4) Decide on process or product focus 5) Decide on the amount of structuring necessary and include scaffolding if required 6) Develop a useful rubric
Help on choosing tasks Does the task truly match the outcome(s) you're trying to measure? Does the task require the students to use critical thinking skills? Is the task a worthwhile use of instructional time? Does the assessment use engaging tasks from the "real world?" Can the task be used to measure several outcomes at once? Are the tasks fair and free from bias? Will the task be credible? Is the task feasible? Is the task clearly defined? --Adapted from Herman, Aschbacher and Winters (1992
What a performance assessment looks like Competencies/skills to assess Authentic innovative performance tasks that elicit the behaviour & skills you require An analysis of what the student will do Instructions for students including scaffolding if required Criteria for scoring & a rubric
Scaffolding explained Task 1 (Without Scaffolding) Write an essay telling how life twenty years from now may be like the present and how it might be different. You may want to conclude with an evaluation: Will the future be better or worse than the present? Task 1 (With Scaffolding) When you studied history, you studied the past and compared it to the present. Now, consider how life twenty years from now may be like the present and how it might be different. Some areas you might write about in your comparison are: family life, transportation, education, food, housing, and government. You may want to conclude with an evaluation: Will the future be better or worse than the present? Write your composition using the following guidelines: State the topic of your essay in the first sentence. Remember to write about how you believe the future may be like the present and how it might be different. Give specific examples and details. Fully explain how and why in the future each area would be like or different from the present. Checkpoints to remember: Take time to plan your essay on scratch paper. Organize you ideas carefully. Remember what you know about writing complete paragraphs. Check that you have used whole sentences, correct punctuation and correct spelling.
Adding Authenticity Table 1 Vision for Authentic Achievement, Pedagogy, and Authentic Student Performance Authentic Assessment Tasks Construction of Knowledge Information Consideration of Alternatives Disciplined Inquiry Process Authentic Student Performance Analysis Authentic Achievement Authentic Instruction Organization of Higher Order Thinking Content Deep Knowledge Substantive Conversation Disciplinary Concepts Elaborated Written Communication Elaborated Written Communication Problem Value Beyond School Connections to the World Beyond the Classroom From Fred M. Newmann, Walter G. Secada, and Gary G. Wehlage, A Guide to Authentic Instruction and Assessment: Vision, Standards and Scoring (Madison, WI: Wisconsin Center for Education Research, 1995): p. 64. Reprint with permission Audience
Authenticity on a continuum Figure 2 Figure 1 European Research Project2 You are a travel agent from one country in Europe 1. Identify four major attractions in your country. Each attraction must be located in a different part of the country 2. For each attraction, research the following questions: Location: Identify on a map the absolute and relative location of the attraction. Place: What are the cultural characteristics of the site? What are the physical characteristics that surround the site? Environment: How has the attraction affected the environment? (Consider population, cities, natural areas, etc.) Movement: To what degree has the attraction affected the movement of goods or ideas through history? Regions: How is the site valuable to the region? does the site have political and/or cultural value? Explain. 3. Create a brochure or poster that includes the information in #2. It should be designed to create interest in visiting the country. As a travel agent from your country, present the brochure/poster to the class, and convince them that yours would be an interesting country to visit. Scandinavia Directions: Read pp. 275-283 in the World Geography textbook and answer the following questions Place: (Physical Features) 1. What is a peninsula? What countries in Scandinavia form peninsulas? Why is this region known as the "land of the midnight sun"? What are fjords? How are they formed? What mountain range runs through Norway and Sweden? What important natural resource exists in the Scandinavian Shield Why is the North European Plane important? Where do most Scandinavians live? How was Iceland formed? 2. 3. 4. 5. 6. 7. 8.
Brainstorming authentic tasks Create a budget. Create a spreadsheet. Write a computer program. Convert one type of graph or chart into another (e.g., pie chart into bar chart). Draw the same graph to different scales. Critique a chart, table or graph and explain how it might be made clearer or more useful. Conduct a poll on consumer preferences, display results graphically, state conclusions. Compare the accuracy over time of two different weather (or other) forecasts. Conduct an opinion poll, create a chart or table of the results and explain your conclusions.
Brainstorming authentic tasks Analyze how the author of an editorial uses persuasion. Keep a journal. Analyze how the author of a letter to the editor uses persuasion. Analyze how a speaker uses persuasion. Analyze a story. Write a poem. Solve an open-ended math problem, preferably one with multiple solutions or multiple paths to the correct solution. Determine which store has the best prices. Use mathematical manipulatives to illustrate a concept. Prove a theorem or corollary that has not been done in class or the text. Draw a floor plan. Measure something. Build a model. Use a bus or train timetable to determine a schedule. Estimate amount of food needed to feed a large group.
