Implications of Universal Design, Accessibility, and Fairness in Test Development
This content discusses the implications of universal design, accessibility, and fairness in test and item development. It covers topics such as common test accommodations, types of assistive technology, innovative item types, accessibility in item interface designs, QTI coding for screen reader software, and procedures for developing accessible tests. It also provides insights into the exam sponsorship process, including accommodations commonly approved for physicians, such as additional testing time and custom testing schedules.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Universal design, accessibility & fairness Implications for test and item development Cynthia G. Parshall, Belinda Brunner, & Dwan Bovell #atpconf
Overview of Session Common test accommodations & types of assistive technology implications for innovative item types; Accessibility and item interface designs QTI coding for screen reader software Item and test development procedures for accessibility Especially for innovative item types. #atpconf 12/2/2014 2
Common Test Accommodations & Types of Assistive Devices #atpconf 9/28/2024 3
Exam Sponsor Overview ABIM tests approximately 35,000 physicians every year 3 separate test administrations Spring Maintenance of Certification (MOC) only Summer Initial Certification only Fall Subspecialty Certification and MOC 18 Subspecialties (Card, Rheum, etc.) 36 exam types due to separate Certification and MOC exams #atpconf 12/2/2014 4
Exam Sponsor Overview Approximately 1 percent of physicians receive any type of accommodation per year 307 in 2015 Typical process for accommodations Physician applies for accommodation by application deadline Application is reviewed by staff and consultants Application is denied or approved Exam containing accommodation is created #atpconf 12/2/2014 5
Exam Sponsor Overview Commonly approved accommodations Additional testing time; Extra break time; Large- font exam materials; Separate testing room, Trackball mouse; Custom testing schedule; Anti-glare computer screen; Left-handed mouse set-up; Permission to have and use diabetic supplies; Permission to have and use light provisions, including water in a spill-proof bottle and/or a small snack Portable air supply; Prescribed medication #atpconf 12/2/2014 6
Types of accommodated exams Double time, large print Double time Large print only Extra break time Additional break time 1 module per day - double time 1 module per day - extra break time 2 modules per day - extra break time 2 modules per day - no extra time - large print 1 module per day - no extra time - large print 2 modules per day - extra break time - large print 1 module per day - no extra time 2 modules per day - no extra time 1 module per day - double time - large print 1 module per day - extra break time - large print #atpconf 9/28/2024 7
Types of Accommodations, and related need Provision Type of disability or need Provision Type of disability or need Extra time Some cognitive challenges Some physical problems (e.g., MS, back problems, pregnancy) Screen magnifier software Low vision Extra breaks Some physical problems (e.g., MS, back problems, pregnancy, nursing) Screen reader software Blindness Low vision Some cognitive challenges Separate testing room Some mobility challenges Use of a human reader/recorder Some cognitive challenges (e.g., dyslexia) Color-blindness Some types of low vision Some cognitive challenges Tired eyes Unfamiliar with computer based testing Color/ contrast Large-font test form Low vision Some cognitive needs Tired eyes Paper exam Audio exam Blindness/low vision Some cognitive challenges #atpconf
Accommodations, Based on Innovation Type of innovation Type of disability or need Examples of solutions Audio Hearing impairments Test description Video Hearing impairments Captioning Video Visual impairments Possibly, provide a written script, including description of the scene and crucial elements Use of screen reader (did you mean software or human?) Caliper Mobility challenges Hotkeys #atpconf
Accessibility and Item Interface Design #atpconf 9/28/2024 10
Goals of this part of the presentation To describe a universal design perspective for the exam user interface Build in a maximally useful design, and/or build in user-control So user interface accommodations are rarely needed To give an 80:20 set of suggestions To address about 80% of the accessibility needs, with 20% of the effort Focusing on just the user interface #atpconf
Universal Design #atpconf 9/28/2024 12
Ways of looking at this 1.By type of accommodation And which needs might be met through this 2.By types of disability/needs And what accessibility choices or accommodations might be needed 3.By type of item or test element And whether it could impact certain needs; and if so, what accessibility choices could help #atpconf
Types of Disability and Accessibility Type of Disability/Need Accessibility Solutions Visual e.g., blind users, users with low-vision, users with obstructed vision, and users with digital eye strain Maintain good size and contrast Use real text vs. image of text Include alt text for images, etc. Provide captioning for video dialogue Provide text description for non-dialogue sound files Avoid auto-play of audio or video Don t make clickable elements too small or too close together Don t rely solely on complex movements such as drag-and-drop Provide hotkey alternatives Auditory e.g., deaf users, hearing impaired users, users with auditory processing issues Motor - users of assistive devices like: specialized keyboards, eye trackers, button switches Cognitive e.g., users with dyslexia, autism, ADD, Down s syndrome, etc. Provide information in a straightforward, way Maintain a clean interface #atpconf See http://www.creativebloq.com/netmag/simple-introduction-web-accessibility-7116888
But first Conflicting access needs When one user s need is in conflict with another s Testing example Conflicting access is: - Certain color schemes that work well for users with low vision are problematic for users with color-blindness Possible solution: - Good default color scheme plus user-control over alternatives Special challenge in testing Some accommodations change the construct When a construct change is needed "adaptations [that] change the intended construct to make it accessible . . . while retaining as much of the original construct as possible." (AERA, APA, & NCME, 2014, p 59). #atpconf 9/28/2024 15
Recommended actions for all exams Design a user interface that is clear and simple Minimal use of panels, windows, reference to other pages, etc. Clear grouping of related elements Minimal scrolling Helpful to: users of screen readers & screen magnifiers; users with certain learning disabilities #atpconf
Images Images Include alt-text - For all images, icons, etc. QTI code: - alt= chest x-ray - alt= Exhibit button - alt= Helpful to: users of screen readers #atpconf
Graphs and Illustrations Graphs & Illustrations Include a second signifier for any color-coding - e.g., red cross-hatching, green pointillism Place keys and legends above or to the right of the image - This makes them easier to find in braille Helpful to: users with color blindness; users of refreshable braille displays #atpconf
Media Audio Text descriptions Video Captioning for video with dialogue Helpful to: users with hearing disabilities Helpful to: users with hearing disabilities __________________ Brief scene setting , to preface recorded dialogue Helpful to: users with visual disabilities #atpconf
Color and Contrast Color / contrast Provide user-select color accommodation - To change foreground and background colors When color is used as a signifier, always include a second signifier - e.g., bold for text; cross-hatching for image Verify that the contrast is sufficient - e.g., through websites such as: http://webaim.org/resources/contrastchecker/ Helpful to: users with color blindness or low vision; users with tired eyes #atpconf
Fonts Font Use an appropriate font type and size - e.g., Tahoma, 14-pt Avoid problematic font displays - e.g., italics, underlining, etc. Consider formatting of text - With sufficient white space and leading (i.e., amount of vertical space between lines of text) - Left-justification - Line length of 70-80 characters maximum If possible, provide user-control over sizing - e.g., user-adjustable font size Helpful to: users with low vision; users with tired eyes #atpconf
Mouse Use Mouse use Make buttons and other selectable areas: - Large enough to avoid the need for precise mouse movements - Close to the associated text to avoid the need for excessive mousing Provide keyboard alternatives (hotkeys), whenever possible - "Most assistive technologies for people with motor disabilities either work through the keyboard or emulate the functionality of the keyboard." - http://webaim.org/articles/motor/assistive Helpful to: users with motor disabilities; users with repetitive strain injuries #atpconf
Tables Tables Include headers - Consider including advance organizers or other context information on the item screen Consider the use of html code - to specify the table content, and - to identify the row and column headers - See http://webaim.org/techniques/tables/data Helpful to: users of screen readers; users of screen magnifiers #atpconf
Sample Table and QTI code States, Capitals, and Populations <table> <caption>States, Capitals, and Populations</caption> US State State Capital Population of Capital <tr> <th scope="col">US State</th> <th scope="col">State Capital</th> <th scope="col">Population of Capital</th> </tr> Alabama Montgomery 200,983 Alaska Juneau 31,118 <tr> <th scope="row">Alabama</th> <td>Montgomery</td> <td>200,983</td> </tr> QTI elements: <caption> <td> <th> <scope> <tr> <th scope="row">Alaska</th> <td>Juneau</td> <td>31,118</td> </tr> </table> #atpconf 9/28/2024 24
Executable Applications Executables, to be accessed within test driver Ensure good use of color and contrast Ensure a large enough display (or user-control over display size) Provide keyboard alternatives for all mouse actions Helpful to: users with color blindness; users with low vision; users with motor challenges #atpconf
Item and Test Development for Accessibility #atpconf 9/28/2024 26
Goals of this part of the presentation To discuss item and test development according to universal design principles To place this development in the context of construct validity To take a closer at innovative item development, in particular #atpconf
Universally designed assessments are designed and developed from the beginning to allow participation of the widest possible range of students, and to result in valid inferences about performance for all students who participate in the assessment ~Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002, p.6). #atpconf 9/28/2024 28
Elements of universally designed assessments (NCEO) Inclusive assessment population Precisely defined constructs Accessible, non-biased items Amenable to accommodations Simple , clear, & intuitive instructions & procedures Maximum readability & comprehensibility Maximum legibility See http://www.cehd.umn.edu/nceo/ #atpconf
Developing universally designed assessments (NCEO) Test conceptualization Test construction Field testing Item analysis Item reviews #atpconf
Test conceptualization What is to be assessed? Precisely defined test constructs How will it be assessed? Test design which considers individual needs Key question: After an item is written, is the item reflective of the desired construct? #atpconf 9/28/2024 31
Construct validity & accessibility Goal: Preserve construct-related validity while increasing access Threat: Construct-irrelevant features interfere with test taker performance on the intended test construct due to test taker characteristics #atpconf 9/28/2024 32
Construct-Centered Approach for Universally Designed Assessments 1. Develop as complete an understanding as possible of the test construct and assessment intent What do you want to measure? Targeted/focal KSAs What else affects successful performance? Ancillary/non-focal KSAs #atpconf 9/28/2024 33
Construct-Centered Approach for Universally Designed Assessments Target KSAs Ancillary KSAs What else affects successful performance? What do you want to measure? Examples Quantitative reasoning Math ability Math ability (in a word problem) Verbal ability Ability to prioritize treatment options (through a drag-&-drop item) Ability to use the mouse #atpconf 9/28/2024 34
Construct-Centered Approach for Universally Designed Assessments 2.Determine how test taker performance can be supported on the ancillary KSAs Examples: Formula sheets Vocabulary lists Alternative ways to move objects #atpconf 9/28/2024 35
Item development for universally designed assessments Introduce item authors/reviewers to universal design concept Provide style guidelines that consider the elements of universal design Include bias/sensitivity review as part of the item review process #atpconf 9/28/2024 36
Style guidelines example Amenable to accommodations Which hospital has the lowest average number of Cesarean sections per 100 births over the three years? Cesarean section rates by hospital 2011-2013 20 Number of Cesarean sections per 100 births 18 16 14 12 10 8 6 4 2 0 Hospital A Hospital B Hospital C Hospital D 2011 2012 2013 #atpconf
Style guidelines example Amenable to accommodations Which hospital has the lowest average number of Cesarean sections per 100 births over the three years? Cesarean section rates by hospital 2011-2013 Number of Cesarean sections per 100 births 2011 2012 2013 20 18 16 14 12 10 8 6 4 2 0 Hospital A Hospital B Hospital C Hospital D #atpconf
Style guidelines example Use of language Elizabeth Salinger is the head of the department. Better: Liz is the head of the department. Most ungulates have common characteristics which include having hooves and primarily eating plant material. Better: Most ungulates have common characteristics. These characteristics include having hooves and being plant eaters. The object of the game was to get as few points as possible. Better: The purpose of the game was to get as few points as possible. The quintessential characteristic of an epic hero is courage. Better: The main characteristic of an epic hero is courage. #atpconf 9/28/2024 39
Universal design & innovative items 9/28/2024 40
What is an innovative item? An item that goes beyond the discrete, text-based, multiple-choice format (Parshall, et al., 2000) Computer-delivered items that include specialized mechanisms for interacting with and responding to the item #atpconf 9/28/2024 41
Why use innovative items? To more fully represent the test construct To assess greater depths and a broader spectrum of knowledge, skills & abilities than can be done with traditional item types #atpconf 9/28/2024 42
Universal design of innovative items (Dolan, 2010) Any item is a collection of components. Innovative items allow for a greater variety of components: Text Tables Graphs Images Audio Video Animations #atpconf 12/2/2014 43
Universal design of innovative items (Dolan, 2010) Test takers interact with each component through a variety of processing mechanisms: Perceptual Linguistic Cognitive Motoric Executive Affective #atpconf 12/2/2014 44
Universal design of innovative items (Dolan, 2010) Variance results from the interplay of item components and test taker processing. Variances related to the target KSAs contribute to valid score interpretation. Variances not related to the target KSAs create construct-irrelevant variance (CIV). #atpconf 12/2/2014 45
Universal design of innovative items (Dolan, 2010) Consider design options for innovative items which minimize CIV measurement. Examples: Item component Processing Ancillary KSA Design option to address CIV Text Linguistic Vocabulary Vocabulary lists Text (calculations) Cognitive Background knowledge Formulae sheets Graphs Perceptual Color perception Avoid common color-blindness combinations Video Executive Working memory Note-taking tools #atpconf 12/2/2014 46
Item design Universal design for computer-based testing (UD-CBT) guidelines (Dolan, et al., 2010) Revise as necessary and re- evaluate Evaluate item design for construct validity Evaluate item design for sources of CIV #atpconf 9/28/2024 47
Usability studies Think aloud methods/cognitive labs Conduct at different points in the development cycle Early-stage using lower fidelity prototypes Late-stage using higher fidelity prototypes Involve subjects that represent the target population, where possible #atpconf 9/28/2024 48
Item templates Identify static and dynamic item elements Constrain the variability each individual item author can contribute Create a uniform user experience Provide a means for item creation consistent with universal design principles #atpconf 9/28/2024 49
In the broad context of item writing and test development, ensuring test accessibility is a simple exercise of employing best practice. ~Rodriguez, 2011, p. 202 #atpconf 9/28/2024 50
