Implications of Universal Design, Accessibility, and Fairness in Test Development

Universal design, accessibility & fairness
Implications for test and item development
Cynthia G. Parshall, Belinda Brunner, & Dwan Bovell
Overview of Session
Common test accommodations &
types of assistive technology
implications for innovative item types;
Accessibility and item interface designs
QTI coding for screen reader software
Item and test development procedures
for accessibility
Especially for innovative item types.
12/2/2014
2
9/28/2024
3
Common Test Accommodations 
& Types of Assistive Devices
Exam Sponsor Overview
ABIM tests approximately 35,000
physicians every year
3 separate test administrations
Spring Maintenance of Certification (MOC) only
Summer Initial Certification only
Fall Subspecialty Certification and MOC
18 Subspecialties (Card, Rheum, etc.)
36 exam types due to separate Certification
and MOC exams
12/2/2014
4
Exam Sponsor Overview
Approximately 1 percent of physicians
receive any type of accommodation
per year
307 in 2015
Typical process for accommodations
Physician applies for accommodation by
application deadline
Application is reviewed by staff and consultants
Application is denied or approved
Exam containing accommodation is created
12/2/2014
5
Exam Sponsor Overview
Commonly approved accommodations
Additional testing time; Extra break time; Large-
font exam materials; Separate testing room,
Trackball mouse; Custom testing schedule;
Anti-glare computer screen; Left-handed mouse
set-up;
Permission to have and use diabetic supplies;
Permission to have and use light provisions,
including water in a spill-proof bottle and/or a
small snack
Portable air supply; Prescribed medication
12/2/2014
6
Types of accommodated exams
9/28/2024
7
Types of Accommodations, and related need
Accommodations, Based on Innovation
9/28/2024
10
Accessibility and 
Item Interface Design
Goals of this part of the
presentation…
To describe a universal design
perspective for the exam user interface
Build in a maximally useful design, and/or
build in user-control
So user interface accommodations are rarely
needed
To give an 80:20 set of suggestions
To address about 80% of the accessibility needs,
with 20% of the effort
Focusing on 
just 
the user interface
Universal Design
9/28/2024
12
Ways of looking at this
1.
By type of accommodation
And which needs might be met through this
2.
By types of disability/needs
And what accessibility choices or
accommodations might be needed
3.
By type of item or test element
And whether it could impact certain needs;
and if so, what accessibility choices could help
Types of Disability and Accessibility
Types of Disability and Accessibility
See 
 
http://www.creativebloq.com/netmag/simple-introduction-web-accessibility-7116888
But first…
“Conflicting access needs”
When one user’s need is in conflict with another’s
Testing example
Conflicting access is:
-
Certain color schemes that work well for users with low vision
are problematic for users with color-blindness
Possible solution:
-
Good default color scheme
plus user-control over alternatives
Special challenge in testing
Some accommodations change the construct
When a construct change is needed –
"adaptations [that] change the intended construct
to make it accessible 
. . .
while retaining 
as much of the original construct as possible
."
 
(AERA, APA, & NCME, 2014, p 59).
9/28/2024
15
Recommended actions for all exams
Design a 
user interface
 that is clear
and simple
Minimal use of panels, windows,
reference to other pages, etc.
Clear “grouping” of related elements
Minimal scrolling
Helpful to
: users of screen readers & screen
magnifiers; users with certain learning disabilities
Images
Images
Include alt-text
-
For all images, icons, etc.
QTI code:
-
alt=“chest x-ray”
-
alt=“Exhibit button”
-
alt=“”
Helpful to
: users of screen readers
Graphs and Illustrations
Graphs & Illustrations
Include a second signifier for any color-coding
-
e.g., red cross-hatching, green pointillism
Place keys and legends above or to the right of
the image
-
This makes them easier to find in braille
Helpful to
: users with color blindness;
users of refreshable braille displays
Media
Audio
Text descriptions
Helpful to
: users with
hearing disabilities
Video
Captioning for video with
dialogue
Helpful to
: users with
hearing disabilities
     __________________
Brief “scene setting”, to
preface recorded dialogue
Helpful to
: users with visual
disabilities
Color and Contrast
Color / contrast
Provide user-select color accommodation
-
To change foreground and background colors
When color is used as a signifier, always include a
second signifier
-
e.g., bold for text; cross-hatching for image
Verify that the contrast is sufficient
-
e.g., through websites such as:
 
http://webaim.org/resources/contrastchecker/
Helpful to
: users with color blindness or low vision;
users with tired eyes
Fonts
Font
Use an appropriate font type and size
-
e.g., Tahoma, 14-pt
Avoid problematic font displays
-
e.g., italics, underlining, etc.
Consider formatting of text
-
With sufficient white space and leading (i.e., amount of
vertical space between lines of text)
-
Left-justification
-
Line length of 70-80 characters maximum
If possible, provide user-control over sizing
-
e.g., user-adjustable font size
Helpful to
: users with low vision; users with tired eyes
Mouse Use
Mouse use
Make buttons and other selectable areas:
-
Large enough to avoid the need for precise mouse movements
-
Close to the associated text to avoid the need for excessive
mousing
Provide keyboard alternatives (hotkeys),
whenever possible
-
"Most assistive technologies for people with motor disabilities
either work through the keyboard or emulate the functionality
of the keyboard."
-
http://webaim.org/articles/motor/assistive
Helpful to
: users with motor disabilities;
users with repetitive strain injuries
Tables
Tables
Include headers
-
Consider including advance organizers or other context
information on the item screen
Consider the use of html code
-
to specify the table 
content
, and
-
to identify the 
row
 and 
column headers
-
See 
http://webaim.org/techniques/tables/data
Helpful to
: users of screen readers;
users of screen magnifiers
Sample Table and QTI code
QTI elements:
<caption>
<
td
>
<th>
<scope>
<table>
<caption>States, Capitals, and Populations</caption>
<tr>
<th 
scope="col"
>US State</th>
<th 
scope="col"
>State Capital</th>
<th 
scope="col"
>Population of Capital</th>
</tr>
<tr>
<th 
scope="row"
>Alabama</th>
<td>Montgomery</td>
<td>200,983</td>
</tr>
<tr>
<th 
scope="row"
>Alaska</th>
<td>Juneau</td>
<td>31,118</td>
</tr>
</table>
9/28/2024
24
States, Capitals, and Populations
Executable Applications
Executables, to be accessed within
test driver
Ensure good use of color and contrast
Ensure a large enough display
(or user-control over display size)
Provide keyboard alternatives for all
mouse actions
Helpful to
: users with color blindness; users with
low vision; users with motor challenges
9/28/2024
26
Item and Test Development 
for Accessibility
Goals of this part of the
presentation…
To discuss item and test development
according to universal design principles
To place this development in the
context of construct validity
To take a closer at innovative item
development, in particular
“’Universally designed assessments’ are
designed and developed from the
beginning
 to allow participation of the
widest possible range of students, and to
result in valid inferences about
performance for all students who
participate in the assessment”
~Thompson, S. J., Johnstone, C. J., &
Thurlow, M. L. (2002, p.6).
9/28/2024
28
Elements of universally designed
assessments (NCEO)
Inclusive assessment population
Precisely defined constructs
Accessible, non-biased items
Amenable to accommodations
Simple , clear, & intuitive instructions &
procedures
Maximum readability & comprehensibility
Maximum legibility
See http://www.cehd.umn.edu/nceo/
Developing universally designed
assessments (NCEO)
Test conceptualization
Test construction
Field testing
Item analysis
Item reviews
Test conceptualization
What
 is to be assessed?
Precisely defined test constructs
How
 will it be assessed?
Test design which considers individual needs
Key question:
After an item is written, is the item
reflective of the desired construct?
9/28/2024
31
Construct validity & accessibility
Goal:
Preserve construct-related validity while
increasing access
Threat:
Construct-irrelevant features interfere
with test taker performance on the
intended test construct due to test taker
characteristics
9/28/2024
32
Construct-Centered  Approach  for
Universally Designed Assessments
1.
Develop as complete an
understanding as possible of the test
construct and assessment intent
What do you want to measure?
Targeted/focal KSAs
What else affects successful
performance?
Ancillary/non-focal KSAs
9/28/2024
33
Construct-Centered  Approach  for
Universally Designed Assessments
9/28/2024
34
Construct-Centered  Approach  for
Universally Designed Assessments
2.
Determine how test taker performance
can be supported on the ancillary KSAs
Examples:
Formula sheets
Vocabulary lists
Alternative ways to move objects
9/28/2024
35
Item development for universally
designed assessments
Introduce item authors/reviewers to
universal design concept
Provide style guidelines that consider
the elements of universal design
Include bias/sensitivity review as part of
the item review process
9/28/2024
36
Style guidelines example −
Amenable to accommodations
Which hospital has the lowest average number of Cesarean sections per 100 births over the
three years?
Style guidelines example −
Amenable to accommodations
Which hospital has the lowest average number of Cesarean sections per 100 births over the
three years?
Style guidelines example –
Use of language
Elizabeth Salinger is the head of the department.
Better: Liz is the head of the department.
Most ungulates have common characteristics which
include having hooves and primarily eating plant material.
Better: Most ungulates have common characteristics.
These characteristics include having hooves and being
plant eaters.
The object of the game was to get as few points as
possible.
Better: The purpose of the game was to get as few
points as possible.
The quintessential characteristic of an epic hero is
courage.
Better:  The main characteristic of an epic hero is
courage.
9/28/2024
39
9/28/2024
40
Universal design
& innovative items
What is an innovative item?
An item that goes 
beyond the “discrete,
text-based, multiple-choice format”
(Parshall, et al., 2000)
Computer-delivered items that include
specialized mechanisms for interacting
with and responding to the item
9/28/2024
41
Why use innovative items?
To more fully represent the test construct
To assess greater depths and a broader
spectrum of knowledge, skills & abilities
than can be done with traditional item
types
9/28/2024
42
Universal design of innovative
items (Dolan, 2010)
Any item is a collection of components.
Innovative items allow for a greater
variety of components:
Text
Tables
Graphs
Images
Audio
Video
Animations
12/2/2014
43
Universal design of innovative
items (Dolan, 2010)
Test takers interact with each
component through a variety of
processing mechanisms:
Perceptual
Linguistic
Cognitive
Motoric
Executive
Affective
12/2/2014
44
Universal design of innovative
items (Dolan, 2010)
Variance results from the interplay of
item components and test taker
processing.
Variances related to the target KSAs
contribute to valid score interpretation.
Variances not related to the target KSAs
create construct-irrelevant variance
(CIV).
12/2/2014
45
Universal design of innovative
items (Dolan, 2010)
Consider design options for innovative
items which minimize CIV measurement.
Examples:
12/2/2014
46
Item design
Universal design for computer-based testing (UD-CBT)
guidelines (Dolan, et al., 2010)
9/28/2024
47
Usability studies
Think aloud methods/cognitive labs
Conduct at different points in the
development cycle
Early-stage using lower fidelity prototypes
Late-stage using higher fidelity prototypes
Involve subjects that represent the
target population, where possible
9/28/2024
48
Item templates
Identify static and dynamic item
elements
Constrain the variability each individual
item author can contribute
Create a uniform user experience
Provide a means for item creation
consistent with universal design
principles
9/28/2024
49
In the broad context of item writing and
test development, ensuring test
accessibility is a simple exercise of
employing best practice.
~Rodriguez, 2011, p. 202
9/28/2024
50
References
Dolan, et al. (October, 2010). Universal design for computer-based
testing (UD-CBT) guidelines. Retrieved 3 March 2016 from
www.pearsonassessments.com/udcbt
 .
Elliott, S.N., Kettler, R.J., Beddow, P.A., Kurz, A. (Eds.)(2011).  
Handbook
of Accessible Achievement Tests for All Students
. New York: Springer.
Johnstone, C., Altman, J., & Thurlow, M. (2006). 
A State Guide to the
Development of Universally Designed Assessments
. Minneapolis, MN:
University of Minnesota, National Center on Educational Outcomes.
Retrieved February 23, 2016 from
http://www.cehd.umn.edu/nceo/OnlinePubs/StateGuideUD/UDmanua
l.pdf
Mislevy, R. J., Hamel, L., Fried, R., Gaffney, T., Haertel, G., Hafter, A., et
al. (2003). 
Design 
patterns for assessing science inquiry 
(Principled
Assessment Designs for Inquiry [PADI] Technical Rep. No. 1). Menlo Park,
CA: SRI International.
Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002). 
Universal
design applied to large scale assessments 
(Synthesis Report 44).
Minneapolis, MN: University of Minnesota, National Center on
Educational Outcomes.
9/28/2024
51
Thank you!
Cynthia G. Parshall, PhD
Measurement Consultant, Touchstone Consulting
CParshall@Touchstone.Consulting
Belinda Brunner
Test Development Strategist, Pearson VUE
Belinda.Brunner@Pearson.com
Dwan Bovell
Director, Exam Administration, ABIM
DBovell@ABIM.ORG
Slide Note
Embed
Share

This content discusses the implications of universal design, accessibility, and fairness in test and item development. It covers topics such as common test accommodations, types of assistive technology, innovative item types, accessibility in item interface designs, QTI coding for screen reader software, and procedures for developing accessible tests. It also provides insights into the exam sponsorship process, including accommodations commonly approved for physicians, such as additional testing time and custom testing schedules.


Uploaded on Sep 28, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Universal design, accessibility & fairness Implications for test and item development Cynthia G. Parshall, Belinda Brunner, & Dwan Bovell #atpconf

  2. Overview of Session Common test accommodations & types of assistive technology implications for innovative item types; Accessibility and item interface designs QTI coding for screen reader software Item and test development procedures for accessibility Especially for innovative item types. #atpconf 12/2/2014 2

  3. Common Test Accommodations & Types of Assistive Devices #atpconf 9/28/2024 3

  4. Exam Sponsor Overview ABIM tests approximately 35,000 physicians every year 3 separate test administrations Spring Maintenance of Certification (MOC) only Summer Initial Certification only Fall Subspecialty Certification and MOC 18 Subspecialties (Card, Rheum, etc.) 36 exam types due to separate Certification and MOC exams #atpconf 12/2/2014 4

  5. Exam Sponsor Overview Approximately 1 percent of physicians receive any type of accommodation per year 307 in 2015 Typical process for accommodations Physician applies for accommodation by application deadline Application is reviewed by staff and consultants Application is denied or approved Exam containing accommodation is created #atpconf 12/2/2014 5

  6. Exam Sponsor Overview Commonly approved accommodations Additional testing time; Extra break time; Large- font exam materials; Separate testing room, Trackball mouse; Custom testing schedule; Anti-glare computer screen; Left-handed mouse set-up; Permission to have and use diabetic supplies; Permission to have and use light provisions, including water in a spill-proof bottle and/or a small snack Portable air supply; Prescribed medication #atpconf 12/2/2014 6

  7. Types of accommodated exams Double time, large print Double time Large print only Extra break time Additional break time 1 module per day - double time 1 module per day - extra break time 2 modules per day - extra break time 2 modules per day - no extra time - large print 1 module per day - no extra time - large print 2 modules per day - extra break time - large print 1 module per day - no extra time 2 modules per day - no extra time 1 module per day - double time - large print 1 module per day - extra break time - large print #atpconf 9/28/2024 7

  8. Types of Accommodations, and related need Provision Type of disability or need Provision Type of disability or need Extra time Some cognitive challenges Some physical problems (e.g., MS, back problems, pregnancy) Screen magnifier software Low vision Extra breaks Some physical problems (e.g., MS, back problems, pregnancy, nursing) Screen reader software Blindness Low vision Some cognitive challenges Separate testing room Some mobility challenges Use of a human reader/recorder Some cognitive challenges (e.g., dyslexia) Color-blindness Some types of low vision Some cognitive challenges Tired eyes Unfamiliar with computer based testing Color/ contrast Large-font test form Low vision Some cognitive needs Tired eyes Paper exam Audio exam Blindness/low vision Some cognitive challenges #atpconf

  9. Accommodations, Based on Innovation Type of innovation Type of disability or need Examples of solutions Audio Hearing impairments Test description Video Hearing impairments Captioning Video Visual impairments Possibly, provide a written script, including description of the scene and crucial elements Use of screen reader (did you mean software or human?) Caliper Mobility challenges Hotkeys #atpconf

  10. Accessibility and Item Interface Design #atpconf 9/28/2024 10

  11. Goals of this part of the presentation To describe a universal design perspective for the exam user interface Build in a maximally useful design, and/or build in user-control So user interface accommodations are rarely needed To give an 80:20 set of suggestions To address about 80% of the accessibility needs, with 20% of the effort Focusing on just the user interface #atpconf

  12. Universal Design #atpconf 9/28/2024 12

  13. Ways of looking at this 1.By type of accommodation And which needs might be met through this 2.By types of disability/needs And what accessibility choices or accommodations might be needed 3.By type of item or test element And whether it could impact certain needs; and if so, what accessibility choices could help #atpconf

  14. Types of Disability and Accessibility Type of Disability/Need Accessibility Solutions Visual e.g., blind users, users with low-vision, users with obstructed vision, and users with digital eye strain Maintain good size and contrast Use real text vs. image of text Include alt text for images, etc. Provide captioning for video dialogue Provide text description for non-dialogue sound files Avoid auto-play of audio or video Don t make clickable elements too small or too close together Don t rely solely on complex movements such as drag-and-drop Provide hotkey alternatives Auditory e.g., deaf users, hearing impaired users, users with auditory processing issues Motor - users of assistive devices like: specialized keyboards, eye trackers, button switches Cognitive e.g., users with dyslexia, autism, ADD, Down s syndrome, etc. Provide information in a straightforward, way Maintain a clean interface #atpconf See http://www.creativebloq.com/netmag/simple-introduction-web-accessibility-7116888

  15. But first Conflicting access needs When one user s need is in conflict with another s Testing example Conflicting access is: - Certain color schemes that work well for users with low vision are problematic for users with color-blindness Possible solution: - Good default color scheme plus user-control over alternatives Special challenge in testing Some accommodations change the construct When a construct change is needed "adaptations [that] change the intended construct to make it accessible . . . while retaining as much of the original construct as possible." (AERA, APA, & NCME, 2014, p 59). #atpconf 9/28/2024 15

  16. Recommended actions for all exams Design a user interface that is clear and simple Minimal use of panels, windows, reference to other pages, etc. Clear grouping of related elements Minimal scrolling Helpful to: users of screen readers & screen magnifiers; users with certain learning disabilities #atpconf

  17. Images Images Include alt-text - For all images, icons, etc. QTI code: - alt= chest x-ray - alt= Exhibit button - alt= Helpful to: users of screen readers #atpconf

  18. Graphs and Illustrations Graphs & Illustrations Include a second signifier for any color-coding - e.g., red cross-hatching, green pointillism Place keys and legends above or to the right of the image - This makes them easier to find in braille Helpful to: users with color blindness; users of refreshable braille displays #atpconf

  19. Media Audio Text descriptions Video Captioning for video with dialogue Helpful to: users with hearing disabilities Helpful to: users with hearing disabilities __________________ Brief scene setting , to preface recorded dialogue Helpful to: users with visual disabilities #atpconf

  20. Color and Contrast Color / contrast Provide user-select color accommodation - To change foreground and background colors When color is used as a signifier, always include a second signifier - e.g., bold for text; cross-hatching for image Verify that the contrast is sufficient - e.g., through websites such as: http://webaim.org/resources/contrastchecker/ Helpful to: users with color blindness or low vision; users with tired eyes #atpconf

  21. Fonts Font Use an appropriate font type and size - e.g., Tahoma, 14-pt Avoid problematic font displays - e.g., italics, underlining, etc. Consider formatting of text - With sufficient white space and leading (i.e., amount of vertical space between lines of text) - Left-justification - Line length of 70-80 characters maximum If possible, provide user-control over sizing - e.g., user-adjustable font size Helpful to: users with low vision; users with tired eyes #atpconf

  22. Mouse Use Mouse use Make buttons and other selectable areas: - Large enough to avoid the need for precise mouse movements - Close to the associated text to avoid the need for excessive mousing Provide keyboard alternatives (hotkeys), whenever possible - "Most assistive technologies for people with motor disabilities either work through the keyboard or emulate the functionality of the keyboard." - http://webaim.org/articles/motor/assistive Helpful to: users with motor disabilities; users with repetitive strain injuries #atpconf

  23. Tables Tables Include headers - Consider including advance organizers or other context information on the item screen Consider the use of html code - to specify the table content, and - to identify the row and column headers - See http://webaim.org/techniques/tables/data Helpful to: users of screen readers; users of screen magnifiers #atpconf

  24. Sample Table and QTI code States, Capitals, and Populations <table> <caption>States, Capitals, and Populations</caption> US State State Capital Population of Capital <tr> <th scope="col">US State</th> <th scope="col">State Capital</th> <th scope="col">Population of Capital</th> </tr> Alabama Montgomery 200,983 Alaska Juneau 31,118 <tr> <th scope="row">Alabama</th> <td>Montgomery</td> <td>200,983</td> </tr> QTI elements: <caption> <td> <th> <scope> <tr> <th scope="row">Alaska</th> <td>Juneau</td> <td>31,118</td> </tr> </table> #atpconf 9/28/2024 24

  25. Executable Applications Executables, to be accessed within test driver Ensure good use of color and contrast Ensure a large enough display (or user-control over display size) Provide keyboard alternatives for all mouse actions Helpful to: users with color blindness; users with low vision; users with motor challenges #atpconf

  26. Item and Test Development for Accessibility #atpconf 9/28/2024 26

  27. Goals of this part of the presentation To discuss item and test development according to universal design principles To place this development in the context of construct validity To take a closer at innovative item development, in particular #atpconf

  28. Universally designed assessments are designed and developed from the beginning to allow participation of the widest possible range of students, and to result in valid inferences about performance for all students who participate in the assessment ~Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002, p.6). #atpconf 9/28/2024 28

  29. Elements of universally designed assessments (NCEO) Inclusive assessment population Precisely defined constructs Accessible, non-biased items Amenable to accommodations Simple , clear, & intuitive instructions & procedures Maximum readability & comprehensibility Maximum legibility See http://www.cehd.umn.edu/nceo/ #atpconf

  30. Developing universally designed assessments (NCEO) Test conceptualization Test construction Field testing Item analysis Item reviews #atpconf

  31. Test conceptualization What is to be assessed? Precisely defined test constructs How will it be assessed? Test design which considers individual needs Key question: After an item is written, is the item reflective of the desired construct? #atpconf 9/28/2024 31

  32. Construct validity & accessibility Goal: Preserve construct-related validity while increasing access Threat: Construct-irrelevant features interfere with test taker performance on the intended test construct due to test taker characteristics #atpconf 9/28/2024 32

  33. Construct-Centered Approach for Universally Designed Assessments 1. Develop as complete an understanding as possible of the test construct and assessment intent What do you want to measure? Targeted/focal KSAs What else affects successful performance? Ancillary/non-focal KSAs #atpconf 9/28/2024 33

  34. Construct-Centered Approach for Universally Designed Assessments Target KSAs Ancillary KSAs What else affects successful performance? What do you want to measure? Examples Quantitative reasoning Math ability Math ability (in a word problem) Verbal ability Ability to prioritize treatment options (through a drag-&-drop item) Ability to use the mouse #atpconf 9/28/2024 34

  35. Construct-Centered Approach for Universally Designed Assessments 2.Determine how test taker performance can be supported on the ancillary KSAs Examples: Formula sheets Vocabulary lists Alternative ways to move objects #atpconf 9/28/2024 35

  36. Item development for universally designed assessments Introduce item authors/reviewers to universal design concept Provide style guidelines that consider the elements of universal design Include bias/sensitivity review as part of the item review process #atpconf 9/28/2024 36

  37. Style guidelines example Amenable to accommodations Which hospital has the lowest average number of Cesarean sections per 100 births over the three years? Cesarean section rates by hospital 2011-2013 20 Number of Cesarean sections per 100 births 18 16 14 12 10 8 6 4 2 0 Hospital A Hospital B Hospital C Hospital D 2011 2012 2013 #atpconf

  38. Style guidelines example Amenable to accommodations Which hospital has the lowest average number of Cesarean sections per 100 births over the three years? Cesarean section rates by hospital 2011-2013 Number of Cesarean sections per 100 births 2011 2012 2013 20 18 16 14 12 10 8 6 4 2 0 Hospital A Hospital B Hospital C Hospital D #atpconf

  39. Style guidelines example Use of language Elizabeth Salinger is the head of the department. Better: Liz is the head of the department. Most ungulates have common characteristics which include having hooves and primarily eating plant material. Better: Most ungulates have common characteristics. These characteristics include having hooves and being plant eaters. The object of the game was to get as few points as possible. Better: The purpose of the game was to get as few points as possible. The quintessential characteristic of an epic hero is courage. Better: The main characteristic of an epic hero is courage. #atpconf 9/28/2024 39

  40. Universal design & innovative items 9/28/2024 40

  41. What is an innovative item? An item that goes beyond the discrete, text-based, multiple-choice format (Parshall, et al., 2000) Computer-delivered items that include specialized mechanisms for interacting with and responding to the item #atpconf 9/28/2024 41

  42. Why use innovative items? To more fully represent the test construct To assess greater depths and a broader spectrum of knowledge, skills & abilities than can be done with traditional item types #atpconf 9/28/2024 42

  43. Universal design of innovative items (Dolan, 2010) Any item is a collection of components. Innovative items allow for a greater variety of components: Text Tables Graphs Images Audio Video Animations #atpconf 12/2/2014 43

  44. Universal design of innovative items (Dolan, 2010) Test takers interact with each component through a variety of processing mechanisms: Perceptual Linguistic Cognitive Motoric Executive Affective #atpconf 12/2/2014 44

  45. Universal design of innovative items (Dolan, 2010) Variance results from the interplay of item components and test taker processing. Variances related to the target KSAs contribute to valid score interpretation. Variances not related to the target KSAs create construct-irrelevant variance (CIV). #atpconf 12/2/2014 45

  46. Universal design of innovative items (Dolan, 2010) Consider design options for innovative items which minimize CIV measurement. Examples: Item component Processing Ancillary KSA Design option to address CIV Text Linguistic Vocabulary Vocabulary lists Text (calculations) Cognitive Background knowledge Formulae sheets Graphs Perceptual Color perception Avoid common color-blindness combinations Video Executive Working memory Note-taking tools #atpconf 12/2/2014 46

  47. Item design Universal design for computer-based testing (UD-CBT) guidelines (Dolan, et al., 2010) Revise as necessary and re- evaluate Evaluate item design for construct validity Evaluate item design for sources of CIV #atpconf 9/28/2024 47

  48. Usability studies Think aloud methods/cognitive labs Conduct at different points in the development cycle Early-stage using lower fidelity prototypes Late-stage using higher fidelity prototypes Involve subjects that represent the target population, where possible #atpconf 9/28/2024 48

  49. Item templates Identify static and dynamic item elements Constrain the variability each individual item author can contribute Create a uniform user experience Provide a means for item creation consistent with universal design principles #atpconf 9/28/2024 49

  50. In the broad context of item writing and test development, ensuring test accessibility is a simple exercise of employing best practice. ~Rodriguez, 2011, p. 202 #atpconf 9/28/2024 50

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#