History of BEI Certification Development
BEI Certification Overview
BEI Certification Overview
Lauri Metcalf
L. Metcalf 2017 Copy only with permission
How it all began!
In 1994, Texas Commission for the Deaf and Hard of
Hearing and the Board for Evaluation of
Interpreters brought in for the first time, outside
experts in the field of Sign Language Interpreting
and certification.
From 1994 to 2001, the BEI worked with numerous
people including Daniel Burch, Gary Sanderson,
Kathy Friedenreich and Dr. Carol Patrie to establish
the need for a new BEI certification exam.
L. Metcalf 2017 Copy only with permission
BEI Test Development Activities
Spring 2001 - The Texas Commission for the Deaf and Hard
of Hearing entered into a contract with the University of
Arizona’s National Center for Interpretation to develop
new testing materials for the agency’s interpreter
certification program.
In June, 2001 - The first test development meeting was
held in Austin to discuss the process and the initial work on
the job analysis began.
Spring of 2001 to Summer of 2006 – The BEI Panel of
Experts worked closely with the University of Arizona’s
National Center for Interpretation to develop the new BEI
Certification Exam.
l metcalf 2013 Copy only with permission
L. Metcalf 2017 Copy only with permission
Test Development Activities, Cont.
January 2007 – DHHS began administering new testing system.
March 2009 – DHHS entered into Licensing Agreement with
Michigan.
January- September 2009 - Development of parallel scripts and
final approval of scripts.
April 2010 DHHS entered into Licensing Agreement with
Illinois.
July 2014 DHHS entered into Licensing Agreement with
Missouri.
L. Metcalf 2017 Copy only with permission
Test Development Activities Cont.
April 2010 – DHHS entered into Licensing
Agreement with Illinois to establish Illinois BEI
June 2010 - Illinois BEI Rater Training
July 2014 - Missouri BEI Rater Training
L. Metcalf 2017 Copy only with permission
University of Arizona
National Center for Interpretation
Dr. Roseann Gonzalez was the key expert in the
development of the Federal Court Interpreter
Certification Exam
This exam has served as a model for high-stakes
spoken language interpreting proficiency exams
This is the model that the BEI adopted for the Sign
Language Interpreting Exam
L. Metcalf 2017 Copy only with permission
University of Arizona
National Center for Interpretation
Their examination process conforms to universally accepted
standards of constructing, administering and evaluation of
employment tests as prescribed by:
American Education Research Association
American Psychological Association
National Council on Measurement in Education (Standards
of Educational and Psychological Testing)
L. Metcalf 2017 Copy only with permission
University of Arizona
National Center for Interpretation
From all the experts we learned how important
it was to develop an evaluation that was valid,
reliable, and legally defensible.
L. Metcalf 2017 Copy only with permission
The purpose of the BEI Assessment is to
answer an important question.
Does this person have the requisite interpreting skills
to carry out the responsibilities of a BEI-certified
interpreter at this level?
When we certify candidates, we are saying that at
this point in time they possess the
minimal
level of
proficiency required at their level of certification –
not the maximum or even average level of ability
when compared to their peers.
L. Metcalf 2017 Copy only with permission
Overarching Goals of the BEI Interpreter
Certification Process
Ensure that consumers have equal access to rights,
services, and education.
Produce tests that are empirically grounded, fair,
reliable, valid, and legally defensible.
Identify proficient interpreters at different credential
levels.
L. Metcalf 2017 Copy only with permission
Developing the Evaluation was a Three-
Phase Process
1. Job/Language Analysis to document
specifications for construct and content validity
2. Develop Test of Language Proficiency
3. Develop Interpreter Performance Examinations
L. Metcalf 2017 Copy only with permission
Survey Structure
Questions we wanted answered
:
Who is currently interpreting?
Where are they working?
What tasks are they performing and how often?
What
knowledge
,
skills
, and
abilities
must
interpreters possess to perform their
tasks
?
How might the certification process be improved to
ensure that interpreters possess these skills?
L. Metcalf 2017 Copy only with permission
Survey Sample
Survey was mailed to 588 incumbents.
All Level II through V interpreters and 10% of Level I
who had been certified over 5 years.
Returned surveys evaluated for any bias with no
correlation found.
Job analysis completed Fall of 2002.
L. Metcalf 2017 Copy only with permission
Example of Findings
Question: In what settings are interpreters being
most used and what level of certification do they
hold?
Question: What skills do you need in order to be
successful at your job?
(1,632 pieces of data per questionnaire - 306,816 pieces of data
total)
L. Metcalf 2017 Copy only with permission
Implications for Test Development
Expert Panel opinion and Job Analysis data
concurred that a three-level certification system
was justified: Basic, Advanced, and Master.
Basic
: Emphasis on educational settings
Advanced
: Emphasis on routine medical,
routine legal, government, and routine mental
health settings.
Master
: Emphasis on complex medical,
complex legal, and complex mental health
settings.
L. Metcalf 2017 Copy only with permission
Implications for Test Development
In addition, Sight Translation is introduced as
a new component of the BEI Performance
Exam.
L. Metcalf 2017 Copy only with permission
Implications for Test Development
Based on the University of Arizona’s testing model, the
job analysis and input from the Expert panel a two-
stage process was decided upon:
Written Test of English Proficiency (TEP)
Interpreting Performance Exam
L. Metcalf 2017 Copy only with permission
Ethics and Protocol Testing
Knowledge of the Standards of Ethical Behavior and
Deaf culture and history are essential to the
responsible fulfillment of interpreter duties.
It was recommended that these areas should be
incorporated into training and continuing education
programs,
not
into the certification exam.
L. Metcalf 2017 Copy only with permission
Justification for this decision
The type of knowledge tested does not directly
relate to the ability to accurately interpret.
They reflect a fairly narrow scope of knowledge,
are easily learned and memorized, and may skew
test results so that deficiencies in other areas,
such as English proficiency, are camouflaged.
L. Metcalf 2017 Copy only with permission
Test of English Proficiency
Screening tool
Measure of essential KSATS
Allows for the assessment of a broader and more
representative sample of a candidates language
proficiency
The 5 sections of the test were designed to assess
candidates proficiency at the lexical, syntactical and
discourse levels
Test items were written to assess the level of English
proficiency at the 11
th
and 12
th
grade
level as identified in the job/linguistic analysis
L. Metcalf 2017 Copy only with permission
Phase 2: Development of TEP
Do candidates posses the minimum level of
English proficiency required to advance to the
performance examination?
16 items in each of 5 subsections
80 items on the final Test of English Proficiency
Test items are multiple choice format
L. Metcalf 2017 Copy only with permission
Development of Pilot Test
A valid written proficiency test
requires pilot testing to select items
that are at the appropriate level of
difficulty and are able to discern
between acceptable and unacceptable
levels of proficiency.
L. Metcalf 2017 Copy only with permission
Development of Pilot Test
BEI Expert Panel and UA NCITRP staff
collaborated in the development and
revision of 161 total initial items for
piloting. After statistical analysis, the
final 80 items were chosen and divided
into five subsections.
L. Metcalf 2017 Copy only with permission
Five Subsections of the TEP
Reading Comprehension
Synonyms
Grammar and Usage
Sentence Completion
Antonyms
L. Metcalf 2017 Copy only with permission
Phase 3
Developing the Performance Exams
Establishing Exam Format
Scripting Process
Filming of Performance Exam
Scoring Methodology
Pilot Testing
L. Metcalf 2017 Copy only with permission
Introduction to
Interpreter Performance Test
Three certification Levels:
Basic
Advanced
Master
L. Metcalf 2017 Copy only with permission
Performance Test Format
Factors Differentiating the Exam Levels:
Complexity of Language
Complexity of Topics/Settings
Speed of Speaker/Signer
L. Metcalf 2017 Copy only with permission
Performance Test Format
Factors Differentiating the Exam Levels:
Basic ~ 110 wpm
(
Click to hear a sample
.)
Advanced ~ 120 wpm
Master ~ 130 wpm
L. Metcalf 2017 Copy only with permission
Scripting Process
Scripting involved attending to a host of issues to
ensure that the resulting exams contain a
representative sample of the relevant aspects of
languages and language abilities required of
interpreters.
Begins with Job Analysis, Review of Current Exams,
and BEI Expert Panel
Hundreds of hours and dozens of drafts
L. Metcalf 2017 Copy only with permission
“Loading” the Scripts
Register Variation
Authentic Language
ASL & English
Breadth and Depth of
Vocabulary
Specialized
Terminology
Grammatical & Linguistic
Elements
Consistent Density
Colloquial & Idiomatic
Speech
Fingerspelling &
Numbers
Proper Nouns
Affect
. . . and so on
L. Metcalf 2017 Copy only with permission
Scoring System
Purpose:
to provide a replicable, fair, and valid device
for assessing the interpreting proficiency of candidates
for certification.
The function of this system is to assess an interpreter’s
ability to transform meaning from the source language
and accurately convey
the same meaning
in the target
language.
L. Metcalf 2017 Copy only with permission
Two-Part System
Objective scoring
used specifically to determine
candidates’ interpreting proficiency by evaluating
underlined scoring units.
Subjective
scoring
will supplement this by
holistically evaluating candidates’
Delivery
,
Adaptability
,
and
Pronunciation/Fluency
.
L. Metcalf 2017 Copy only with permission
What the Exams Measure
Objective Scoring System
Interpreting Proficiency
The ability to meaningfully and accurately
understand, produce, and transform ASL to and from
English in a culturally appropriate way.
L. Metcalf 2017 Copy only with permission
Objective Assessment
The objective assessment of a candidate’s level of
interpreting proficiency will be determined by how
many
Objective Scoring Units
the candidate renders
appropriately.
Objective
Scoring Units
are contained in every part of
each exam. They represent significant words, phrases,
and clauses that are critical to communication.
L. Metcalf 2017 Copy only with permission
Examples of Objective Assessment
•
Specialized Terminology
•
Register Variation
•
Affect/ Other Rhetorical features
•
General vocabulary
•
Grammatical structures
L. Metcalf 2017 Copy only with permission
Examples of Objective Assessment
•
Idiomatic language
•
Use of classifiers and non-manual markers
•
Accuracy of fingerspelling
•
Use of sign space and grammatical space
L. Metcalf 2017 Copy only with permission
Sample Script from
the DARS Study Guide
L. Metcalf 2017 Copy only with permission
Subjective Scoring System
Holistic Scoring
Three Dimensions:
Delivery
—is the ability to maintain appropriate delivery, pacing,
coherence, and composure consistency throughout the
interpretation
Adaptability
—is the level of resourcefulness the candidate display in
adapting to changes, patterns, and challenges in the text
Subjective Scoring
Scale
3 = Exceeds Expectations
2 =Meets Expectations
1 = Fails to Meet
Expectations
L. Metcalf 2017 Copy only with permission
Pronunciation/Fluency
•
Scored holistically for candidate feedback only
•
Pronunciation
—is the ability to produce spoken language, including
accurate English phonology, and to produce clear signs, as well as the
appropriate use of rhythm, stress, and intonation, without interfering
with meaning or undermining comprehensibility
•
Fluency
—is the ease with which a candidate can produce native-like
language, including the degree of hesitation and the clarity of signs
L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam
In this section, candidates will be asked to watch a
video recording of
spoken English and render it into
ASL
.
Expressive Interpreting
L. Metcalf 2017 Copy only with permission
Expressive Transliterating
In this section, candidates will be asked to watch a
video recording of
spoken English and render it into
signed English
.
Parts of the Performance Exam
L. Metcalf 2017 Copy only with permission
Comparison of
Interpreting vs. Transliterating
Click to launch YouTube
Remember, grammatical perfection is not the goal
Expectation for grammatical mastery increases with
the Advanced and Master Evaluations
L. Metcalf 2017 Copy only with permission
Receptive
Only the Basic Level test has a section entitled,
“Receptive.” In this section of the Basic Level test,
candidates will be asked to watch a video recording
of sign (some combination of ASL and signed English)
and render it into spoken English.
Parts of the Performance Exam
L. Metcalf 2017 Copy only with permission
Receptive Interpreting
The Advanced and Master Level tests both include
a section entitled, “Receptive Interpreting.” In this
section, candidates will be asked to watch a video
recording of a presentation in
ASL and render it
into spoken English
.
Parts of the Performance Exam
L. Metcalf 2017 Copy only with permission
Receptive Transliterating
The Advanced and Master Level tests both include
a section entitled, “Receptive Transliterating.” In
this section, candidates will be asked to watch a
video recording of a presentation in
signed English
and render it into spoken English
.
Parts of the Performance Exam
L. Metcalf 2017Copy only with permission
Sight Translation
The Sight Translation part of each performance
test differs from the other parts of the tests in that
it does not include a pre-recorded video stimulus.
At the beginning of this section, the candidate will
be instructed to sight translate a short,
written
English document into ASL.
Parts of the Performance Exam
L. Metcalf 2017 Copy only with permission
Basic Level Performance Test Content
Focuses primarily on the language found in routine
educational and social service settings, as
determined by the empirical job analysis of
incumbent Texas interpreters.
Examples of possible topics include: K-12 classroom
presentations by students or teachers, special
school assemblies, homework assignments, school
memos, informational meetings or orientations.
L. Metcalf 2017 Copy only with permission
Performance Test Video Instructions
INTRODUCTION:
This is the
Expressive Interpreting
part of the exam. This part
consists of a pre-recorded simulated lecture.
In a moment, you will hear a pre-recorded “warm-up” that will introduce the
lecture. You may interpret it if you wish, but the warm-up will not be scored.
During the warm-up you may adjust the volume to your level of comfort.
Following the warm-up, you will have
one minute
to prepare for this part of the
examination, after which you will be prompted by the words, “Begin
Interpreting
Now”, which mark the beginning of this part of the exam.
Any questions?
L. Metcalf 2017Copy only with permission
Sight Translation Instructions
INTRODUCTION:
This is the Sight Translation part of the exam.
You have been
asked to sight translate a written English document into sign.
You will have a
total of seven (7) minutes both to prepare and to deliver your translation. You
may start your translation when you wish, but if you have not started in two
(2) minutes, the technician will instruct you to begin.
Each introduction will also contain a sentence describing the setting in which the
document is being used, and for whom you are interpreting. For example, the
sentence could read,
“You have been asked to sight translate a PTA flyer for
the deaf mother of an elementary school student.”
L. Metcalf 2017 Copy only with permission
Advanced Level Performance Test
Content
Focuses primarily on the language found in a variety
of high stakes settings, such as routine medical,
social services, higher education, routine mental
health, and routine quasi-legal, as determined by
the empirical job analysis of incumbent interpreters.
Examples of possible topics include: post secondary
student and professor classroom presentations,
professional development seminars, doctor or
dentist visits, application for services, employment
forms, counseling sessions.
L. Metcalf 2017 Copy only with permission
Master Level Performance Test Content
Focuses on the language found in a variety of
complex high stakes settings, such as complex
medical, complex mental health, complex quasi-legal
and complex educational settings, as determined by
an empirical job analysis of incumbent Texas
interpreters.
Example of possible topics include: patient
information forms, intake forms, contracts or contract
negotiations, meetings with medical specialists,
special education meetings, and so on.
L. Metcalf 2017 Copy only with permission
The Overarching Goal of the BEI
When we certify candidates, we are saying that at
this point in time:
they posses the
minimal
level of proficiency
required at their level of certification
not the maximum or even average level of
ability when compared to their peers
L. Metcalf 2017 Copy only with permission
Rater Qualifications
Raters must meet minimum qualifications
including certification level, years of experience,
academic or formal training.
All raters must attend training on a regular basis.
BEI uses both Deaf and hearing raters.
L. Metcalf 2017 Copy only with permission
Rater’s Recusal
Raters must recuse themselves if there is any question
they might not be impartial or objective because:
they know a candidate personally
have worked with the candidate in the past six months in any capacity
have mentored the candidate
or any other situation that could cause question or controversy
L. Metcalf 2017 Copy only with permission
Rating Sessions
Rating sessions are always held in a place of business .
Raters are assigned to a specific level and only rate
candidates taking that level.
Raters are a part of a rating team which includes a team
leader and at least 4 members of the team.
Teams change for each rating session.
Inter-rater reliability statistics are calculated for each
candidate after every rating session.
L. Metcalf 2017 Copy only with permission
Rating the Performance Exam
Raters are trained to focus only on the scoring units.
Scoring units include specific discrete language and
interpreting features.
Candidates are assessed on the basis of a representative
sample of the KSAT.
Raters evaluate candidates on exactly the same features.
L. Metcalf 2017 Copy only with permission
Total Objectivity in Scoring
Candidates are judged solely on their
interpretation performance,
not
on extraneous
factors such as mannerisms, physical appearance,
education/cultural background, regionalisms or
personal information.
L. Metcalf 2016 Copy only with permission
Objective Scoring Units
Contained in every part of each exam.
They represent significant words, phrases, and clauses that are
challenging, part of the discourse and critical to communication.
Are identified in the test of the exam by a combination of underlining
and superscript enumeration.
Each underlined and numbered scoring unit is worth one point.
50 scoring units for all sections except sight translation which has 20
scoring units.
L. Metcalf 2016 Copy only with permission
Consensus Scoring
The consensus process is not a simple arithmetic
average. It requires the raters to negotiate and agree on
the final number of errors.
The raters will often take notes on unusual or
questionable candidate renditions.
Team leaders will record the final objective score for each
test section and the subjective consensus score.
L. Metcalf 2017 Copy only with permission
Final Decision - Basic
The candidate’s level of proficiency is established by
the Objective Scoring errors.
There are
three possible outcomes
, which can lead
the raters to either
PASS
or
FAIL
the candidate based
on their performance on the exam.
L. Metcalf 2017 Copy only with permission
Final Decision - Basic
If the candidate misses
43
or fewer
objective units,
the decisions clear-cut. The candidate
passes
.
If the candidate misses
47
or more
objective scoring
units, the decision is clear-cut. The candidate
f
ails
.
L. Metcalf 2017 Copy only with permission
Final Decision - Advanced & Master
If the candidate misses
55
or fewer
objective scoring
units, the decision is clear-cut. The candidate
passes
.
If the candidate misses
59
or more
objective scoring
units, the decision is clear-cut. The candidate
fails
.
L. Metcalf 2017 Copy only with permission
Critical Range
As a means of refining the assessment process and
allowing the subjective assessment to be a more
significant factor in the decision-making process, the
cut-off number for objective unit errors is not a
discrete number. Instead, a critical range of objective
unit errors has been established.
L. Metcalf 2017 Copy only with permission
The “critical range” was created based on the
experience that an absolute objective score might
not differentiate between an acceptable and an
unacceptable performance, for several reasons:
Linguistically use of the critical range in assessment
reflects the complexity and dynamism of language.
It allows for greater consideration of the candidate's
linguistic strengths without undermining the importance of
the primary linguistic testing construct, as measured by
the objective scoring system.
L. Metcalf 2017 Copy only with permission
Basic Trigger Pass
If a candidate scores within the critical range of
44-
46
errors and the candidate subjective rating is
18
or
higher, the
candidate will pass
.
Objective score
44-46
Subjective score
18-24
L. Metcalf 2017 Copy only with permission
If a candidate scores within the critical range of
44-
46
errors and the candidate’s subjective rating is
17
or lower, the candidate fails.
Basic Trigger Fail
L. Metcalf 2017 Copy only with permission
Advanced/Master Trigger Pass
If a candidate scores within the critical range of
56-
58
errors and the candidate’s subjective rating is
24
or higher, the candidate will pass.
Objective scores
56-58
Subjective scores
24-30
L. Metcalf 2017 Copy only with permission
Advanced/Master Trigger Fail
If a candidate scores within the critical range of
56-58
errors and the candidate’s subjective rating
is
23
or
lower
, the candidate fails.
L. Metcalf 2017 Copy only with permission
What Can I Learn From My Results?
We sometimes hear from candidates that they didn’t
get any feedback with their results.
The focus of BEI is to certify interpreters, not to offer
suggestions on how to improve.
However, you can learn much about your work from
looking at the objective and subjective scores.
L. Metcalf 2017 Copy only with permission
For Example: Objective Assessment
L. Metcalf 2017 Copy only with permission
For Example: Subjective Assessment
L. Metcalf 2017 Copy only with permission
For Example: Performance Summary
L. Metcalf 2017 Copy only with permission
In Closing
The final criterion is whether the candidate has
sufficient linguistic and interpreting capabilities to
ensure that the client receives information that is as
complete and comprehensive as would a person who
did not need interpreter services.
Lauri Metcalf
laurimetcalf@gmail.com
L. Metcalf 2017 Copy only with permission
The history of BEI certification development began in 1994 when the Texas Commission for the Deaf and Hard of Hearing initiated collaboration with experts to establish a new certification exam. Over the years, partnerships were formed, test development activities were conducted, and licensing agreements were made with other states. The University of Arizona's National Center for Interpretation played a key role in shaping the BEI certification process. Dr. Roseann Gonzalez's work on the Federal Court Interpreter Certification Exam influenced the model adopted by BEI for the Sign Language Interpreting Exam.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
BEI Certification Overview BEI Certification Overview Lauri Metcalf L. Metcalf 2017 Copy only with permission
How it all began! In 1994, Texas Commission for the Deaf and Hard of Hearing and the Board for Evaluation of Interpreters brought in for the first time, outside experts in the field of Sign Language Interpreting and certification. From 1994 to 2001, the BEI worked with numerous people including Daniel Burch, Gary Sanderson, Kathy Friedenreich and Dr. Carol Patrie to establish the need for a new BEI certification exam. L. Metcalf 2017 Copy only with permission
BEI Test Development Activities Spring 2001 - The Texas Commission for the Deaf and Hard of Hearing entered into a contract with the University of Arizona s National Center for Interpretation to develop new testing materials for the agency s interpreter certification program. In June, 2001 - The first test development meeting was held in Austin to discuss the process and the initial work on the job analysis began. Spring of 2001 to Summer of 2006 The BEI Panel of Experts worked closely with the University of Arizona s National Center for Interpretation to develop the new BEI Certification Exam. L. Metcalf 2017 Copy only with permission
Test Development Activities, Cont. January 2007 DHHS began administering new testing system. March 2009 DHHS entered into Licensing Agreement with Michigan. January- September 2009 - Development of parallel scripts and final approval of scripts. April 2010 DHHS entered into Licensing Agreement with Illinois. July 2014 DHHS entered into Licensing Agreement with Missouri. L. Metcalf 2017 Copy only with permission
Test Development Activities Cont. April 2010 DHHS entered into Licensing Agreement with Illinois to establish Illinois BEI June 2010 - Illinois BEI Rater Training July 2014 - Missouri BEI Rater Training L. Metcalf 2017 Copy only with permission
University of Arizona National Center for Interpretation Dr. Roseann Gonzalez was the key expert in the development of the Federal Court Interpreter Certification Exam This exam has served as a model for high-stakes spoken language interpreting proficiency exams This is the model that the BEI adopted for the Sign Language Interpreting Exam L. Metcalf 2017 Copy only with permission
University of Arizona National Center for Interpretation Their examination process conforms to universally accepted standards of constructing, administering and evaluation of employment tests as prescribed by: American Education Research Association American Psychological Association National Council on Measurement in Education (Standards of Educational and Psychological Testing) L. Metcalf 2017 Copy only with permission
University of Arizona National Center for Interpretation From all the experts we learned how important it was to develop an evaluation that was valid, reliable, and legally defensible. L. Metcalf 2017 Copy only with permission
The purpose of the BEI Assessment is to answer an important question. Does this person have the requisite interpreting skills to carry out the responsibilities of a BEI-certified interpreter at this level? When we certify candidates, we are saying that at this point in time they possess the minimal level of proficiency required at their level of certification not the maximum or even average level of ability when compared to their peers. L. Metcalf 2017 Copy only with permission
Overarching Goals of the BEI Interpreter Certification Process Ensure that consumers have equal access to rights, services, and education. Produce tests that are empirically grounded, fair, reliable, valid, and legally defensible. Identify proficient interpreters at different credential levels. L. Metcalf 2017 Copy only with permission
Developing the Evaluation was a Three- Phase Process 1. Job/Language Analysis to document specifications for construct and content validity 2. Develop Test of Language Proficiency 3. Develop Interpreter Performance Examinations L. Metcalf 2017 Copy only with permission
Survey Structure Questions we wanted answered: Who is currently interpreting? Where are they working? What tasks are they performing and how often? What knowledge, skills, and abilities must interpreters possess to perform their tasks? How might the certification process be improved to ensure that interpreters possess these skills? L. Metcalf 2017 Copy only with permission
Survey Sample Survey was mailed to 588 incumbents. All Level II through V interpreters and 10% of Level I who had been certified over 5 years. Returned surveys evaluated for any bias with no correlation found. Job analysis completed Fall of 2002. L. Metcalf 2017 Copy only with permission
Example of Findings Question: In what settings are interpreters being most used and what level of certification do they hold? Question: What skills do you need in order to be successful at your job? (1,632 pieces of data per questionnaire - 306,816 pieces of data total) L. Metcalf 2017 Copy only with permission
Implications for Test Development Expert Panel opinion and Job Analysis data concurred that a three-level certification system was justified: Basic, Advanced, and Master. Basic: Emphasis on educational settings Advanced: Emphasis on routine medical, routine legal, government, and routine mental health settings. Master: Emphasis on complex medical, complex legal, and complex mental health settings. L. Metcalf 2017 Copy only with permission
Implications for Test Development In addition, Sight Translation is introduced as a new component of the BEI Performance Exam. L. Metcalf 2017 Copy only with permission
Implications for Test Development Based on the University of Arizona s testing model, the job analysis and input from the Expert panel a two- stage process was decided upon: Written Test of English Proficiency (TEP) Interpreting Performance Exam L. Metcalf 2017 Copy only with permission
Ethics and Protocol Testing Knowledge of the Standards of Ethical Behavior and Deaf culture and history are essential to the responsible fulfillment of interpreter duties. It was recommended that these areas should be incorporated into training and continuing education programs, not into the certification exam. L. Metcalf 2017 Copy only with permission
Justification for this decision The type of knowledge tested does not directly relate to the ability to accurately interpret. They reflect a fairly narrow scope of knowledge, are easily learned and memorized, and may skew test results so that deficiencies in other areas, such as English proficiency, are camouflaged. L. Metcalf 2017 Copy only with permission
Test of English Proficiency Screening tool Measure of essential KSATS Allows for the assessment of a broader and more representative sample of a candidates language proficiency The 5 sections of the test were designed to assess candidates proficiency at the lexical, syntactical and discourse levels Test items were written to assess the level of English proficiency at the 11th and 12th grade level as identified in the job/linguistic analysis L. Metcalf 2017 Copy only with permission
Phase 2: Development of TEP Do candidates posses the minimum level of English proficiency required to advance to the performance examination? 16 items in each of 5 subsections 80 items on the final Test of English Proficiency Test items are multiple choice format L. Metcalf 2017 Copy only with permission
Development of Pilot Test A valid written proficiency test requires pilot testing to select items that are at the appropriate level of difficulty and are able to discern between acceptable and unacceptable levels of proficiency. L. Metcalf 2017 Copy only with permission
Development of Pilot Test BEI Expert Panel and UA NCITRP staff collaborated in the development and revision of 161 total initial items for piloting. After statistical analysis, the final 80 items were chosen and divided into five subsections. L. Metcalf 2017 Copy only with permission
Five Subsections of the TEP Reading Comprehension Synonyms Grammar and Usage Sentence Completion Antonyms L. Metcalf 2017 Copy only with permission
Phase 3 Developing the Performance Exams Establishing Exam Format Scripting Process Filming of Performance Exam Scoring Methodology Pilot Testing L. Metcalf 2017 Copy only with permission
Introduction to Interpreter Performance Test Three certification Levels: Basic Advanced Master L. Metcalf 2017 Copy only with permission
Performance Test Format Factors Differentiating the Exam Levels: Complexity of Language Complexity of Topics/Settings Speed of Speaker/Signer L. Metcalf 2017 Copy only with permission
Performance Test Format Factors Differentiating the Exam Levels: Basic ~ 110 wpm (Click to hear a sample.) Advanced ~ 120 wpm Master ~ 130 wpm L. Metcalf 2017 Copy only with permission
Scripting Process Scripting involved attending to a host of issues to ensure that the resulting exams contain a representative sample of the relevant aspects of languages and language abilities required of interpreters. Begins with Job Analysis, Review of Current Exams, and BEI Expert Panel Hundreds of hours and dozens of drafts L. Metcalf 2017 Copy only with permission
Loading the Scripts Register Variation Authentic Language ASL & English Breadth and Depth of Vocabulary Specialized Terminology Grammatical & Linguistic Elements Consistent Density Colloquial & Idiomatic Speech Fingerspelling & Numbers Proper Nouns Affect . . . and so on L. Metcalf 2017 Copy only with permission
Scoring System Purpose: to provide a replicable, fair, and valid device for assessing the interpreting proficiency of candidates for certification. The function of this system is to assess an interpreter s ability to transform meaning from the source language and accurately conveythe same meaningin the target language. L. Metcalf 2017 Copy only with permission
Two-Part System Objective scoring used specifically to determine candidates interpreting proficiency by evaluating underlined scoring units. Subjectivescoringwill supplement this by holistically evaluating candidates Delivery, Adaptability, and Pronunciation/Fluency. L. Metcalf 2017 Copy only with permission
What the Exams Measure Objective Scoring System Interpreting Proficiency The ability to meaningfully and accurately understand, produce, and transform ASL to and from English in a culturally appropriate way. L. Metcalf 2017 Copy only with permission
Objective Assessment The objective assessment of a candidate s level of interpreting proficiency will be determined by how many Objective Scoring Units the candidate renders appropriately. ObjectiveScoring Units are contained in every part of each exam. They represent significant words, phrases, and clauses that are critical to communication. L. Metcalf 2017 Copy only with permission
Examples of Objective Assessment Specialized Terminology Register Variation Affect/ Other Rhetorical features General vocabulary Grammatical structures L. Metcalf 2017 Copy only with permission
Examples of Objective Assessment Idiomatic language Use of classifiers and non-manual markers Accuracy of fingerspelling Use of sign space and grammatical space L. Metcalf 2017 Copy only with permission
Sample Script from the DARS Study Guide L. Metcalf 2017 Copy only with permission
Subjective Scoring System Holistic Scoring Three Dimensions: Delivery is the ability to maintain appropriate delivery, pacing, coherence, and composure consistency throughout the interpretation Adaptability is the level of resourcefulness the candidate display in adapting to changes, patterns, and challenges in the text Subjective Scoring Scale 3 = Exceeds Expectations 2 =Meets Expectations 1 = Fails to Meet Expectations L. Metcalf 2017 Copy only with permission
Pronunciation/Fluency Scored holistically for candidate feedback only Pronunciation is the ability to produce spoken language, including accurate English phonology, and to produce clear signs, as well as the appropriate use of rhythm, stress, and intonation, without interfering with meaning or undermining comprehensibility Fluency is the ease with which a candidate can produce native-like language, including the degree of hesitation and the clarity of signs L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam Expressive Interpreting In this section, candidates will be asked to watch a video recording of spoken English and render it into ASL. L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam Expressive Transliterating In this section, candidates will be asked to watch a video recording of spoken English and render it into signed English. L. Metcalf 2017 Copy only with permission
Comparison of Interpreting vs. Transliterating Click to launch YouTube Remember, grammatical perfection is not the goal Expectation for grammatical mastery increases with the Advanced and Master Evaluations L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam Receptive Only the Basic Level test has a section entitled, Receptive. In this section of the Basic Level test, candidates will be asked to watch a video recording of sign (some combination of ASL and signed English) and render it into spoken English. L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam Receptive Interpreting The Advanced and Master Level tests both include a section entitled, Receptive Interpreting. In this section, candidates will be asked to watch a video recording of a presentation in ASL and render it into spoken English. L. Metcalf 2017 Copy only with permission
Parts of the Performance Exam Receptive Transliterating The Advanced and Master Level tests both include a section entitled, Receptive Transliterating. In this section, candidates will be asked to watch a video recording of a presentation in signed English and render it into spoken English. L. Metcalf 2017Copy only with permission
Parts of the Performance Exam Sight Translation The Sight Translation part of each performance test differs from the other parts of the tests in that it does not include a pre-recorded video stimulus. At the beginning of this section, the candidate will be instructed to sight translate a short, written English document into ASL. L. Metcalf 2017 Copy only with permission
Basic Level Performance Test Content Focuses primarily on the language found in routine educational and social service settings, as determined by the empirical job analysis of incumbent Texas interpreters. Examples of possible topics include: K-12 classroom presentations by students or teachers, special school assemblies, homework assignments, school memos, informational meetings or orientations. L. Metcalf 2017 Copy only with permission
Performance Test Video Instructions INTRODUCTION: This is the Expressive Interpreting part of the exam. This part consists of a pre-recorded simulated lecture. In a moment, you will hear a pre-recorded warm-up that will introduce the lecture. You may interpret it if you wish, but the warm-up will not be scored. During the warm-up you may adjust the volume to your level of comfort. Following the warm-up, you will have one minute to prepare for this part of the examination, after which you will be prompted by the words, Begin InterpretingNow , which mark the beginning of this part of the exam. Any questions? L. Metcalf 2017Copy only with permission
Sight Translation Instructions INTRODUCTION: This is the Sight Translation part of the exam. You have been asked to sight translate a written English document into sign. You will have a total of seven (7) minutes both to prepare and to deliver your translation. You may start your translation when you wish, but if you have not started in two (2) minutes, the technician will instruct you to begin. Each introduction will also contain a sentence describing the setting in which the document is being used, and for whom you are interpreting. For example, the sentence could read, You have been asked to sight translate a PTA flyer for the deaf mother of an elementary school student. L. Metcalf 2017 Copy only with permission
Advanced Level Performance Test Content Focuses primarily on the language found in a variety of high stakes settings, such as routine medical, social services, higher education, routine mental health, and routine quasi-legal, as determined by the empirical job analysis of incumbent interpreters. Examples of possible topics include: post secondary student and professor classroom presentations, professional development seminars, doctor or dentist visits, application for services, employment forms, counseling sessions. L. Metcalf 2017 Copy only with permission
