Role of Research Partnerships in EVEA Project
Role of the Research Partner in
the EVEA Project:
The State of Washington
Alison L. Bailey,
UCLA
CCSSO, NCSA,
Detroit, June 22, 2010
Outline
•
Highlight the role of the research
partnerships
•
Chronology of activities as a Research
Partner (RP) with the State of Washington
•
Example deliverables and products
•
Reflections/lessons learned
EVEA Project Goals
Addressing the validity of
English Language
Proficiency Assessments
(ELPA), research
partners and states work together to:
1.
Build:
- individual State Interpretive Arguments (SIA),
- a Common Interpretive Argument (CIA),
2. Design a set of studies and instruments to support
and test these arguments (pilot level)
3. Make instruments publicly available
Role of Research Partnerships
•
Collate information about the partner
State’s ELPA system
•
Identify and prioritize validity issues; input
from Expert Panel (language and
measurement experts, on-going contact)
•
Create and pilot validation plans
•
Foster collaboration between all RPs on
protocol development, common issues, etc.
WA State Project Goals
•
Create State Interpretative Argument
(SIA) for the validity of the ELPA
•
Identify:
- Claims/assumptions in the ELPA
- Dimension of validity evidence
- Source of evidence
•
Prioritize, design and pilot validation plans
1. Collating Information on WA
State ELD Tests and Standards
(Assessment & Student Information Division;
Migrant & Bilingual Education Program)
Description of the WA State
ELPA System
System Key Components
1.
Identifying population of potential ELL
students with the Home Language Survey:
“Triggered” by affirmative response to
Question#2:
Is your child’s first language
a language other than English?
2. Screening students with WLPT-II Placement
Test
3. Requiring WLPT-II Annual Assessment of all
Transitional Bilingual Instructional Program
(TBIP)-eligible students (EL services)
The Washington Language
Proficiency Test-II (WLPT-II)
•
Augmented Stanford English Language Proficiency
Test (Pearson/Harcourt)
–
additional items aligned with WA ELD Standards
–
4 levels of proficiency: Beginning, Intermediate, Advanced
& Transitional
•
First administered 2005-2006 school year
•
WLPT–II Placement Test
–
Transitional (Level 4) not eligible for TBIP services
•
WLPT–II Annual Assessment
–
Level 1 indicates minimal or no English language proficiency
–
Level 4 indicates a level of English language proficiency
sufficient to be instructed through an English-only
instructional program
Intended Purposes and Uses of
The WLPT-II Annual Assessment
Federal Accountability:
•
Annual Measurable Achievement Objective 1
(progress)
•
Annual Measurable Achievement Objective 2
(proficiency)
–
Criterion for AMAO 2 (achieving proficiency)
»
Transitional (Level 4) on WLPT-II Annual
Assessment
Additional Uses
•
Reclassification to Fluent English Proficient
(R-FEP):
–
Transitional (Level 4) to exit TBIP services
•
Program evaluation instrument
2. Identify and Prioritize ELPA
Validity Issues
(OPSI Staff, Expert Panel, RP & EVEA team)
WLPT-II Validity Plans and
Creation of the WA SIA
E
L
P
A
W
A
S
t
a
t
e
I
n
t
e
r
p
r
e
t
i
v
e
A
r
g
u
m
e
n
t
Foundations Document (Generic)
•
Identifies the external linguistic and
developmental factors by which to judge the
legitimacy of the assumptions expressed in the
existing ELPA system, namely:
–
second language acquisition theories
–
articulation of learning progressions for English language
proficiency (ELP), and
–
the ELP construct adopted (or implied) by the ELD
standards and assessments
•
Raises issues that need to be considered in light of
these assumptions, and
•
Makes suggestions for moving forward.
Foundations Document
(WA Specific)
•
SIA
•
Description of the WLPT-II
•
Description of the WA State ELD Standards
•
Catalogue of technical reports, existing validity studies:
–
Example: Evaluations of the Content of WLPT-II:
»
Studies of alignment with WA ELD Standards (2005-6)
»
Item writers trained to write augmented items aligned
with the test blueprint
»
Existing SELP items modified if necessary
»
Items were sampled in ELL classrooms (directions
clear/items “reliable” indictors of students
achievement)
»
Results from IRT and DIF analyses for different groups
of test-takers (limited to potential gender biases)
E
L
P
A
W
A
S
t
a
t
e
I
n
t
e
r
p
r
e
t
i
v
e
A
r
g
u
m
e
n
t
The ELPA is used to calculate growth in English
Language acquisition over the year to determine
school/program effectiveness
EXAMPLE Organization Chart (EVEA Project)
Purpose 1:
Identifying & Prioritizing Issues
•
Vertical scaling/year-to-year fluctuations:
–
Variation in exit (Transitional Level 4)
percentages year-to-year by grade level and
test form
–
Concerns: used as the TBIP services exit
criterion (R-FEP)
Identifying & Prioritizing Issues
•
Different ELL program eligibility and exit
criteria:
–
Home Language Survey (HLS) and WLPT-II
Placement Test used to qualify new students
for TBIP services
–
WLPT-II Annual Assessment used as program
exit criterion (and as program evaluation
instrument)
–
Concerns: HLS used as the initial identifying
instrument (false positives/negatives)
Ideas for Investigating Fluctuations
•
Study of Test-level factors:
–
Comparable forms: Issues with equating forms?
–
Document and evaluate item selection process
(Content analysis?)
•
Study of External factors:
–
ELL program changes
–
Reduction in funding (less admin. training)
–
Changes in enrollment demographics
»
Language background/new immigrant groups – direct and
mediated by HLS less “accurate” for some groups (over/
under-identified)
Ideas for Improving Initial
Identification of ELL Population
•
Creation of an “Enhanced” HLS:
–
Pilot additional questions based on language
use in specific activity settings (e.g., research-
base showing predictive validity of parent
questionnaires about oral and print home
practices and later oral language and reading
outcomes in English; Reese & Goldenberg,
2008).
•
Possible oral interview protocol for non-
literate parents
–
Responsive to recent Somali refugee
population
Additional Areas of Study
•
The Role of the WLPT-II in Language and
Content-Area Instruction (surfaced by EP):
-
Survey teachers to determine:
-
Whether teachers use WLPT-II results to plan
content-area instruction for students
(adaptation of existing CRESST teacher
survey of science OTL and academic language
exposure), and
-
What level of instruction supports the English
language needed for success on the WLPT-II
Reflections/Lessons Learned
1. Liaising across OSPI Divisions and
Programs:
–
Dialogue across “silos”, often for first time
–
Easier for an outsider?
2. Meeting with Expert Panel & 5 EVEA States
(tailoring SIA surfaced additional issues):
–
What are the boundaries of the Assessment
Division’s responsibility and sphere of influence?
–
Do teachers use ELPA scores? If so, how?
–
Need for the Foundations Document & more time!
Reflections/Lessons Learned
3. RP Monthly Discussions:
–
Address individual state and larger, across-
state issues/RP interests (e.g., wider adoption
of Foundation Document; White paper on HLS
practices: differences and similarities)
–
Accountability (impetus for steady progress)
4. Time(ing) & Attention:
–
Variation in size/capacity of state Assessment
Divisions (recommend staff designated to work
with RP)
–
EVEA versus… RTTT.
Thank You!
Contact Information:
Alison Bailey:
abailey@gseis.ucla.edu
Project Website:
eveaproject.com
Additional Background Slides
Programs
successfully move
more ELL students
into the Proficient
category and exit
them out of the ELP
programs
The ELD standards have
been developed to
support the acquisition
of English language
proficiency necessary to
achieve academic
content and
performance
expectations.
The ELP assessment has been
designed to yield scores that
reflect students’ knowledge and
skills in relation to academic
English language expectations
defined in the ELD standards.
ELP assessment
scores/
performance levels
are used
appropriately to
inform decisions
about progress in
attaining English
language
proficiency
ELP assessment
scores/
performance levels
accurately reflect
students’ English
language
proficiency
The ELP assessment is
administered as
intended
ELL students
become proficient
in English,
acquiring the
academic language
skills necessary to
participate fully in
instructional
discourse
conducted in
English.
E
V
E
A
P
r
o
j
e
c
t
:
E
L
P
A
C
o
m
m
o
n
I
n
t
e
r
p
r
e
t
i
v
e
A
r
g
u
m
e
n
t
(
2
n
d
D
r
a
f
t
)
The EVEA Project in collaboration with research partners aims to address the validity of English Language Proficiency Assessments (ELPA) by building interpretive arguments, designing studies, and making instruments publicly available. Research partners play a crucial role in collating information, identifying validity issues, creating validation plans, and fostering collaboration among all partners.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Role of the Research Partner in the EVEA Project: The State of Washington Alison L. Bailey, UCLA CCSSO, NCSA, Detroit, June 22, 2010 Text-Only Version
Outline Highlight the role of the research partnerships Chronology of activities as a Research Partner (RP) with the State of Washington Example deliverables and products Reflections/lessons learned Text-Only Version
EVEA Project Goals Addressing the validity of English Language Proficiency Assessments (ELPA), research partners and states work together to: 1. Build: - individual State Interpretive Arguments (SIA), - a Common Interpretive Argument (CIA), 2. Design a set of studies and instruments to support and test these arguments (pilot level) 3. Make instruments publicly available Text-Only Version
Role of Research Partnerships Collate information about the partner State s ELPA system Identify and prioritize validity issues; input from Expert Panel (language and measurement experts, on-going contact) Create and pilot validation plans Foster collaboration between all RPs on protocol development, common issues, etc. Text-Only Version
WA State Project Goals Create State Interpretative Argument (SIA) for the validity of the ELPA Identify: - Claims/assumptions in the ELPA - Dimension of validity evidence - Source of evidence Prioritize, design and pilot validation plans Text-Only Version
1. Collating Information on WA State ELD Tests and Standards (Assessment & Student Information Division; Migrant & Bilingual Education Program) Description of the WA State ELPA System Text-Only Version
System Key Components 1. Identifying population of potential ELL students with the Home Language Survey: Triggered by affirmative response to Question#2: Is your child s first language a language other than English? 2. Screening students with WLPT-II Placement Test 3. Requiring WLPT-II Annual Assessment of all Transitional Bilingual Instructional Program (TBIP)-eligible students (EL services) Text-Only Version
The Washington Language Proficiency Test-II (WLPT-II) Augmented Stanford English Language Proficiency Test (Pearson/Harcourt) additional items aligned with WA ELD Standards 4 levels of proficiency: Beginning, Intermediate, Advanced & Transitional First administered 2005-2006 school year WLPT II Placement Test Transitional (Level 4) not eligible for TBIP services WLPT II Annual Assessment Level 1 indicates minimal or no English language proficiency Level 4 indicates a level of English language proficiency sufficient to be instructed through an English-only instructional program Text-Only Version
Intended Purposes and Uses of The WLPT-II Annual Assessment Federal Accountability: Annual Measurable Achievement Objective 1 (progress) Annual Measurable Achievement Objective 2 (proficiency) Criterion for AMAO 2 (achieving proficiency) Transitional (Level 4) on WLPT-II Annual Assessment Text-Only Version
Additional Uses Reclassification to Fluent English Proficient (R-FEP): Transitional (Level 4) to exit TBIP services Program evaluation instrument Text-Only Version
2. Identify and Prioritize ELPA Validity Issues (OPSI Staff, Expert Panel, RP & EVEA team) WLPT-II Validity Plans and Creation of the WA SIA Text-Only Version
ELPA WA State Interpretive Argument External documentation/judgment: For adequacy of SLA theory, articulation of progressions of ELD/P and ELD/P construct definition adopted by ELD standards and assessments AMAO 1: ELPA scores/ performance levels are used appropriately to inform decisions about progress in attaining English language proficiency The ELD standards have been developed to support the acquisition of English language proficiency necessary to achieve academic content and performance expectations. Programs successfully moving more ELL students into the Proficient category and exit them out of the ELP programs ELPA (annual) administered as intended to yield scores for Use and interpretation: Consequences for Student Intake: Identifying the ELL population: HLS > Placement Test ELL students becoming proficient in English, acquiring the academic language skills necessary to participate fully in instructional discourse conducted in English. (Teacher role) (School staff, teacher, TA roles) The ELPA has been designed to yield scores that reflect students knowledge and skills in relation to academic English language expectations defined in the ELD standards. AMAO 2: ELPA scores/ performance levels accurately reflect students English language proficiency (School/teacher role in this process) Text-Only Version
Foundations Document (Generic) Identifies the external linguistic and developmental factors by which to judge the legitimacy of the assumptions expressed in the existing ELPA system, namely: second language acquisition theories articulation of learning progressions for English language proficiency (ELP), and the ELP construct adopted (or implied) by the ELD standards and assessments Raises issues that need to be considered in light of these assumptions, and Makes suggestions for moving forward. Text-Only Version
Foundations Document (WA Specific) SIA Description of the WLPT-II Description of the WA State ELD Standards Catalogue of technical reports, existing validity studies: Example: Evaluations of the Content of WLPT-II: Studies of alignment with WA ELD Standards (2005-6) Item writers trained to write augmented items aligned with the test blueprint Existing SELP items modified if necessary Items were sampled in ELL classrooms (directions clear/items reliable indictors of students achievement) Results from IRT and DIF analyses for different groups of test-takers (limited to potential gender biases) Text-Only Version
ELPA WA State Interpretive Argument External documentation/judgment: For adequacy of SLA theory, articulation of progressions of ELD/P and ELD/P construct definition adopted by ELD standards and assessments AMAO 1: ELPA scores/ performance levels are used appropriately to inform decisions about progress in attaining English language proficiency The ELD standards have been developed to support the acquisition of English language proficiency necessary to achieve academic content and performance expectations. Programs successfully moving more ELL students into the Proficient category and exit them out of the ELP programs ELPA (annual) administered as intended to yield scores for Use and interpretation: Consequences for Student Intake: Identifying the ELL population: HLS > Placement Test ELL students becoming proficient in English, acquiring the academic language skills necessary to participate fully in instructional discourse conducted in English. (Teacher role) (School staff, teacher, TA roles) The ELPA has been designed to yield scores that reflect students knowledge and skills in relation to academic English language expectations defined in the ELD standards. AMAO 2: ELPA scores/ performance levels accurately reflect students English language proficiency (School/teacher role in this process) Text-Only Version
EXAMPLE Organization Chart (EVEA Project) Purpose 1: The ELPA is used to calculate growth in English Language acquisition over the year to determine school/program effectiveness Dimension of Validity Evidence Evidence based on internal structure Claim/Assumption Sources of Evidence The ELPA is administered as intended Study monitoring administration (e.g., two teachers, same student) Cluster analysis by administrator Triangulate with other measures of student ELP Statistical criteria in technical manual ELPA scores accurately reflect student s ELP Scales have been created to accurately reflect growth Growth scores are used appropriately to make judgments about program effectiveness Evidence based on response processes Evidence based on internal structure Evidence based on test consequences Focus group on interpreting data Triangulate with other measures of school effectiveness Text-Only Version
Identifying & Prioritizing Issues Vertical scaling/year-to-year fluctuations: Variation in exit (Transitional Level 4) percentages year-to-year by grade level and test form Concerns: used as the TBIP services exit criterion (R-FEP) Text-Only Version
Identifying & Prioritizing Issues Different ELL program eligibility and exit criteria: Home Language Survey (HLS) and WLPT-II Placement Test used to qualify new students for TBIP services WLPT-II Annual Assessment used as program exit criterion (and as program evaluation instrument) Concerns: HLS used as the initial identifying instrument (false positives/negatives) Text-Only Version
Ideas for Investigating Fluctuations Study of Test-level factors: Comparable forms: Issues with equating forms? Document and evaluate item selection process (Content analysis?) Study of External factors: ELL program changes Reduction in funding (less admin. training) Changes in enrollment demographics Language background/new immigrant groups direct and mediated by HLS less accurate for some groups (over/ under-identified) Text-Only Version
Ideas for Improving Initial Identification of ELL Population Creation of an Enhanced HLS: Pilot additional questions based on language use in specific activity settings (e.g., research- base showing predictive validity of parent questionnaires about oral and print home practices and later oral language and reading outcomes in English; Reese & Goldenberg, 2008). Possible oral interview protocol for non- literate parents Responsive to recent Somali refugee population Text-Only Version
Additional Areas of Study The Role of the WLPT-II in Language and Content-Area Instruction (surfaced by EP): - Survey teachers to determine: - Whether teachers use WLPT-II results to plan content-area instruction for students (adaptation of existing CRESST teacher survey of science OTL and academic language exposure), and - What level of instruction supports the English language needed for success on the WLPT-II Text-Only Version
Reflections/Lessons Learned 1. Liaising across OSPI Divisions and Programs: Dialogue across silos , often for first time Easier for an outsider? 2. Meeting with Expert Panel & 5 EVEA States (tailoring SIA surfaced additional issues): What are the boundaries of the Assessment Division s responsibility and sphere of influence? Do teachers use ELPA scores? If so, how? Need for the Foundations Document & more time! Text-Only Version
Reflections/Lessons Learned 3. RP Monthly Discussions: Address individual state and larger, across- state issues/RP interests (e.g., wider adoption of Foundation Document; White paper on HLS practices: differences and similarities) Accountability (impetus for steady progress) 4. Time(ing) & Attention: Variation in size/capacity of state Assessment Divisions (recommend staff designated to work with RP) EVEA versus RTTT. Text-Only Version
Thank You! Contact Information: Alison Bailey: abailey@gseis.ucla.edu Project Website: eveaproject.com Text-Only Version
Additional Background Slides Text-Only Version
EVEA Project: ELPA Common Interpretive Argument (2nd Draft) The ELD standards have been developed to support the acquisition of English language proficiency necessary to achieve academic content and performance expectations. Students have been appropriatel y identified to participate in the ELPA ELP assessment scores/ performance levels are used appropriately to inform decisions about progress in attaining English language proficiency Programs successfully move more ELL students into the Proficient category and exit them out of the ELP programs Teachers have the knowledge, skills, and orientation necessary to provide instruction in support of academic English language acquisition The ELP assessment has been designed to yield scores that reflect students knowledge and skills in relation to academic English language expectations defined in the ELD standards. ELL students become proficient in English, acquiring the academic language skills necessary to participate fully in instructional discourse conducted in English. ELP assessment scores/ performance levels accurately reflect students English language proficiency The ELP assessment is administered as intended Text-Only Version
