Evaluating Bias in Value-Added Models Using Prior Scores

 
 
Using Prior Scores to Evaluate Bias in Value-Added Models
 
 
 
 
 
Raj Chetty, Stanford University and NBER
John N. Friedman, Brown University and NBER
Jonah Rockoff, Columbia University and NBER
 
 
January 2016
 
 
 
Outcome-based value added (VA) models increasingly used to
measure the productivity of many agents
 
Teachers, schools, neighborhoods, doctors, CEOs…
 
 
Central question in determining whether VA measures are useful for
policy: to what extent are VA estimates biased by selection?
[e.g., Rothstein 2009; Kane and Staiger 2008; Chetty, Friedman, Rockoff 2014]
 
Ex: do differences in latent abilities of students assigned to
teachers bias estimates of teacher VA?
 
 
Introduction: Bias in Value-Added Models
 
 
One intuitive approach to assessing degree of bias in VA models: test
for balance in lagged values of the outcome
 
Simple to implement: regress prior scores on current teacher VA
[Chetty, Friedman, Rockoff 2014; Rothstein 2015]
 
Intuition: current teachers cannot have causal effects on prior
scores
 
Analogous to standard analysis of pre-trends in outcomes used
to evaluate bias in program evaluation literature
 
 
Evaluating Bias Using Lagged Outcomes
 
 
We show that balance tests using lagged values of the outcome are
sensitive to model specification in VA models
 
Prior scores will typically be correlated with VA estimates even
when VA estimates are unbiased
 
More generally, tests using prior scores are uninformative about
degree of forecast bias when VA model is misspecified
 
Intuition: Correlated shocks 
enter 
both
 current VA estimate and
lagged outcome in ways that are sensitive to model specification
 
 
Overview
 
 
Why are lagged outcome tests of balance more robust in conventional
treatment effect settings (e.g., class size)?
 
Two key differences in VA models:
 
1.
Treatment itself is estimated, rather than exogenously observed
 
2.
Estimation error does not vanish in large datasets because sample
size per teacher remains small asymptotically
 
 
With exogenous treatments, noise in lagged outcomes uncorrelated with
treatment and estimation error vanishes asymptotically
 
 
Experimental/quasi-experimental methods provide a less model-
dependent way to assess bias in VA models
 
 
Overview
 
 
1.
Specification of Value-Added Model
 
 
2.
Monte Carlo Simulation Results
 
 
3.
Other Approaches to Evaluating Bias
 
 
Outline
 
 
Model Setup: Students and Teachers
 
 
Model Setup: Tracks and Correlated Shocks
 
 
Data Generating Process for Test Scores
 
fixed
ability
 
ability
trend
 
teacher
effect
 
class
shock
 
track
shock
 
student
shock
 
 
Estimator for Value-Added
 
 
Forecast Bias: Definition
 
 
Estimating Forecast Bias
 
 
B
a
s
e
l
i
n
e
 
P
a
r
a
m
e
t
e
r
s
 
f
o
r
 
M
o
n
t
e
 
C
a
r
l
o
 
S
i
m
u
l
a
t
i
o
n
s
Governing Student, Classroom, Year, and Track Effects
 
 
Begin by considering case with only static tracking, so there is no
bias in VA estimates
 
 
First examine relationship between test score gains under random
assignment and VA estimates based on observational data
 
As expected, prediction coefficient is 1 in this experiment (no
forecast bias)
 
 
Simulation Results
 
T
e
s
t
 
S
c
o
r
e
 
G
a
i
n
s
 
U
n
d
e
r
 
R
a
n
d
o
m
 
A
s
s
i
g
n
m
e
n
t
 
v
s
.
 
V
A
 
E
s
t
i
m
a
t
e
s
 
 
Correlation with Prior Scores
 
L
a
g
g
e
d
 
T
e
s
t
 
S
c
o
r
e
 
G
a
i
n
s
 
v
s
.
 
C
u
r
r
e
n
t
 
V
A
 
E
s
t
i
m
a
t
e
s
 
 
 
 
 
 
Slope: b = 0.709
 
(0.013)
 
-.1
 
0
 
.1
 
Lagged Test Score Gain
 
-.2
 
-.1
 
0
 
.1
 
.2
 
Value-Added Estimate
 
 
 
Correlation with Prior Scores
 
 
Correlation with Prior Scores
 
L
a
g
g
e
d
 
T
e
s
t
 
S
c
o
r
e
 
G
a
i
n
s
 
v
s
.
 
T
r
u
e
 
T
e
a
c
h
e
r
 
V
A
 
 
 
 
 
Common variants of lagged outcome balance test suffer from the same
problem
 
For instance, testing whether controlling for lagged gain affects
forecast coefficient on VA estimate
 
 
Variants of Lagged Outcome Balance Test
 
 
We have focused thus far on forecast bias (average prediction)
[Kane and Staiger 2008, Chetty, Friedman, Rockoff 2014]
 
 
Alternative, more stringent measure: teacher-level bias
[Rothstein 2010]
 
Is there excess variance across teachers in lagged gains?
 
Typically implemented using an F test in a regression of lagged
gains on teacher fixed effects
 
 
Analysis of Variance: Teacher-Level Bias
 
N
o
t
e
s
:
 
S
t
a
n
d
a
r
d
 
e
r
r
o
r
s
 
c
l
u
s
t
e
r
e
d
 
b
y
 
t
r
a
c
k
.
 
E
f
f
e
c
t
s
 
o
f
 
T
e
a
c
h
e
r
 
V
A
 
o
n
 
C
u
r
r
e
n
t
 
a
n
d
 
L
a
g
g
e
d
 
T
e
s
t
 
S
c
o
r
e
 
G
a
i
n
s
Results from Monte Carlo Simulations
 
 
F test rejects despite fact that lagged gains are not grouped by teacher
because variance structure is incorrectly specified
 
Does not account for correlated errors within tracks
 
Accounting for this error structure would fix the problem, but again
illustrates sensitivity of test to model specification
 
Specification matters more in VA models because estimation error
does not vanish in large samples
 
In conventional treatment effect settings, 
misspecification of error
structure does not matter for inference in large datasets
 
Sample size per treatment group grows asymptotically
 
In VA models, misspecification matters even in large samples
because sample size per teacher does not grow asymptotically
 
 
Analysis of Variance: Teacher-Level Bias
 
 
 
Now turn to case where VA estimates are in fact biased due to sorting
on gains
 
 
In model without track shocks, straightforward to show that coefficient
from regression of lagged gains on VA exactly matches forecast bias
 
 
No longer true once we allow for correlated shocks within tracks
 
 
Sorting on Gains
 
E
s
t
i
m
a
t
e
s
 
o
f
 
B
i
a
s
 
w
i
t
h
 
S
o
r
t
i
n
g
Baseline Case: Common Track-Year Shocks Across Grades
 
Lagged Gain Bias Est.
 
Actual Forecast Bias
 
 
 
Results above consider naïve implementation of lagged outcome test
that does not respect error structure used to estimate VA model
 
Unfair comparison: information used to estimate VA model not
used when implementing lagged score test
 
 
Potential solution: adjust lagged outcome test to account for
mechanical correlation due to common track shocks
 
Subtract out variance due to common track shocks to form an
adjusted estimate
 
Resolves problem when VA model is correctly specified
 
 
Model-Based Correction to Lagged Outcome Test
 
E
s
t
i
m
a
t
e
s
 
o
f
 
B
i
a
s
 
w
i
t
h
 
S
o
r
t
i
n
g
Baseline Case: Common Track-Year Shocks Across Grades
 
Lagged Gain Bias Est.
 
Adjusted Lagged Gain Bias Est.
 
Actual Forecast Bias
 
 
Deeper problem: such parametric corrections rely heavily on model
specification
 
 
More plausible case: model used to estimate VA itself mis-specified
 
 
For example, suppose track-year shocks are in fact not perfectly
correlated across grades
 
But econometrician continues to assume they are 
both
 when
estimating VA and when implementing lagged outcome test
 
Now parametric correction to lagged outcome test under assumed
model no longer works
 
 
Bias Tests Under Misspecification
 
Lagged Gain Bias Est.
 
Adjusted Lagged Gain Bias Est.
 
Actual Forecast Bias
 
 
Another potential correction: use leave three years out when
estimating VA
 
With iid track shocks, eliminates link between lagged gains and
current VA estimates
 
But this method fails if track shocks are serially correlated
 
 
Bias Tests Under Misspecification
 
 
 
General lesson: results of lagged outcome tests in VA models are
sensitive to model specification
 
Given a VA model, one can always devise a test using lagged
outcomes that will yield consistent estimates of bias
 
But proper specification of test depends heavily on model
 
Of course, misspecification will create biased VA estimates too
 
Key point: lagged outcome test does not provide a robust guide to
the degree of bias is such situations
 
 
Sensitivity of Lagged Outcome Balance Tests
 
 
Given sensitivity of lagged outcome tests to model specification, what
alternative methods can be used to assess bias in VA models?
 
 
Conceptually, need methods that use data guaranteed to be
unrelated to estimation error in VA
 
 
Two existing approaches
 
 
Other Approaches to Evaluating Bias
 
 
1.
Use pre-determined, exogenous covariates (e.g., race or parental
income) to evaluate balance
 
Advantage: Noise in outcomes does not directly enter such
variables, making such tests less fragile
 
Drawback: does not necessary account for dynamic selection
effects
 
 
Other Approaches to Evaluating Bias
 
 
2.
Out-of-sample experiments/quasi-experiments
[Kane and Staiger 2008]
 
Randomly assign new students to teachers and test whether
prediction coefficient on VA estimates is 1
 
More difficult to implement than tests for balance and typically
yields less precise estimates
 
But several studies have now estimated forecast bias in VA
models in education using this approach
 
Glazerman and Protnik (2014) present a summary of estimates
for teacher VA models
 
 
Other Approaches to Evaluating Bias
 
E
x
p
e
r
i
m
e
n
t
a
l
/
Q
u
a
s
i
-
E
x
p
e
r
i
m
e
n
t
a
l
 
E
s
t
i
m
a
t
e
s
 
o
f
 
F
o
r
e
c
a
s
t
 
B
i
a
s
 
i
n
 
T
e
a
c
h
e
r
 
V
A
 
S
o
u
r
c
e
:
 
G
l
a
z
e
r
m
a
n
 
a
n
d
 
P
r
o
t
i
k
 
 
(
2
0
1
4
)
.
 
V
a
l
i
d
a
t
i
n
g
 
V
a
l
u
e
-
a
d
d
e
d
 
M
e
a
s
u
r
e
s
 
o
f
 
T
e
a
c
h
e
r
 
P
e
r
f
o
r
m
a
n
c
e
 
Note: error bars represent 90% confidence intervals
 
S
o
u
r
c
e
s
:
 
B
i
f
u
c
l
o
,
 
C
o
b
b
,
 
a
n
d
 
B
e
l
l
 
(
2
0
0
9
,
 
T
a
b
l
e
 
6
,
 
C
o
l
s
 
1
 
a
n
d
 
3
)
;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
D
e
m
i
n
g
 
(
2
0
1
4
,
 
T
a
b
l
e
 
1
,
 
C
o
l
 
6
)
;
 
 
A
n
g
r
i
s
t
 
e
t
 
a
l
.
 
(
2
0
1
5
,
 
T
a
b
l
e
 
3
,
 
C
o
l
 
3
)
 
E
x
p
e
r
i
m
e
n
t
a
l
/
Q
u
a
s
i
-
E
x
p
e
r
i
m
e
n
t
a
l
 
E
s
t
i
m
a
t
e
s
 
o
f
 
F
o
r
e
c
a
s
t
 
B
i
a
s
 
i
n
 
S
c
h
o
o
l
 
V
A
 
Note: error bars represent 90% confidence intervals
 
0
.
9
7
 
 
 
Estimation of VA creates a complex error structure for the treatment
that is correlated with prior outcomes in non-transparent ways
 
Makes tests for bias using lagged outcomes more sensitive to
model specification than when treatment is directly observed
 
 
Experimental/quasi-experimental methods provide an approach to
assessing bias that is less sensitive to model specification
 
 
Potential directions for future work:
 
Compare alternative VA estimators when model is misspecified
 
In addition to measuring bias, gauge welfare gain from using
biased estimates 
[e.g., Angrist, Hull, Pathak, Walters 2015]
 
 
Conclusion
Slide Note
Embed
Share

Outcome-based value-added (VA) models are commonly used to assess productivity in various fields. This study explores the use of prior scores to evaluate bias in VA estimates, focusing on the correlation between current teacher VA and lagged outcomes. The analysis highlights the sensitivity of balance tests to model specification and the limitations of using prior scores in determining forecast bias in VA models.


Uploaded on Sep 02, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Using Prior Scores to Evaluate Bias in Value-Added Models Raj Chetty, Stanford University and NBER John N. Friedman, Brown University and NBER Jonah Rockoff, Columbia University and NBER January 2016

  2. Introduction: Bias in Value-Added Models Outcome-based value added (VA) models increasingly used to measure the productivity of many agents Teachers, schools, neighborhoods, doctors, CEOs Central question in determining whether VA measures are useful for policy: to what extent are VA estimates biased by selection? [e.g., Rothstein 2009; Kane and Staiger 2008; Chetty, Friedman, Rockoff 2014] Ex: do differences in latent abilities of students assigned to teachers bias estimates of teacher VA?

  3. Evaluating Bias Using Lagged Outcomes One intuitive approach to assessing degree of bias in VA models: test for balance in lagged values of the outcome Simple to implement: regress prior scores on current teacher VA [Chetty, Friedman, Rockoff 2014; Rothstein 2015] Intuition: current teachers cannot have causal effects on prior scores Analogous to standard analysis of pre-trends in outcomes used to evaluate bias in program evaluation literature

  4. Overview We show that balance tests using lagged values of the outcome are sensitive to model specification in VA models Prior scores will typically be correlated with VA estimates even when VA estimates are unbiased More generally, tests using prior scores are uninformative about degree of forecast bias when VA model is misspecified Intuition: Correlated shocks enter both current VA estimate and lagged outcome in ways that are sensitive to model specification

  5. Overview Why are lagged outcome tests of balance more robust in conventional treatment effect settings (e.g., class size)? Two key differences in VA models: Treatment itself is estimated, rather than exogenously observed 1. Estimation error does not vanish in large datasets because sample size per teacher remains small asymptotically 2. With exogenous treatments, noise in lagged outcomes uncorrelated with treatment and estimation error vanishes asymptotically Experimental/quasi-experimental methods provide a less model- dependent way to assess bias in VA models

  6. Outline 1. Specification of Value-Added Model 2. Monte Carlo Simulation Results 3. Other Approaches to Evaluating Bias

  7. Model Setup: Students and Teachers We consider estimation of teacher effects for concreteness, but results translate directly to other applications Data on students test scores and classroom assignments in years t = 1, 2 used to predict teacher quality in years t > 2 Student i is assigned in year ? to classroom ? ?,? and teacher ?(? ?,? ) = ?(?,?) Each teacher j teaches C classrooms per year in a single grade Each classroom c has I students

  8. Model Setup: Tracks and Correlated Shocks Key new element used to assess sensitivity to model specification: classrooms grouped into tracks (?) Ex: regular vs. honors classes Students and teachers assigned to a given track ?(?) in all years Classroom shocks within tracks are correlated, both within and across grades For instance, curriculum in a given track may line up particularly well with tests in certain years

  9. Data Generating Process for Test Scores Student s test score in year t is given by ???= ??+ ??? + ??(?,?)+ ?? ?,? ,?+ ?? ?,? ,?+ ??? fixed ability ability trend teacher effect class shock track shock student shock Assume that teacher value-added (??) does not vary over time Student assignment to teachers may be correlated with ability [Rothstein 2010] Static tracking: ?? correlated with ?? (fixed ability) Dynamic tracking: ?? correlated with ?? (ability trends)

  10. Estimator for Value-Added Teacher VA estimated using a standard gains specification Average change in students end-of-year test scores, adjusting for noise using a standard shrinkage factor Let ???= ??? ??,? 1 denote student ?'s test score gain in year ? Estimator for VA of teacher j using test score data from years 1 and 2: ??= ? ??,?=2 2 ?? 2/?+??? where ? = 2+?? 2/?? 2+??? ?? This estimator minimizes MSE of out-of-sample forecasts of test scores and is posterior expectation of VA with Normal distributions

  11. Forecast Bias: Definition Consider running an experiment where students are randomly assigned to teachers and estimating the regression: ???= ? + ? ??(?,?)+ ??? Prediction coefficient in this regression identifies degree of forecast bias 1 b [Kane and Staiger 2008; Chetty, Friedman, Rockoff 2014] If VA estimates are forecast unbiased (? = 1) , assigning a student to a teacher with one unit higher estimated VA will increase his score by one unit

  12. Estimating Forecast Bias Gains model yields forecast unbiased estimates when there is static tracking (sorting on ??) but not with dynamic tracking (sorting on ??) How can we distinguish these two cases and, more generally, estimate degree of forecast bias? Is correlation of VA estimates with prior scores informative? Use a set of Monte Carlo simulations to answer this question

  13. Baseline Parameters for Monte Carlo Simulations Governing Student, Classroom, Year, and Track Effects Parameter Value Number of Schools Number of Tracks per School Number of Teachers per Track Number of Classrooms per Teacher (?) Number of Students per Classroom (?) 2000 5 4 4 25 SD of Student Ability (??) SD Of Trend Differences Across Students (??) SD Of Teacher Value-Added (??) SD of Classroom Shocks (??) SD of Track-Year Shock (??) 0.88 0.15 0.10 0.08 0.06 Degree of Sorting (Level) Degree of Sorting (Trend) 0.25 0.00

  14. Simulation Results Begin by considering case with only static tracking, so there is no bias in VA estimates First examine relationship between test score gains under random assignment and VA estimates based on observational data As expected, prediction coefficient is 1 in this experiment (no forecast bias)

  15. Test Score Gains Under Random Assignment vs. VA Estimates .1 Test Score Gain 0 Slope: b = 1.010 (0.007) -.1 -.2 -.1 0 .1 .2 Value-Added Estimate

  16. Correlation with Prior Scores Now regress lagged gains ??,? 1on current teacher s VA estimate ??,? 1= ? + ? ??(?,?)+ ???

  17. Lagged Test Score Gains vs. Current VA Estimates .1 Lagged Test Score Gain 0 Slope: b = 0.709 (0.013) -.1 -.2 -.1 0 .1 .2 Value-Added Estimate

  18. Correlation with Prior Scores Why does current teacher s VA predict lagged test score gain even though there is no bias in this model? Track-specific shock ??? enters both VA estimate and lagged gains because ??? affects students in all grades in a given track Ex.: Suppose VA estimated for 6th grade from 1995 gains Positive track shock in 1995 artificially increases gains, VA estimates Lagged gains for 6th graders in 1996 also affected by the same track shock Therefore VA estimates and lagged gains are correlated

  19. Correlation with Prior Scores More generally, relationship between current VA and lagged gains is governed by variance of track-specific shocks: 2> 0 ??? ?? ?,?, ??,? 1 = ?? ? In a model with no track shocks, lagged outcome balance test correctly diagnoses bias Root of problem: estimation error in VA If one observed true VA directly (or is studying an exogenous treatment like class size), no correlation with lagged gains

  20. Lagged Test Score Gains vs. True Teacher VA .1 Lagged Test Score Gain 0 Slope: b = 0.002 (0.004) -.1 -.2 -.1 0 .1 .2 True Value Added

  21. Variants of Lagged Outcome Balance Test Common variants of lagged outcome balance test suffer from the same problem For instance, testing whether controlling for lagged gain affects forecast coefficient on VA estimate

  22. Analysis of Variance: Teacher-Level Bias We have focused thus far on forecast bias (average prediction) [Kane and Staiger 2008, Chetty, Friedman, Rockoff 2014] Alternative, more stringent measure: teacher-level bias [Rothstein 2010] Is there excess variance across teachers in lagged gains? Typically implemented using an F test in a regression of lagged gains on teacher fixed effects

  23. Effects of Teacher VA on Current and Lagged Test Score Gains Results from Monte Carlo Simulations Randomized experiment Lagged scores Lagged scores versus true VA Observational out-of-sample forecast (4) Observational, controlling for lagged gain (5) (1) (2) (3) Dependent Variable: Current gain Lagged gain Lagged gain Current gain Current gain VA estimate 1.010 (0.007) 0.709 (0.013) 0.991 (0.017) 0.833 (0.016) True VA 0.002 (0.004) Control for lagged gain X Na ve F-test for teacher effects F = 2.238 p<0.001 Notes: Standard errors clustered by track.

  24. Analysis of Variance: Teacher-Level Bias F test rejects despite fact that lagged gains are not grouped by teacher because variance structure is incorrectly specified Does not account for correlated errors within tracks Accounting for this error structure would fix the problem, but again illustrates sensitivity of test to model specification Specification matters more in VA models because estimation error does not vanish in large samples In conventional treatment effect settings, misspecification of error structure does not matter for inference in large datasets Sample size per treatment group grows asymptotically In VA models, misspecification matters even in large samples because sample size per teacher does not grow asymptotically

  25. Sorting on Gains Now turn to case where VA estimates are in fact biased due to sorting on gains In model without track shocks, straightforward to show that coefficient from regression of lagged gains on VA exactly matches forecast bias No longer true once we allow for correlated shocks within tracks

  26. Estimates of Bias with Sorting Baseline Case: Common Track-Year Shocks Across Grades .8 .6 Estimates of Bias .4 .2 0 -.2 0 .05 .1 .15 .2 .25 Degree of Sorting on Gains Actual Forecast Bias Lagged Gain Bias Est.

  27. Model-Based Correction to Lagged Outcome Test Results above consider na ve implementation of lagged outcome test that does not respect error structure used to estimate VA model Unfair comparison: information used to estimate VA model not used when implementing lagged score test Potential solution: adjust lagged outcome test to account for mechanical correlation due to common track shocks Subtract out variance due to common track shocks to form an adjusted estimate Resolves problem when VA model is correctly specified

  28. Estimates of Bias with Sorting Baseline Case: Common Track-Year Shocks Across Grades .8 .6 Estimates of Bias .4 .2 0 -.2 0 .05 .1 .15 .2 .25 Degree of Sorting on Gains Lagged Gain Bias Est. Actual Forecast Bias Adjusted Lagged Gain Bias Est.

  29. Bias Tests Under Misspecification Deeper problem: such parametric corrections rely heavily on model specification More plausible case: model used to estimate VA itself mis-specified For example, suppose track-year shocks are in fact not perfectly correlated across grades But econometrician continues to assume they are both when estimating VA and when implementing lagged outcome test Now parametric correction to lagged outcome test under assumed model no longer works

  30. Estimates of Bias with Sorting and Mis-Specification of VA Model Imperfectly Correlated Track-Year Shocks Across Grades (? = 0.67) .8 .6 Estimates of Bias .4 .2 0 -.2 0 .05 .1 .15 .2 .25 Degree of Sorting on Gains Lagged Gain Bias Est. Actual Forecast Bias Adjusted Lagged Gain Bias Est.

  31. Bias Tests Under Misspecification Another potential correction: use leave three years out when estimating VA With iid track shocks, eliminates link between lagged gains and current VA estimates But this method fails if track shocks are serially correlated

  32. Sensitivity of Lagged Outcome Balance Tests General lesson: results of lagged outcome tests in VA models are sensitive to model specification Given a VA model, one can always devise a test using lagged outcomes that will yield consistent estimates of bias But proper specification of test depends heavily on model Of course, misspecification will create biased VA estimates too Key point: lagged outcome test does not provide a robust guide to the degree of bias is such situations

  33. Other Approaches to Evaluating Bias Given sensitivity of lagged outcome tests to model specification, what alternative methods can be used to assess bias in VA models? Conceptually, need methods that use data guaranteed to be unrelated to estimation error in VA Two existing approaches

  34. Other Approaches to Evaluating Bias Use pre-determined, exogenous covariates (e.g., race or parental income) to evaluate balance 1. Advantage: Noise in outcomes does not directly enter such variables, making such tests less fragile Drawback: does not necessary account for dynamic selection effects

  35. Other Approaches to Evaluating Bias Out-of-sample experiments/quasi-experiments [Kane and Staiger 2008] 2. Randomly assign new students to teachers and test whether prediction coefficient on VA estimates is 1 More difficult to implement than tests for balance and typically yields less precise estimates But several studies have now estimated forecast bias in VA models in education using this approach Glazerman and Protnik (2014) present a summary of estimates for teacher VA models

  36. Experimental/Quasi-Experimental Estimates of Forecast Bias in Teacher VA Note: error bars represent 90% confidence intervals Source: Glazerman and Protik (2014). Validating Value-added Measures of Teacher Performance

  37. Experimental/Quasi-Experimental Estimates of Forecast Bias in School VA 1.20 0.96 0.97 1.00 0.86 0.82 0.80 0.60 0.40 0.20 0.00 Deming (2014) Bifulco et al. (2009) Angrist et al. (2015) Note: error bars represent 90% confidence intervals Sources: Bifuclo, Cobb, and Bell (2009, Table 6, Cols 1 and 3); Deming (2014, Table 1, Col 6); Angrist et al. (2015, Table 3, Col 3)

  38. Conclusion Estimation of VA creates a complex error structure for the treatment that is correlated with prior outcomes in non-transparent ways Makes tests for bias using lagged outcomes more sensitive to model specification than when treatment is directly observed Experimental/quasi-experimental methods provide an approach to assessing bias that is less sensitive to model specification Potential directions for future work: Compare alternative VA estimators when model is misspecified In addition to measuring bias, gauge welfare gain from using biased estimates [e.g., Angrist, Hull, Pathak, Walters 2015]

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#