Understanding the Connection Between Economic Disadvantage Classification and Ohio Test Performance

Relationships between Relationships between Economic Disadvantage Classification Economic Disadvantage Classification and and Ohio Test Performance Ohio Test Performance Terrence, 27 July 2017

Economic Disadvantage Classification Economic Disadvantage Classification (From ODE website) (From ODE website) Meet any of the following (there are 4 ways): 1. A student with an approved application on file for free or reduced- price meals

Economic Disadvantage Classification Economic Disadvantage Classification Meet any of the following: 1. A student with an approved application on file for a free or reduced-price lunch 2. Students who reside in a household in which a member (e.g., sibling) is known to be eligible for free or reduced-price lunch via an approved application or through direct certification.

Economic Disadvantage Classification Economic Disadvantage Classification Meet any of the following: 1. A student with an approved application on file for a free or reduced-price lunch 2. Students who reside in a household in which a member (e.g., sibling) is known to be eligible for free or reduced-price lunch via an approved application or through direct certification. 3. Students who are known to be recipients of or whose guardians are known to be recipients of public assistance. E.g.: Education Monetary Assistance Distribution (EMAD) system.

Economic Disadvantage Classification Economic Disadvantage Classification Meet any of the following: 1. A student with an approved application on file for a free or reduced- price lunch 2. Students who reside in a household in which a member (e.g., sibling) is known to be eligible for free or reduced-price lunch via an approved application or through direct certification. 3. Students who are known to be recipients of or whose guardians are known to be recipients of public assistance. E.g.: Education Monetary Assistance Distribution (EMAD) system. 4. Students [with] a Title I student income form and meet the income guidelines at or below 130% of the federal poverty level = free lunch at or below 185% of the federal poverty level = reduced price lunch Federal poverty level updated annually by U.S.D.A. and the ODE

Data source for performance data Data source for performance data ODE EMIS records via a public web site Aggregate data at the state level Data limited to counts of students at performance levels Only 5 performance levels Not as good as scale scores

Plan of presentation Plan of presentation Look at 3rdgrade data Learn about 3rdgrade students Learn about principles that may be applicable to other grades and/or cohorts Look at data for other years and cohorts Criteria for treatments Criteria for project outcomes

Choosing measures Choosing measures Index counts at all performance measures into a single metric abandoned Chose to focus on counts and percents of students classed as proficient and above Have the data Sufficient to capture many students (somewhat immune to outliers) Easy to interpret

Grade 3 Grade 3 Reading Reading percent Proficient or percent Proficient or Above Above 100% 80% 60% 40% 20% 0% 2005 2007 2009 2011 2013 2015 Economic disadvantage Not classed as ED All

Grade 3 Reading percent Proficient or Grade 3 Reading percent Proficient or Above Above 100% 80% 60% 40% 20% 0% 2005 2007 Economic disadvantage 2009 2011 2013 2015 Not classed as ED

Difference in Percent Difference in Percent Proficient or Above Not-ED minus ED Proficient or Above Estimate of the size of gap in G3 Reading 30% 25% 20% 15% 10% 5% 0% 2005 2007 2009 2011 2013 2015

Classification of third graders by year 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 2005 2007 Economically disadvataged 2009 2011 2013 2015 2017 Not-ED

Increase in the probability that a third grader is classified as ED 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 -0.05

Classification of the 2006 cohort of third graders 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 2 3 4 5 6 7 8 9 10 11 Grade Economically disadvantage Not ED

Classification of third graders by year 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 2005 2007 Economically disadvataged 2009 2011 2013 2015 2017 Not-ED

Cohort of 2006 third graders, Percent proficient or above 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 3 4 5 6 7 8 9 10 Grade Economically Disadvantaged Not ED

Looks like the gap goes down at higher grades 30% 25% 20% Gap 15% 10% 5% 0% 3 4 5 6 7 8 9 10 Grade

but Gap goes down with difficulty 30% 25% 20% GAP 15% y = -0.9552x + 1.0517 R = 0.9164 10% 5% 0% 82% 84% 86% 88% 90% 92% 94% 96% Percent Not-ED found Proficinet or Above

Other grades in Reading and Math? Look much like Grade 3 Reading including the characterization for FRPM

Meal subsidy eligibility, CPI, and wage data (raw data) FRPM criterion (Family of 4) CPI Average Ohio annual wage Ohio Year (urban, midwest) annual payroll 192.4 196.629 204.763 202.527 207.356 213.47 218.294 221.96 225.265 223.645 225.173 $ 38,568 39,917 40,784 40,900 41,788 42,972 44,244 44,671 46,000 47,146 47,586 $ 205 B 212 B 214 B 202 B 205 B 214 B 223 B 228 B 238 B 248 B 253 B 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 $ 37,000 38,203 39,220 40,793 40,793 41,348 42,653 43,568 44,123 44,863

Meal subsidy eligibility, CPI, and wage data (indexed to the base year of 2007 for clarity) FRPM criterion (Family of 4) CPI Average Ohio annual wage Ohio Year (urban, midwest) annual payroll 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 0.98 1.00 1.04 1.03 1.05 1.09 1.11 1.13 1.15 1.14 1.15 0.97 1.00 1.02 1.02 1.05 1.08 1.11 1.12 1.15 1.18 1.19 0.97 1.00 1.01 0.95 0.97 1.01 1.05 1.08 1.13 1.17 1.20 1.00 1.03 1.06 1.10 1.10 1.12 1.15 1.18 1.19 1.21

What the data show For the third grade reading data, it seems that 1. There are far more students being qualified for FRPM than ten years ago. Based on the high and increasing subscription rate to FRPMs, it is difficult to think that FRPM status is useful in determining to whom a special educational treatment is to be conferred on the basis of economic disadvantage. 2. Students under the FRPM plan do not perform on the third grade reading test (and other tests) as well as those not qualified under the FRPM plan. It is better to think of this as an association than as a cause; e.g. there may be some other factor or factors that are causing both FRPM participation and a performance differential.

Educational treatments What makes a good treatment? 1. Evidence-based (per ESEA, amended PL 114-95; SEC 8601 20 U.S.C. 7801 (21)(A), page 393) demonstrates a statistically significant effect on improving student outcomes or other relevant outcomes based on (I) strong evidence experimental study; (II) moderate evidence quasi-experimental study; (III)promising evidence correlational study with statistical controls for selection bias; Concept illustrated by extracts from ESSA in the folders

Educational treatments What makes a good treatment? 1. Evidence based 2. Reproducibility at scale & Generalizable A. Evidence that teachers can be taught to administer the treatment to consistently get the desired result? B. Evidence that the desired results can be obtained from most of the population or evidence that defines the sub-population where this treatment works?

Educational treatments What makes a good treatment? 1. Evidence based 2. Reproducibility at scale & Generalizable 3. Affordable and efficient What does the treatment cost to train? To administer? Are the effect sizes sufficient to justify the cost? Is there another treatment that produces more bang for our bucks? Is there another treatment that takes fewer hours or days to produce results?

Educational diversion threat Doing something that does not work very well but does interfere with applying something that might work much better

Commonly espoused opportunities for treatments Class size reduction Improving teacher quality & Quality of instruction & Technology Early Childhood Education Food, clothing, shelter Discipline/student management/school climate Health services: physical, mental/reducing psycho-stressors Expanded instructional time/year-round schooling Stimulating striving & self-control/self-determination and others

Interactions When considering the treatments on the previous slide Choosing the Good, Better, or Best treatment may vary depending on: The student situation & deficiency The student population The personnel available Financial resources

end