Effective Assessment Reporting and English Learner Support Training 2022

Assessment Reporting (Editing and Checking Assessment Data, Third Period Cumulative ADM, and English Learners) May 12, 2022 Video Training 2021-2022 (to participate, get handouts, and view recorded webinars) Welcome! Oregon Department of Education 1

Contacts Regional ESD Partners Accountability Reporting Team Jon Wiens, Manager 503-947-5764 Cindy Barrick, Research Analyst 503-947-5822 Bianca Llamas, Research Analyst Elyse Bean, Research Analyst 503-947-5831 Oregon Department of Education 2

Todays Agenda 1 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 3

Assessments and Accountability 1 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 4

ESSA and School Profiles Every Student Succeeds Act (ESSA) ESSA maintains annual testing in English language arts and mathematics (grades 3-8 and 11), and science (grades 5, 8, and 11). ESSA maintains report cards and requires the inclusion of English Learner progress and at least one measure of school quality/student success as indicators to display on report cards. ESSA requires that the denominator for the achievement calculations includes at least 95% of students enrolled at the school, or the number of students participating, whichever is higher. Oregon Department of Education 5

School and District At-A-Glance Profiles Displays: Information on students and staff Class Size and Regular Attenders Academic Progress Statewide assessments Ninth Grade On-Track and Graduation Local information on school programs, parent/community engagement TENTATIVE availability date: July 28, 2022 (subject to change) More details in the webinar in July Oregon Department of Education 6

Accountability Details Elementary & Middle Indicator Levels for all students and 12 student groups ELA Achievement, Growth, and Participation Math Achievement, Growth, and Participation Regular Attenders On Track to English Language Proficiency (ELP) TENTATIVE availability date: July 21, 2022 (subject to change) More details in the June webinar Oregon Department of Education 7

Accountability Details High School Indicator Levels for all students and 12 student groups ELA Achievement and Participation Math Achievement and Participation Regular Attenders On Track to English Language Proficiency (ELP) Ninth Grade On-Track Four-year Cohort Graduation Rate Five-year Completers Rate Oregon Department of Education 8

Other Accountability Reports Other Reports using Assessment Data (outside of ESSA) Public Assessment Group Reports for current and previous years Title III English Learners (formerly AMAO) Special Education At-A-Glance Profiles for Districts Federal EDFacts Reports Assessment Participation (C185, C188, C189) Academic Achievement (C175, C178, C179) Title III Former LEP (C126) ELPA Participation and Proficiency (C138, C139, C205) Indicator Status (C200, C201) Oregon Department of Education 9

Assessments and Accountability 2 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 10

Timelines for Reports Student and Classroom Reports (in Secure Assessment Reports 2.0 application) ELA, Math, Science, ELPA Summative available since March 24, 2022 ELPA Summative Batch 2 available April 5; Batch 3 available May 10; Batch 4 will be available May 24; Remote administration (Batch 5) will be available June 2 Oregon Extended Assessments will be available June 17 Public Test Results (ELA, Math, Science) Validation Window (in ADI) August 11-19, 2022 Final 2021-22 Test Results Posted mid-September Oregon Department of Education 11

Timelines for Reports contd At-A-Glance School and District Profiles, Accountability Details Accountability Details Validation Window: July 21 October 5, 2022 At-A-Glance Profiles Validation Window: July 28 October 5, 2022 Final reports posted for district preview: October 6, 2022 Final reports posted on the public website: mid-October, 2022 Oregon Department of Education 12

Timelines for Data Sources Institutions for Accountability Reporting Validation Window in ADI: April 14 May 9, 2022 Boundary Calculators for 2021-22 due to ODE: May 1 @ noon (Institution changes for 2022-23 due to ODE: September 15, 2022) Third Period Cumulative ADM Collection Data Collection Window: April 28 May 23, 2022 Audit Review Window: May 26 June 3, 2022 Validation Window in ADI: June 2 August 19, 2022 Title III English Learners Collection Spring Data Collection Window: April 14 May 27, 2022 Review Window: June 9 - 20, 2022 Oregon Department of Education 13

Timeline for Assessments See slides 29-31 for Assessments Load Schedule Non-participation (virtual) records generated based on Spring Membership starting June 1 (editable in the ARUA) Demographic and Program fields filled from Third Period Cumulative ADM (Spring Membership) after May 27 Refreshed and available weekly through the summer (same schedule as for Accountability Details refresh) Oregon Department of Education 14

Timelines for Final Edits Last date for edits to enrollment and assessment records: August 19, 2022 The ARUA closes August 19 at 5 PM Third Period Cumulative ADM closes May 23 at 11:59 PM; reopened during audit window (May 26 June 3) and upon request to Regional ESD Partner or Amanda Leopard; final editing closes August 19 @ 11:59 PM Oregon Department of Education 15

Assessments and Accountability 3 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 16

Links for Webpages Quick Links to Accountability Information Assessment & Accountability Checklist Every Student Succeeds Act (ESSA) Assessment Applications (on the ODE District secure website) Video Training 2021-2022 (on the ODE District secure website) Oregon Department of Education 17

Links for Assessment Help Documents, tools, guidance Quick Links to Accountability Information OSAS Online User Guides Contacts Regional ESD Partners Application assistance: ODE Helpdesk Accountability Reporting: Elyse Bean Oregon Department of Education 18

Questions? 1 Oregon Department of Education 19

Todays Agenda 2 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 20

Assessment Data Reviewing and Editing 1 Changes to Applications and Assessments for 2021-22 Things that Did Not Change for 2021-22 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 21

Changes for 2021-22 - ARUA New application! Assessment Record Updating Application (ARUA) Located within Consolidated Collections application permissions must be assigned by DSA Once permissions have been assigned, you will see Assessment Collections in the menu bar between Staff Collections and Reports. When you hover over Assessment Collections, you will see the Assessment Transactional System 21- 22 collection menu item. There is a stand-alone training video on the Video Training 2021-2022 page (from February) which provides a complete walkthrough of the new application. This new application is a complete replacement for the old Student Centered Staging application. Oregon Department of Education 22

Changes for 2021-22 English Language Arts (ELA), Mathematics, and Science assessments restored to full length; will have claim scores at the individual student level Oregon Extended Assessments will have field test items. The State Board suspended the Essential Skills requirement for this year s juniors. There will be no claim scores for Extended ELA tests. Oregon Department of Education 23

Assessment Data Reviewing and Editing 2 Changes to Applications and Assessments for 2021-22 Things that Did Not Change for 2021-22 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 24

Not Changed in 2021-22 Achievement Standards for ELA, Mathematics, and Science assessments remain the same as for 2020-21 Performance Standards for ELPA Summative remain the same as for 2020-21 Banked test records from this year s 11th graders who met the high school standard in a prior year OAKS Science tests count for participation ONLY no scores or performance levels Identify previous years records by test year (field R in Production Download) or by test date (field S in Production Download) Oregon Department of Education 25

Not Changed in 2021-22 contd Merged SSIDs If test record is already posted, then merged SSID automatically changes assessment record If test record is not yet posted, then loaded with error, and district can confirm SSID (and save record to post) Student demographics and District/School filled with information from SSID record Be sure SSID System has correct demographics at least two days prior to start of student testing; can get help from Regional ESD Partners How to Edit Demographic and Program Fields document EngLrnrCd filled from spring Title III English Learners Collection (calculated by ODE) See Inclusion Rules Manual, Table 8, for list of codes Oregon Department of Education 26

Not Changed in Applications Transactional (ARUA) Student & Institution info (Institution IDs, Student name, grade, etc.) Reporting (AWE) Student & Institution info (Institution IDs, Student name, grade, etc.) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Administration & Accommodation Codes (editable fields) Administration & Accommodation Codes Claim Scores & Perf Levels Reporting Flags Oregon Department of Education 27

Assessment Data Reviewing and Editing 3 Changes to Applications and Assessments for 2021-22 Things that Did Not Change for 2021-22 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 28

Assessments Load Schedule for 2021-22 By May 27 (first closure of the ARUA for processing): ELA & Math tests completed through mid-May Scoring turnaround time for PT is about two weeks Scores are sent to ODE as soon as both parts are completed & scored All ELPA Summative IN-PERSON tests loaded by May 24 High School Domain Proficiency rescores will be in by May 27 Science tests completed by May 26 Oregon Department of Education 29

Assessments Load Schedule for 2021-22 (contd) By June 7 (reopening of the ARUA after second closure for processing): More ELA & Math tests completed through mid-May All ELPA Summative REMOTE tests loaded by June 2 Science tests completed by June 2 (loaded on June 3) Non-Participation ( Virtual ) Records Initial load and available to districts on June 1 by 9 AM Refreshed and available weekly through summer (except week of July 4) Student Group Flags updated from Third Period Cumulative ADM & other student collections Same schedule as Virtual records Oregon Department of Education 30

Assessments Load Schedule (contd) Remaining Loads in June 2022: Oregon Extended Assessments in ELA, Mathematics, and Science Loaded in one batch: tests data-entered by June 3 will be loaded by June 17 Science Partials Loaded by June 17 ELA and Mathematics completed tests Final load by June 21 ELA and Mathematics Partials Loaded by June 30 Oregon Department of Education 31

Assessment Data Reviewing and Editing 4 Changes to Applications and Assessments for 2021-22 Things that Did Not Change for 2021-22 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 32

Accessing Data Different Applications for Different Functions OSAS Portal (vendor for ELA, Math, Science, and ELPA) Student, Class Roster, and Summary Reports BRT s website (Oregon Extended Assessments vendor) Tablet administration, Data Entry, Reports (raw scores) ODE District Secure Login Applications Accountability Warehouse Extract (AWE) Consolidated Collections ARUA for editing test records & correcting errors Secure Assessment Reports for Official Student, Class Roster, and Class Summary Reports Oregon Department of Education 33

Accessing Data - TIDE Cambium s Test Information Distribution Engine (TIDE) on the OSAS Portal Participation Reports Advantages: immediate status reports of tests in progress and completed Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 34

Accessing Data - CRS Cambium s Centralized Reporting System (CRS) on the OSAS Portal Student, Class Roster, and Summary Reports Advantages: immediate Science scores and performance levels; easy to access Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 35

Accessing Data CRS (contd) Cambium s Centralized Reporting System (CRS) CRS User Guide Training Videos (under OSAS Portal ) Oregon Department of Education 36

Accessing Data ODE Applications ODE District Secure Login Applications Oregon Department of Education 37

Accessing Data - AWE Accountability Warehouse Extract (AWE) Download Test Records in Bulk Files or by Selection Criteria Options: Student Search, Bulk Download, or Regional Warehouse Advantages: final scale scores and performance levels; complete test records with all fields; download as spreadsheet Limitations: one-day lag after records loaded or edited in the ARUA, to show up in AWE; cannot edit any data fields Accountability Limitations: none; all accountability report inclusion flags are available for download Oregon Department of Education 38

Accessing Data - SAR Secure Assessment Reports 2.0 (SAR) Individual Student, Class Roster, and Class Summary Reports Advantages: final scale scores and performance levels for total test and all reported score reporting categories Limitations: one-day lag after records loaded or edited in the ARUA, to show up in SAR Accountability Limitations: students are not identified as accountable; class summary reports may have more students included than are accountable Oregon Department of Education 39

Accessing Data - ARUA Assessment Record Updating Application (ARUA) (Consolidated Collections Assessment Collections) View or Edit Test Records, Fix Errors, Download/Upload Files Advantages: district can edit some data fields; displays final scores and performance levels Limitations: one-day lag after scores posted in CRS, to show up in ARUA; subset of fields from test record (no score reporting category / claim scores or performance levels) Accountability Limitations: accountability report inclusion flags not available for download Oregon Department of Education 40

Accessing Data ARUA Closures ARUA Temporary Closures: Closed May 27 (@ 5 PM) through May 31 Reporting & student group flags will be set Virtual records will be created Reopen June 1 by 9 AM Closed June 3 (@ 5 pm) through June 6 Reporting & student group flags will be set Virtual records will be created Reopen June 7 by 9 AM; will stay open until August 19 @ 5 PM 2021-22 Subjects will not be available from Secure Assessment Reports during these closures. Oregon Department of Education 41

Accessing Data ARUA Errors ARUA Error Management Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) If none listed, then no errors to fix! ARUA User Guide Oregon Department of Education 42

Accessing Data ARUA Participation ARUA Participation Report All Records column all Best Scores Not in Denominator column not counted in accountability Participation denominator Participation Denominator column records counted in accountability Participation denominator Participants column records counted in Participation numerator Non-Participants column records counted in Participation denominator as non-participants Oregon Department of Education 43

Accessing Data ARUA Participation Report NOTE: All counts will be in the Not in Denominator column until Participation flags are filled (after May 27, 2022). Oregon Department of Education 44

Accessing Data ARUA Test Lookup Report ARUA Test Lookup Report Look up students at any school to see what subjects they ve tested in No scores are displayed on this report, but there are student names and SSIDs, so please keep secure. Oregon Department of Education 45

Assessment Data Reviewing and Editing 5 Changes to Applications and Assessments for 2021-22 Things that Did Not Change for 2021-22 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 46

Quality Checking Did all grades 3 8 and 11 students enrolled on May 2, 2022 participate in the required assessments? Last day to test students is June 3, 2022 This year s 11thgraders who met the high school standard in a prior year have banked test records. Did all the Oregon Extended Assessments get entered? Last day for ORExt data entry is June 3, 2022 Contact: Caitlin Gonzales Oregon Department of Education 47

Quality Checking (contd) Are these my students? (Does Resident School = May 1 School?) Am I missing students? Review AWE records for accountability reporting flags Are my Non-Participants accurate? Log into AWE, Assessment Student Extract tab Choose the year and subject (i.e., 2021-2022 ELA) Choose Non-Participation under Test Type Open Record Resolution Attributes section, set desired Record Resolution Attribute to Y Refer to Correcting Non-Participants document Oregon Department of Education 48

Quality Checking - Errors Correcting Errors in Assessment Data ARUA Record Management Fix errors online (one record at a time) Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) Download/Upload Errors (in a file, all subjects in one file) Most Common Types of Errors Student ID Match (incorrect or merged SSID) Special Ed Flag not Y (Oregon Extended or Challenge Down) Invalid Accommodation Code 1-6 (must match subject, and accommodation flag must be Y) Oregon Department of Education 49

Quality Checking Non-Participation Valid TstValid Values Test Attempt Code Applicable Subjects Definition Use in Participation Denominator Use in Participation Numerator Eligible for use in Performance Denominator N (Not Valid) P (Partial Attempt) Oregon Extended (CE, CM, NS) Not applicable (NA) NA NA NA N (Not Valid) N (Non-Attempt) OSAS ELA & Math (CE, CM) Any incomplete (1 or more but fewer than 5 CAT responses or no PT responses) test Yes, except CalcAdmnCd = 6, 9 No No N (Not Valid) N (Non-Attempt) OSAS Science (NS) Not applicable (NA) NA NA NA N (Not Valid) N (Non-Attempt) Oregon Extended (CE, CM, NS) Any incomplete (1 or more but fewer than 10 responses) test Yes, except CalcAdmnCd = 6, 9 No No V (Virtual Record) V (Virtual Record) ODE created (CE, CM, NS) Expected test matching Third Period Cumulative ADM Collection, but no test found or test was not attempted Yes, except CalcAdmnCd = 6, 9 No No [Exception in ELA where first year EL can take ELPA in lieu of ELA (EngLrnrCd = B)] Oregon Department of Education 50