Insights on Student Engagement and School Climate: VOCAL Survey Findings

Slide Note

A webinar on exploring student perceptions of school climate and engagement through the VOCAL survey, focusing on Massachusetts students and subgroup variances. The research aims to identify factors influencing student engagement and understand how students view their school experience. Content covers authentic work, grade-level standards mastery, 21st-century skills, and the goals of the study by the Massachusetts Department of Elementary and Secondary Education.

moritzca Follow

Uploaded on Oct 04, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Student engagement webinar: Views of Climate and Learning (VOCAL) survey January 28, 2020

01 Study context VOCAL survey: Measuring student perceptions of school climate and engagement 02 Overview of Massachusetts student perceptions of student engagement CONTENTS 03 Study: Examining subgroup perceptions and factors that might promote or inhibit student engagement 04 Study: Comparing black and white student engagement 05

01 Study context

Study context: Our way forward toward deeper learning Authentic work: o Meaningful or relevant problems in the field of study Authentic work Mastery of grade-level standards: o Deep content knowledge o All learners access content o Embedded assessment/feedback for students Mastery of grade- level standards 21st century skills 21stcentury skills: o Complex problem solving o Communication o Collaboration DESE 4

Study context: Goals of research To examine and better understand: how Massachusetts students perceive their engagement in school how subgroups of students differ in their perceptions of student engagement, and what factors appear to promote or inhibit student engagement Massachusetts Department of Elementary and Secondary Education 5

VOCAL survey: Measuring student perceptions of school climate and engagement 02

2019 VOCAL design: Defining school climate C C IC I Engagement T E M S IIC School Climate Safety IIIC Environment Students respond: Always true (3); Mostly true (2) Mostly untrue (1); Never true (0) *Based on the United States Department of Education s conceptual framework for school climate. DESE 7

2019 VOCAL survey: Engagement topics defined* Engagement dimension Sample items: Grade Students like to have friends who are different from themselves (for example, boys and girls, rich and poor, or classmates of different color). G4, G5 Cultural competence The extent students feel adults/students value diversity, manage dynamics of differences, and avoid stereotypes. My textbooks or class materials include people and examples that reflect my race, cultural background and/or identity. G8 Within school, I am encouraged to take upper level courses (honors, AP). G10 Relationships The extent students feel there is a social connection and respect between staff/teachers and students, and between students and their peers. My teachers care about me as a person. G4, G5 Students respect one another. G4, G5, G8, G10 My teachers promote respect among students. G8, G10 Participation The extent students feel engaged intellectually, emotionally, and behaviorally in the classroom, and the extent that students or their parents are engaged in school life. When I am stuck, my teachers want me to try again before they help me. G4, G5 In my classes, my teachers use students' interests to plan class activities. G8 If I finish my work early, I have a opportunity to do more challenging work. G10 *Based on the United States Department of Education s conceptual framework for school climate. DESE 8

2019 VOCAL survey: Safety topics defined* Safety dimension Sample items: Grade Teachers support (help) students who come to class upset. G4, G5, G8, G10 Emotional safety The extent students feel a bond to the school, and the extent adults/students support the emotional needs of students. I am happy to be at our school. G4, G5 I feel comfortable reaching out to teachers/counselors for emotional support if I need it. G8 Physical safety The extent that students feel physically safe within the school environment. I feel safe at our school. G4, G5 I have seen students with weapons at our school. G8 Students are sexually harassed at my school (for example, bothered by unwanted touching and/or indecent name-calling). G10 Bullying/Cyber-bullying The extent that students report different types of bullying behaviors occurring in the school and the extent that school/staff/students try to counteract bullying. If I tell a teacher or other adult that someone is being bullied, the teacher/adult will do something to help. G5, G8, G10 In my school, groups of students tease or pick on one student. G4, G5, G8, G10 I have been teased or picked on more than once because of my race or ethnicity. G10 *Based on the United States Department of Education s conceptual framework for school climate. DESE 9

2019 VOCAL survey: Environment topics defined* Supportive Environment dimension Sample items: Grade Students help each other learn without having to be asked by the teacher. G5, G8, G10 Instructional environment The extent that students feel the instructional environment is collaborative, relevant, challenging and supportive of learning. My teachers are proud of me when I work hard in school. G4, G5, G8 My teachers set high expectations for my work. G8, G10 Mental health environment The extent that students have access to support systems that effectively support their social, emotional and mental health well-being. At our school, students learn to care about other students' feelings. G4, G5 Our school offers guidance to students on how to mediate (settle) conflicts (e.g., arguments, fights) by themselves. G8 The level of pressure I feel at school to perform well is unhealthy. G10 Discipline environment The extent that discipline is fair, applied consistently and evenly, and a shared responsibility among staff, teachers, and students. Teachers give students a chance to explain their behavior when they do something wrong. G5, G8, G10 School staff are consistent when enforcing rules in school. G8 The consequence for the same inappropriate behavior (e.g., disrupting the class) are the same, no matter who the student is. G10 *Based on the United States Department of Education s conceptual framework for school climate. DESE 10

Indices are used as an objective metric for measuring differences in student perceptions of school climate and engagement Student-level (all grades): Baseline average (2018) 70 50 30 1 99 SD = 20 2019 state-level dimension scores (engagement, safety, and environment) are centered around 51 to 52 points 2019 state-level topic scores average from 50 (Bullying) to 53 (Physical safety) DESE 11

Overview of Massachusetts student perceptions of engagement 03

Students in upper grades have less favorable views of their engagement when compared to students in lower grades 70 65 ENGAGEMENT INDEX SCORE 59 60 55 15point difference in engagement score between grade 4 and grade 8 is equivalent to a 32- percentile difference 55 50 45 44 45 40 35 30 GRADE 4 GRADE 5 GRADE 8 GRADE 10 DESE

The gap between lower and upper grades is of a similar magnitude across all three topic areas of engagement G4 G5 G8 G10 70 59 60 59 57 55 55 60 Index scaled score 55 55 47 47 47 50 45 45 45 44 44 40 30 20 10 0 Engagement Participation Relationships Cultural competence 12 point gap in cultural competence score between G4 and G8 is equivalent to a 26 percentile difference 15 point gap in relationships score between G4 and G8 is equivalent to a 29 percentile difference DESE 14

The decline in average engagement scores between lower and upper grades mirrors the decline in other dimensions of school climate G4 G5 G8 G10 70 61 62 60 59 57 56 60 Index scaled score 56 55 50 45 45 44 44 44 44 43 43 40 30 20 10 0 Overall Engagement Safety Environment 16 point difference in safety score between grade 4 and grade 8 is equivalent to a 30 percentile difference 18 point difference in environment score between grade 4 and grade 8 is equivalent to a 34 percentile difference (=1 standard deviation) DESE 15

Take-aways: Overview The high average levels of student engagement reported by younger students diminish considerably as students grow older Older students also view their learning environments as less supportive and safe than younger students This decline in student engagement is not unique to Massachusetts, it has been found in other states and in nationally representative samples. DESE

Research study: Examining subgroup perceptions of student engagement and factors that might promote or inhibit student engagement 04

Research study context: Sample design Grade 4 55,796 (82%) Grade 5 59,456 (85%) Grade 8 59,180 (85%) 53,274 (76%) Grade 10 Number of students with VOCAL data Number of students in study Number of schools in study Number of items 48,848 (72%) 54,915 (78%) 55,009 (79%) 51,864 (74%) 647 623 367 296 14 14 13 13 Inclusion criteria: Students in schools with N 10 students, and A school-level engagement score reliability of 0.7 DESE 18

Study design: Explaining differences in engagement perceptions (1) School 1 School 2 Different school children stand set in flat style vector illustration Different school children stand set in flat style vector illustration Differences between schools ~5% to 7% ~93 to 95% ~93 to 95% Differences within schools Differences within schools * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 19

Study design: Explaining differences in engagement perceptions (2) Between schools: Urban Charter Chronic absence Discipline rates Percent Black, Hispanic School enrollment Teacher license Teacher attendance Teacher retention Principal retention School average: Emotional & Physical safety Bullying Instructional environment Mental-health environment Discipline environment Different school children stand set in flat style vector illustration Within schools (individual): Gender Race/ethnicity Economically disadvantaged English learner Student with disabilities Lowest performing group In-school suspension Out-of-school suspension Attendance College plans (G8, G10) Individual perception: Emotional & Physical safety Bullying Instructional environment Mental-health environment Discipline environment * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 20

Study design: Interpreting differences (very small to very large effects) -ve: negative effect +ve: positive effect Different school children stand set in flat style vector illustration Differences between schools or Difference in: Size of effect (-ve or +ve) Very small Standard deviation units Percentiles Less than .04 Less than 2 Small .04 to less than .10 2 to less than 4 Moderate .10 to less than .20 4 to less than 8 Large .20 to less than .30 8 to less than 12 Very large .30 and above 12 and above Differences within schools * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 21

Study results: Student characteristics that help explain variation in student engagement (controlling for all other factors in the model) Different school children stand set in flat style vector illustration Student characteristic Male Direction -ve G4, G5 ; +ve G8, G10 +ve (G10 only) -ve (no effect G4) +ve G4, G5 -ve (no effect G4) Magnitude of effect Small G4, G5, G8 Moderate G10 Small Small (G5) to moderate Small Small (G5) to moderate Asian Black Hispanic NAMP (Native American; Multi-race, Non- Hispanic; Pacific Islander, Native Hawaiian) Economically disadvantaged English learner (EL) Student with disability (SWD) Lowest performing group (LPG) +ve G4, G5 +ve all grades +ve all grades +ve G4, G5 ; -ve G8, G10 -ve all grades +ve (no effect G4) +ve v. Small Moderate (v. large G10) v. Small to Small (upper) Moderate (small G8) Within schools (individual): -ve: negative effect +ve: positive effect In-school suspension (ISS) Attendance College plans Large (v. large G8) Small Moderate (G8) Large (G10) * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 22

Study results: School-level factors that help explain variation in student engagement (controlling for all other factors in the model) School factor Urban Charter Chronic Absence rate Discipline rate Percent Black Percent Hispanic School enrollment Teacher license Teacher attendance Teacher retention Principal retention Direction of effect No effect No effect -ve (G10 only) -ve all grades No effect +ve (G8 and G10) -ve (G10 only) No effect No effect +ve (G8 and G10) No effect Magnitude of effect No effect No effect Small Small No effect Small v. Small No effect No effect Small No effect Between schools: Different school children stand set in flat style vector illustration -ve: negative effect +ve: positive effect * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 23

Study results: Students perceptions of safety and environment that help explain variation in student engagement (controlling for all other factors) Different school children stand set in flat style vector illustration Individual perceptions Emotional safety Physical safety Bullying Instructional environment Mental-health environment Discipline environment Direction +ve all grades +ve all grades +ve all grades +ve all grades +ve G4, G5, G8 +ve all grades Magnitude of effect Moderate, Large (G5) v. Small Moderate Very large Moderate Large Within schools (individual): -ve: negative effect +ve: positive effect A higher bullying climate score corresponds to a more favorable bullying climate * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 24

Study results: Safety and environment school-level factors that help explain variation in student engagement (controlling for all other factors) Between schools: Different school children stand set in flat style vector illustration School-level factor (average) Emotional safety Physical safety Bullying Instructional environment Mental-health environment Discipline environment Direction No effect +ve G4 +ve G8, G10 +ve all grades -ve G8, G10 No effect Magnitude of effect No effect Small Moderate Moderate (Large G5) Moderate No effect -ve: negative effect +ve: positive effect * Image from Dreamtime.com Massachusetts Department of Elementary and Secondary Education 25

Summary data observations: Most differences in student engagement are among students attending the same schools (~94%) rather than among students attending different schools (~6%) Across several subgroup comparisons (e.g., male, black, NAMP, EL, ISS, OSS), the size of the gap (-ve or +ve) in student engagement tends to widen as students move into the upper grades School-level factors and individual student characteristics explain only a small, albeit significant, percentage of the overall variation in student engagement (2% G4 to 7% G10) When student views of school safety and of their school environment are included, a larger proportion of the variation in student engagement is explained (66% G4 to 73% G10) DESE

Key within-school findings: Student characteristics associated with lower engagement (after controlling for all other factors!) Students who have been suspended at least once (especially ISS) report lower levels of student engagement; the ISS gap is large or very large In comparison to white students, black and NAMP students report lower student engagement and the magnitude of the gap widens in later grades In younger grades, male students report lower engagement than females; in later grades this gap reverses direction and widens DESE

Key within-school findings: Student characteristics associated with higher engagement (after controlling for all other factors!) English learners report higher levels of student engagement; the size of this positive gap increases when students are in middle or high school Students with disabilities consistently report more positive student engagement across all grades Students with higher attendance report higher levels of student engagement In comparison to white students, Hispanic students report more positive student engagement in lower grades; however, this positive effect is not present in upper grades In lower grades, students in the lowest performing group (LPG) within schools report higher engagement than non-LPG students; in later grades, this gap reverses direction and widens DESE

Key within-school findings: Students who view their own safety and their environment more positively report higher levels of engagement Students who report more positive instructional environments (strongest predictor) and discipline environments within their schools are more likely to report higher levels of engagement The positive effect of students emotional security on student engagement appears to lessen as students get older The beneficial effect of a positive bullying climate on student engagement strengthens (slightly) as students get older By grade 10, there is no association between a students level of engagement and their views on the mental-health supports within their school; in the lower grades this association was positive Across all grades, students who feel physically safer within their school report higher levels of engagement DESE

Key between-school findings: School factors that likely promote or inhibit student engagement (after controlling for all other factors!) Across all grades, in schools with higher average discipline rates, student engagement is inhibited In upper grades, schools with higher average teacher retention rates and a higher percentage of Hispanic students are associated with higher student engagement In high schools with higher average chronic absence rates, student engagement is inhibited Larger high schools are associated with lower student engagement DESE

Key between-school findings: School factors that likely promote or inhibit student engagement (after controlling for all other factors!) When students attend schools with more positive instructional environments and bullying climates (upper grades only), student engagement is promoted In grade 4 only, when students attend schools with higher average levels of physical safety, student engagement is promoted In upper grades, when students attend schools with a more positive mental health environment, student engagement is inhibited! DESE

Research study: Comparing black and white student engagement 05

In the lower grades, relationships differentiate engagement perceptions between black and white students GRADE 4 (N = 48,848) GRADE 5 (54,915) White (56.4%) Black (9.4%) White (58.7%) Black (8.9%) 70 70 61 59 59 59 58 58 56 56 56 60 60 55 55 54 54 54 54 Index scaled score 49 50 50 40 40 30 30 20 20 10 10 0 0 Engagement Cultural competence Relationships Participation Engagement Cultural competence Relationships Participation Overall Engagement difference: Small effect Overall Engagement difference: Not significant DESE 33

In younger grades, black students view their school environment as less safe and to a lesser extent supportive when compared to their white peers Standardized difference between groups Grade 5 Grade 4 Discipline -0.10 -0.10 Mental-health -0.16 -0.26 Instructional 0.02 0.10 Bullying -0.36 -0.37 Physical safety -0.37 -0.32 Emotional safety -0.14 -0.09 Participation 0.11 0.17 Relationships -0.36 -0.36 Cultural comp. -0.05 -0.05 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 -0.4 -16 -0.3 -12 -0.2 -8 -0.1 -4 0.0 0 0.1 +4 0.2 +8 -16 -12 -8 -4 0 +4 +8 Percentile difference Percentile difference DESE 34

In younger grades, relationships differentiate engagement perceptions between black and white students Grade 4 Black 58 90 91 Grade 5 Black 54 84 89 Percent always true and mostly true Think of the last 30 days in school: Adults working at this school treat all students with respect. Teachers at this school accept me for who I am. Students like to have friends that are different from themselves (e.g., boys and girls, people of different color) I read books in class that include people who are similar to me (e.g., we look the same, speak the same, or live in ..) Students respect each other in my school. Students at my school get along with each other. My teachers care about me as a person. My classmates behave the way my teachers want them to. My teachers use my ideas to help my classmates learn. When I need help, my teachers use my interests to help me learn. White 59 94 94 White 55 91 93 Topic CLC: CLC: % -4 -3 % -7 -4 CLC: -3 -4 91 88 90 86 CLC: +3 + 3 56 59 52 55 REL: REL: REL PAR: PAR: -15 -12 -8 -9 +6 -27 -14 -3 -11 +5 80 81 96 72 72 65 69 94 63 78 83 77 94 65 68 56 63 91 54 73 PAR: +5 +3 78 82 73 76 DESE 35

In the lower grades, black students view their school environment as less safe and to a lesser extent supportive when compared to their white peers Grade 4 Black Grade 5 Black Percent always true and mostly true Think of the last 30 days in school: I am happy to be at our school. Topic EMO: EMO: PSF: PSF: BUL: BUL: BUL: White % White % -7 -3 +5 +10 +9 +15 +9 87 82 -5 -2 +5 +11 +13 +16 +5 84 77 Teachers support (help) students who come to class upset. I am safe at our school. I have seen more than one fight at my school in the last . I have been hit by other students... (G5: punched & shoved) groups of students tease or pick on one student. Students at school try to stop bullying when they see it My teachers are proud of me when I work hard. 92 93 38 32 30 79 90 88 49 45 46 74 91 92 37 32 33 73 88 87 47 41 48 64 INS: -2 0 90 88 95 95 At our school, students learn to care about other students feelings. School rules are fair for all students. Teachers give students a chance to explain when they do something wrong. MEN: -4 -7 87 83 82 75 DIS: -5 -7 90 85 87 80 DESE DIS: -4 -8 89 84 80 72 36

In the upper grades, cultural competence and relationships differentiate engagement perceptions between black and white students GRADE 8 (N = 55,009) GRADE 10 (N = 51,864) White (62.2%) Black (9.0%) White (65.2%) Black (8.1%) 70 70 60 60 Index scaled score 48 48 47 46 50 50 45 45 44 44 44 44 44 43 43 42 41 41 40 40 30 30 20 20 10 10 0 0 Engagement Cultural competence Relationships Participation Engagement Cultural competence Relationships Participation Overall Engagement difference: Moderate effect Overall Engagement difference: Small effect DESE 37

In the upper grades, black students view their school environment as less safe and to a small extent less supportive when compared to their white peers Standardized difference between groups Grade 8 Grade 10 Discipline 0.03 -0.05 Mental-health -0.10 0.00 Instructional 0.17 0.10 Bullying -0.11 -0.23 Physical safety -0.31 -0.21 Emotional safety -0.06 -0.25 Participation 0.06 0.16 Relationships -0.21 -0.10 Cultural comp. -0.32 -0.26 -0.4 -16 -0.3 -12 -0.2 -8 -0.1 -4 0.0 0 0.1 +4 0.2 +8 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 -12 -8 -4 0 +4 +8 -16 Percentile difference Percentile difference DESE 38

In upper grades, cultural competence and relationships differentiate engagement perceptions between black and white students Grade 8 Black Grade 10 Black Percent always true and mostly true Think of the last 30 days in school: Adults working at this school treat all students with respect regardless of a student s race, culture, family income ... Students from different backgrounds respect each other in our school regardless of their race, culture, family income My textbooks or class materials ..reflect my race, ..identity. Within school, I am encouraged to take upper level courses. My teachers promote respect among students. Students respect one another in my school. Adults at our school are respectful of student ideas even it the ideas expressed are different from their own. Teachers are available when I need to talk with them. My teachers use my ideas to help my classmates learn. ..students are asked to teach a lesson or part of a lesson. White 44 White 45 Topic 41 % 44 % CLC: 90 81 -9 89 80 -9 CLC: 81 72 -9 85 75 -10 CLC: CLC: REL: REL: 61 NA 90 60 55 NA 85 47 -6 NA -5 -13 NA 77 91 69 NA 73 87 56 NA -4 -4 -13 REL 79 75 -4 79 76 -3 REL: PAR: PAR: 85 42 NA 78 47 NA -7 +5 NA 88 53 28 84 55 39 -4 +2 +11 DESE 39

In the upper grades, black students view their school environment as less safe and to a lesser extent supportive when compared to their white peers Grade 8 Black 70 NA NA 49 NA 33 47 NA 90 Grade 10 Black Percent always true and mostly true Think of the last 30 days in school: Teachers support students who come to class upset. Topic EMO: EMO: EMO: PSF: PSF: BUL: BUL: BUL: INS: DIS: White 75 NA NA 33 NA 25 38 NA 89 % -5 NA NA -16 NA -8 -9 NA +1 White 73 82 58 NA 17 NA 29 12 90 % -5 -9 -10 NA -9 NA -10 -12 -1 68 73 48 NA 26 NA 39 24 89 I feel as though I belong in my school community. Student . work out their problems respectful way. Students at school damage/steal other students property. Students are sexually harassed at my school (e.g., ) .bigger students taunt or pick on smaller students groups of students tease or pick on one student. I have been teased or picked because of my race/ethnicity My teachers set high expectations for my work. Teachers give students a chance to explain their behavior when they do something wrong. The consequences for the same inappropriate behavior are the same, no matter who the student is. -6 -5 53 47 54 49 DIS: -4 NA NA NA 63 59 DESE 40

Take-aways: Engagement differences between black and white students On average, black and white students have comparable views of their engagement; however, differences are apparent in some aspects of student engagement For the most part, black students across all grades are more likely to report they have opportunities for meaningful participation in the classroom; they also tend to express more positive views of the instructional environment used to engender student participation in the classroom Poorer and less respectful relationships among students, especially among younger students, may inhibit black student engagement A lack of respect and valuing of diversity may inhibit black student engagement (this negative effect is strongest in later grades) DESE

Take-aways: Safety and environment differences between black and white students Black students are more likely to perceive that bullying and physical violence is a problem in their schools; concerns for their safety may inhibit their ability to engage in the classroom and school In three of the four grades, black students report a harsher discipline environment; when combined with lower levels of reported emotional security and of mental-health supports, these factors may impede black students ability to engage in school DESE

Our way forward: Some thoughts! Could a move toward deeper learning help engage students in the upper grades? Exposure to more authentic work in their field of study may help older students view the relevancy of what they learn in school more favorably and increase their engagement Learning through collaborating and communicating with their peers in solving complex problems could not only support their higher-order thinking skills, but their engagement in school Providing students with more embedded feedback on their learning and including them in the conversation could not only support their desire for autonomy, but their engagement and self-efficacy. If all students are to take full advantage of deeper learning opportunities, all students will need equal access to safe, enriching, and supportive environments DESE

VOCAL report resources: http://www.doe.mass.edu/research/vocal/2019/ and Dropbox Annotated report (Dropbox only) o Describes the layout of the report and each data element of the VOCAL report School climate profiles by grade o Describes the average student s experience within a relatively weak, typical, and relatively strong school Interpretive guide and Action planning guide and Sample report o Describes how to systematically analyze your data to identify strengths and areas to improve o Describes how to develop an action plan based on your key findings User guide o Describes the appropriate use and interpretation of the two types of VOCAL scores Item-level crosswalk o CASEL s SEL competencies; PBIS framework; AI framework VOCAL contact vocalsurvey@doe.mass.edu DESE 44

Feedback opportunities: Your feedback helps DESE support you better! 1. February is VISTA survey month o Principals o Superintendents 2. Health Profiles o Principals o Health Coordinators DESE