Strategies for Assessing Learning Support Initiatives

This presentation by Nisha Abraham explores the effectiveness of learning support initiatives with a focus on the Supplemental Instruction (SI) program at UT Austin. It discusses challenges in assessing SI, components of SI, assessment methods, and real-world examples of program assessments.

• Learning support
• Assessment strategies
• Supplemental Instruction
• Program effectiveness

sellers_m Follow

Uploaded on Feb 28, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Strategies for Assessing the Effectiveness of Learning Support Initiatives Nisha Abraham

2. Welcome Former Roles: Assistant Director Course Support and Instruction Lead Senior SI Program Coordinator Sanger Learning Center, UT Austin

3. A quick poll!

4. Presentation Objectives Together, we will: Learn about the Supplemental Instruction (SI) program and what makes it difficult to assess Explore four real-world examples of assessments conducted within an SI program Review a diverse range of data sources that can be employed to gauge program effectiveness Discuss what you would like to evaluate in your program and what data sources are accessible

5. Supplemental Instruction Supplemental Instruction (SI) is an internationally recognized academic support program that employs students to conduct two identical sessions a week that help students work through difficult content in specific courses while engaging in effective study strategies. Course Content Study Skills SI The program at UT Austin is run by the Sanger Learning Center, in collaboration with departments and faculty members.

6. Components of Supplemental Instruction 1. 2. Student facilitated Historically difficult gateway classes Offered to all enrolled students in that course Regularly scheduled Content and skills integrated 3. 4. 5.

7. How SI is Assessed AY 17-18, QDFW Rates for EE302, SI vs no-SI groups

8. The challenges in assessing SI Voluntary Nature One Semester Other Supports Perceptions Who is attending SI? Is SI really teaching students long term skills? If courses have other resources students are using, can you make causation claims? How do students perceive group work? How do we accurately compare students? What if you see no impact in the current semester? How do students perceive peer-led instruction? Maybe only correlations?

9. We wanted to go further: Make comparisons with course grade GPAs Approach Accurately compare students in SI and non-SI groups Understand long-term effects, if any Assess students perception of group work and its impact to learning

10. Program Assessment Research Question Data Collected Analysis When controlling for certain demographic information, do students who attended SI sessions receive higher end of semester course grades? SI attendance data Compared average course GPAs for SI and non-SI groups -matching students based on demographics for a more apples to apples comparison Demographic data from university database (gender, ethnicity, parent s income level, and SAT scores) Course grade data

11. Students who attended SI sessions received, on average, end of semester course grades 0.383 points higher than those not attending SI sessions for ECO 304K (p < 0.05, 95 % CI[0.294, 0.471], n =1,178)

12. Research Study -Long Term Effects Research Question Data Collected Analysis Is there any correlation between SI attendance in EE 302 and performance in EE 411? SI session attendance Compared median course GPAs for SI and non-SI groups, then tracked them into subsequent course Course grades and semester grades Pre-semester and post- semester cumulative GPAs Is there a relationship between SI attendance and future semester grade point averages? SAT scores

13. The no-SI group had an 8% higher mean SAT score, and about a 4% higher predicted GPA compared to the SI group. However the SI group outperformed the no-SI group in EE 411, though the SI group s course grades for EE 302 were lower. (n=86)

14. Research Study -Additional Supports Research Question Data Collected Analysis SI session attendance Compared average course GPAs for SI and non-SI groups, using SAT score ranges How does students initial awareness and perceptions of resources influence SI attendance? Course grades and semester grades SAT scores How does students perceptions of SI and it s learning model (peer-led collaboration) influence intention to attend SI sessions? Open coding to determine general themes Pre-survey -assessed awareness of resources and plan to use resources Post-survey -assessed perceived benefits of SI Identifying trends

15. Students who attended SI sessions received higher average end of semester course GPAs than those not attending SI sessions, when comparing students in SAT score ranges (n=266)

16. Survey Responses The two aspects of interest with these resources were the expertise offered and the student ratio and interaction. A significant decrease is shown between their awareness of one-on- one tutoring and SI sessions and their intention to use certain services (one-on-one tutoring and SI).

17. Research Study -Cohort Based Research Question Data Collected Analysis Compared average course GPAs for SI and non SI groups within each category of previous programming experience SI session attendance Did having students register for the same SI Session and requiring attendance to that same session affect attendance trends and therefore overall course GPAs and QDFW rates for SI students? Course grades and GPAs Previous programming experience End of semester surveys What were SI students perceptions of the registration approach and cohort-based SI sessions? Emerging themes from small group/one-on-one interviews Small group/one-on-one interviews

18. Students who attended SI sessions received higher average end of semester course GPAs than those not attending SI sessions, when comparing students with no programming experience * indicates t-test p value = 0.032 (statistically significant)

19. EOS Survey Responses Responses to the end of semester survey questions demonstrated a much higher positive response to SI sessions than previous semesters, despite a completely virtual offering.

20. Emerging Themes 1. Impact of knowing/having the same peers in sessions each week 2. Impact of the SI Leader 3. Accurate understanding of SI as a different type of learning environment 4. Perceptions of small group work 5. Perceptions of initial email and ongoing communications 6. Perceptions of being required to attend sessions regularly 7. Translating SI activities/changing study strategies to own learning 8. Impact of the demographics of the group or SI Leader Vestibulum congue Impact to learning Impact to Vestibulum congue community/connectedness

21. Your turn! What metrics are you using to evaluate your program? What innovations would you like to make?

22. Questions? Thank you!

23. References Abraham, N., & Telang, N. K. (2021, July). Cohort-based supplemental instruction sessions as a holistic retention approach in a first-year engineering course. In 2021 ASEE Virtual Annual Conference Content Access. Abraham, N., & Telang, N. (2018). Understanding behaviors of attendance in Supplemental Instruction and subsequent academic success in a first year engineering course Conference Proceedings of the ASEE Gulf-Southwest Section Annual Conference The University of Texas at Austin Retrieved from http://www.aseegsw18.com/uploads/1/0/2/0/102073464/3a1_abraham_final_paper_understanding_behaviors_of_attendance_an d_academic_success_to_supplemental_instruction_in_first_year_engineering_course_revision_12mar18.pdf Wilmot, J., & Telang, N. K. (2017). Assessment of Supplemental Instruction programming and continued academic success. Conference Proceedings of the 2017 American Society of Electrical Engineers Annual Conference & Exposition, Columbia, OH. Retrieved from https://peer.asee.org/27643 Wilmot, J., & Telang, N. K. (2017). Assessment of Supplemental Instruction programming on first year academic success. Conference Proceedings of the 2017 American Society of Electrical Engineers Annual Conference & Exposition, Columbia, OH. Retrieved from https://peer.asee.org/27644 International Center for Supplemental Instruction (n.d.). University of Missouri, Kansas City. Retrieved from https://info.umkc.edu/si/