Early Alert and Retention Prediction: Improving Student Success in Higher Education
The Early Alert system enables timely communication between faculty, advisers, and students, directing students to resources such as tutoring centers. By targeting weaker students and improving contact rates, it has shown positive impacts on student performance and retention, especially in developmental courses. However, challenges remain in ensuring student utilization of referral resources and email communication effectiveness.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Early Alert and Retention Prediction Elisabeth Lackner elackner@qcc.cuny.edu Patrick Wynne pwynne@qcc.cuny.edu
FUNCTIONALITY OF THE EARLY-ALERT SYSTEM Enables timely communication Enables timely communication between address academic problems and address academic problems and direct students to resources between faculty, advisers, and students faculty, advisers, and students to direct students to resources such as tutoring. to such as tutoring. Faculty Faculty raise flags attendance, send kudos refer refer students to resources such as tutoring. flags for course performance and kudos for good or improved work, Adviser/Counselor Adviser/Counselor review all alerts and take action: contact students, set up appointments, refer students. Tutoring & Support Tutoring & Support Centers Centers review referrals, schedule appointments take notes, and notify faculty. Students Students receive an email message for every alert, kudos, and referral with faculty comments. The system allows for reports on all activities. The system allows for reports on all activities.
Are students in need reached? Freshmen flagged for poor performance or risk of failure had lower high school GPA averages and higher remedial needs (developmental mathematics, reading, and writing). Continuing students who were flagged for poor course performance or risk of failure had on average a lower cumulative GPA (2.18) at the start of the semester for which they were flagged than continuing students who were not flagged at all (2.66). 4.00 3.00 2.00 1.00 0.00 Histogram: Cumulative GPA difference between continuing students flagged and not flagged, fall 2012, spring 2013, fall 2013, and spring 2014 combined
SUMMARY OF FORMATIVE FINDINGS (HOW IS THE INTERVENTION IMPLEMENTED?) 1. The intervention targets weaker students. 2. Referrals to actual resources, in particular, to tutoring centers have not been utilized heavily. 3. Less than 60% of the students with referrals actually did seek help from a tutoring center. 4. Students don t reliably read college email messages. 5. The implementation of Starfish in fall 2013 increased the successful contact of students. 6. Students were contacted more often about their academic performance than they had expected at the start of school. 7. Faculty and advisers both reported that the system enhances their work and ability to serve the needs of students.
Does Early Does Early- -Alert improve student performance in the course? Alert improve student performance in the course? Quantitative: Participating vs Non-Participating Courses Developmental courses (remedial reading, writing, and mathematics) that participated in early- alert had higher pass (percent of P) and completion rates than courses with the same subject that did not participate in early-alert. Course pass rates in developmental courses 80.0% 70.0% 60.0% 44.1% 50.0% 40.2% 40.1% 39.9% 38.6% 37.1% 34.7% 33.8% 40.0% 30.0% 20.0% 10.0% 0.0% Fall '13 Spring '14 Fall '14 Spring '15 Course sections not using Early Alert Course sections using Early Alert
Does Early Does Early- -Alert improve student performance in the course? Alert improve student performance in the course? Quantitative: Participating vs Non-Participating Courses Course completion rates in developmental courses 79.7% 77.1% 76.9% 75.5% 80.0% 74.5% 74.1% 74.0% 70.4% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Fall '13 Spring '14 Fall '14 Spring '15 Course sections not using Early Alert Course sections using Early Alert
Do interventions improve the academic success of students in need in general? Semester Specific Performance Outcomes (fall 2012, spring 2013, fall 2013) High-Risk Continuing Students (cum GPA < 2.0) Semester credits and hours earned Semester GPA Cumulative GPA Withdrawal from (a) course(s) yes/no
Semester outcomes of high-risk continuing students who were not flagged after propensity score matching High-risk continuing students who had a cumulative GPA below a 2.0 at the start of the semester for which the outcomes were observed, and who were not flagged through early-alert all outperformed high-risk continuing students who were flagged (with and without tutoring). N = 1,770 in percent 63% 58% 54% 53% 40% 35% 35% 33% 32% 25% Semester Credits & Hours Passed >= 10 Semester GPA >=2 Withdrawal credits - yes Cumulative GPA >=2 Passed at Least 75% of Completed Courses Had Tutoring (N=885) Had No Tutoring (N=885)
SUMMATIVE: Semester outcomes of high-risk continuing students who were flagged after propensity score matching Tutoring seems to have a positive effect on course persistence for the most vulnerable student group high risk students who were also flagged through the system. N = 2,220 in percent 50% 44% 37% 34% 31% 31% 21% 21% 19% 17% Semester Credits & Hours Passed >= 10 Semester GPA >=2 Withdrawal credits - yes Cumulative GPA >=2 Passed at Least 75% of Completed Courses Had Tutoring (N=1,110) Had No Tutoring (N=1,110)
SUMMARY OF SUMMATIVE FINDINGS 1. Course pass and completion rates have not improved on a campus- wide scale since the start of the intervention. 2. Unofficial withdrawal rates have declined campus-wide since the start of the intervention. 3. Developmental reading, writing, and mathematics courses that participated in the intervention had higher pass and completion rates than courses with the same subject that did not participate in the intervention. 4. The evidence suggests that the combination of early-alert with tutoring benefits students the most, particularly struggling students. 5. Students who were flagged through early-alert and had no tutoring had the lowest outcomes.
PREDICTIVE ANALYTICS Starfish provides us with data on student performance in the form of flags , kudos, and referrals. Starfish also provides us with a record of all student learning center visits As a platform, Starfish allows us to easily share information with faculty, advisors, or tutors
FUNCTIONALITY OF PREDICTIVE ANALYTICS The Starfish platform allows us to easily relay messages or data to the proper The Starfish platform allows us to easily relay messages or data to the proper student support personnel student support personnel 805 805 Extract Extract Prediction Prediction Model Model Starfish Starfish Environment Environment Starfish Starfish Tracking Report Tracking Report The system allows for reports on all activities. The system allows for reports on all activities.
HOW DO WE DEFINE AT-RISK? Our analysis is currently focused on identifying students at risk of not returning the following semester. Students enrolled in Spring that do not return for the following Fall semester. Students that graduate in the spring or summer are marked as retained
805 TABLES USED TO TRAIN MODELS The models using historic 805 extracts in conjunction with tracking item and appointment data from the Starfish CRM The 805 tables currently included in the model: addresses class_tbl ethnic_group ext_degr Stdnt_test_comp persons personal_data stdnt_car_term stdnt_enrl
DROP OUT RISK: BOOSTED TREES MODEL XGBoost package (Chen et al., 2015) Parameters 2,000 trees, Each tree 5 deep Sub sample 90% of the cases Column sample 50% of variables
SEMESTER PERSISTENCE MODEL PERFORMANCE Probabilty Probabilty Cut Cut- -Off Off Total Total Selected Selected Drop Outs Drop Outs Selected Selected Accuracy Rate Accuracy Rate (% Correct) (% Correct) Non Non- -Drop Outs Drop Outs Selected Selected Drop Outs not Drop Outs not Selected Selected Sensitivity Sensitivity Rate Rate False Positive False Positive Rate Rate 40 40 45 45 50 50 55 55 60 60 65 65 70 70 75 75 80 80 85 85 90 90 95 95 100 100 524 713 922 1,157 1,489 1,858 2,338 2,796 3,343 3,969 4,564 4,818 4,839 356 469 568 657 766 880 1,011 1,109 1,222 1,306 1,365 1,378 1,379 68% 66% 62% 57% 51% 47% 43% 40% 37% 33% 30% 29% 29% 168 244 354 500 723 978 1,327 1,687 2,121 2,663 3,199 3,440 3,460 1,023 910 811 722 613 499 368 270 157 26% 34% 41% 48% 56% 64% 73% 80% 89% 95% 99% 100% 100% 5% 7% 10% 14% 21% 28% 38% 49% 61% 77% 92% 99% 100% 73 14 1 0
SEMESTER PERSISTENCE MODEL PERFORMANCE Probabilty Probabilty Cut Cut- -Off Off Total Total Selected Selected Drop Outs Drop Outs Selected Selected Accuracy Rate Accuracy Rate (% Correct) (% Correct) Non Non- -Drop Outs Drop Outs Selected Selected Drop Outs not Drop Outs not Selected Selected Sensitivity Sensitivity Rate Rate False Positive False Positive Rate Rate 40 40 45 45 50 50 55 55 60 60 65 65 70 70 75 75 80 80 85 85 90 90 95 95 100 100 524 713 922 1,157 1,489 1,858 2,338 2,796 3,343 3,969 4,564 4,818 4,839 356 469 568 657 766 880 1,011 1,109 1,222 1,306 1,365 1,378 1,379 68% 66% 62% 57% 51% 47% 43% 40% 37% 33% 30% 29% 29% 168 244 354 500 723 978 1,327 1,687 2,121 2,663 3,199 3,440 3,460 1,023 910 811 722 613 499 368 270 157 26% 34% 41% 48% 56% 64% 73% 80% 89% 95% 99% 100% 100% 5% 7% 10% 14% 21% 28% 38% 49% 61% 77% 92% 99% 100% 73 14 1 0
EARLY ALERT IMPACT Difference in differences approach Impute courses that would have used the early alert system Random forest model Predictors Enrollment cap Enrollment total Course level (100, 200, 300) Remedial Subject Time of day (letter code in section)
Retained Continuing Freshmen Advanced Transfers Treatment 0.122*** -0.216 -0.269 Post Treatment:Post Treatment:Post Age CreditsEnrolled CumulativeCredits CumulativeGPA Constant Observations Log Likelihood Akaike Inf. Crit. 0.097* 0.162*** -0.004* 0.078*** 0.005*** 0.498*** -2.302*** 51,680 -26,544.59 53,249.18 -1.307*** 0.407** 0.407** -0.014** 0.186*** 0.097*** -0.089 0.235 0.235 0.0003 0.127*** -0.005 - -0.162*** -2.723 10,788 -4,340.36 8,792.73 14.562 3,817 -2,055.05 4,266.09
