Evaluation of Guided Pathways Implementation and Research Findings

Guided Pathways Implementation Evaluation Prepared by the Office of Planning, Research & Institutional Effectiveness For the Guided Pathways Steering Committee April 19, 2022

Research Questions Posed How well have we implemented each of the Guided Pathways essential practices? In particular, how are the Success Teams operating? Are they able to operate in a way that is consistent with the Steering Committee s vision from September 2021 and 2020? How are they interacting with the larger Interest Area groups? How are students experiencing Interest Areas and Success Teams? Where can we improve our implementation? Suggestions for improvement

Research Methods Survey of those most familiar with our Guided Pathways implementation (18 out of 28 invited responded) Steering Committee Members Interest Area and Success Team Members Group reflection exercise with each Success Team 1:1 interviews Survey of students via the Interest Area Canvas shells (102 students) PRIE Field Notes

Survey of Steering Committee, Interest Area and Success Team Members Results

CLARIFY THE PATH: Academic programs are clearly mapped out for students. Students know which courses they should take and in what sequence. Courses critical for success in each program and other key progress milestones are clearly identified. All this information is easily accessible on the college s website. How well have we implemented Program Mapper? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 6.1 Concerns about Program Mapper? We must establish a process for updating Program Maps that aligns with Curriculum Committee, the Catalog, and Degree Works (SEPs) update processes Ensure everyone knows about Program Maps especially Counselors, Outreach, Student Ambassadors; put them on course syllabi and on Discipline websites? Make sure Program Maps are consistent with our annual Course Offer (Availability) Pattern in the Catalog and that that gets updated regularly as well

Program Mapper Users and Page Views by Interest Area 2500 2280 2000 1530 1508 1500 Count 1075 1060 1000 721 695 512 500 0 Users Unique Pageviews Science and Health Art, Design, and Performance Business Human Behavior and Culture Page views for 4/17/21 to 4/17/22. Source: Google Analytics

Top 10 Program Maps by Users 300 251 232 250 223 200 174 173 171 169 165 158 157 153 153 136 150 123 117 113 107 99 97 92 100 50 0 Users Unique Pageviews Business Administration Radiologic Technology Computer Science Medical Assisting Interior Design Allied Health Biology Studio Arts Digital Art and Animation Engineering Page views for 4/17/21 to 4/17/22. Source: Google Analytics

HELPING STUDENTS CHOOSE AND ENTER A PATHWAY: Every new student is helped to explore career/college options, choose a program of study, and develop a full-program plan as soon as possible. How well have we implemented First Year Experience for new students? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 3.4 Concerns or suggestions about FYE? FYE does not exist yet. Some elements exist but they need to improve and be coordinated into something cohesive and well messaged to everyone. Annual planning for FYE must begin the prior fall with CRER and other cohorted classes identified and scheduled. Who runs it? The roles of Interest Area groups and Success Teams in FYE should be clearer in terms of what each does in terms of early messaging, community building, and support for on-boarding and orientation. Peer mentors should have a bigger role. Colts-Con should not just happen in August. It should take place over the first 6 weeks of the term. JAMS too.

KEEPING STUDENTS ON THE PATH: Success Teams monitor which program every student is in and how far along the student is toward completing the program requirements. Counselors, Retention Specialists and students are alerted when students are at risk of falling off their program plans and have policies and supports in place to intervene in ways that help students get back on track. How well have we implemented Success Teams? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 4.7 Concerns or suggestions about Success Teams? Some are working much better than others. When everyone is clear on their roles, and communicate well, they work better. Many still do not understand how Success Teams differ from the larger Interest Area groups. Having the Retention Specialists and Counselors in place is critical. This makes Early Alerts, progress reports, registration and SEP pushes work more smoothly and yield better results.

Early Alert Data 2021 - 2022 Course Count Reason SKY CAN CSM Total MEDA190 22 Academic Integrity 1 1 ACTG121 20 Attendance Concern 18 18 SPAN150 18 Failure Concern 5 61 10 76 ENGL100 15 Grade Concern 2 67 1 70 ENGL200 15 Missing/Late Assignment 3 28 8 39 HIST201 14 Other (Describe in Comments) 24 24 ENGL105 13 Participation Concern 34 34 LING200 13 Preparedness Concern 7 7 BUS.100 10 Recognition 12 12 SPAN110 9 Grand Total 10 252 19 281 Data Source: CRM (Student Success Link) includes CAN home campus students only (but enrollments are districtwide); 46 faculty participated in Early Alert.

KEEPING STUDENTS ON THE PATH: Ensure all students experience a sense of belonging to the College, faculty, staff and their peers. Create a sense of community that can support a student in choosing and persisting on an academic pathway so that students feel supported to complete their pathway. At Ca ada, we have created Interest Areas to implement this essential practice. How well have we implemented Interest Areas that foster a sense of belonging and community related to student s program of study? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 4.4 Concerns or suggestions about Interest Areas? The purpose of the Interest Area group is not clear to many nor who leads and how it relates to it s Success Team Creating and inviting all IA students and faculty (including adjuncts!) to join Canvas shells for each Interest Area is a great first step in helping create awareness. But many students and faculty still don t really know what they are or why Interest Areas are. The pandemic has impeded creating more of a sense of community related to the Interest Areas. The relationship between Divisions and Interest Areas is confusing. How can we fix this? Getting FYE and Career Exploration events better organized will also help.

KEEPING STUDENTS ON THE PATH: The college schedules courses to ensure students can take the courses they need when they need them, can plan their lives around school from one term to the next, and can complete their programs in as short a time as possible. How well have we implemented Course Scheduling to ensure students can take the courses they need when they need them? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 4.1 Concerns or suggestions about improving the course schedule? Very little progress has been made on this. Students are swirling more than ever (SKY, CSM), so we should coordinate with our sister colleges and align our schedule better with theirs. Students no longer see us as individual colleges, but take whatever suits their schedule. Students still report they can t get the classes they want (esp. on Tu/Th). Our 4-day/week schedule creates problems for students.

ENSURING THAT STUDENTS ARE LEARNING: Students have ample opportunity to apply and deepen knowledge and skills through projects, internships, co-ops, clinical placements, group projects outside of class, service learning, study abroad and other experiential learning activities that program faculty intentionally embed into coursework. How well have we implemented career exploration and work-based and project-based learning? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 5.0 Concerns or suggestions about providing students with a range of career exploration opportunities? Who leads on this? Interest Area Faculty? Career Center? Some combination? Needs staff support (like the temp PSC) to engage more employers, scale, and coordinate with other community partnership efforts. How do we continue to scale practices like the Speaker Series across the campus?

ENSURING THAT STUDENTS ARE LEARNING: The College helps students document their learning for employers and universities through portfolios and other means beyond transcripts. How well have we implemented helping students to document their learning for employers and universities through means beyond transcripts? 1 2 3 4 5 6 7 8 9 10 Min Max Mean 5.3 Concerns or suggestions about helping students to document their learning? Better partnership between the Transfer Center and Interest Area groups could help. Responses indicate that many do not see the utility of doing this.

Additional feedback on the redesign process What is working? Retention Specialists are awesome! Canvas shells have great potential. We ve accomplished a lot but it all needs to continue to evolve, improve and be maintained (e.g., program mapper). Having Ron and Margarita as dedicated staff to support the effort is critical and gives hope!

Additional feedback on the redesign process What is not working? Awareness and buy-in is still needed. Most students and faculty do not know what is going on or what it means. Clarity in roles and responsibilities is essential. Have we really redesigned anything? Feels like many processes are the same and we haven t really operationalized changes.

Anything else you would like to share about Guided Pathways at Ca ada College? We need more support from various services still, training on the CRM and maybe this could be done during the Summer including peer mentoring training. In the beginning, GP was a novel way to rethink the student experience from application to completion. It was and still is a heuristic of sorts--a way for us to speak about and think through how best to serve students. I think we've lost sight of this core mission. As with any initiative, GP's charm has faded. And we can hold on to it, but we may just need to recenter ourselves, assess what we think we have done, and plan for change. I have faith that we can get it together! I think having new staff is helping because their questions help to clarify our goals. Now, we just need to execute. I think there is a possibility to disrupt the problematic systems in place, but it will take a long time. The culture of this college has to change if this effort is to be truly impactful. The idea that key decisions and strategies can be determined by group consensus has never been realistic; the past two years have only served to prove that point over and over again. Fingers crossed for a strong president who is not afraid to lead, not afraid to make it clear that this is where we're going and it's time to get on the f---ing bus. I am happy to see that we have a position to support the work of Guided Pathways.

Success Team Group Reflections Results

Success Team Group Reflections: Operations Retention Specialists are leading Success Teams but hours may not be aligned with workload Clear roles are essential: Teams with a clearer understanding of what each member does (role) are working better (key members: retention specialist, counselor, faculty lead) The Community of Practice group is working well and is effective in helping align Interest Area Success Teams and Special Programs The CRM (Student Success Link) could be used better to case manage students, share notes, track outcomes training in its use is needed Regular meetings (bi-weekly) help a lot, but so does the ability to check in with each other via email (asynchronously) Counselor involvement in efforts to address Early Alerts and update SEPs is critical Canvas shells are working well to help students and faculty (even adjuncts) understand what the Interest Area is and what is going on The Interest Area weekly data reports from PRIE are essential

Success Team Group Reflections: Effectiveness More clarity regarding the roles and responsibilities of each member of the Success Team is needed. The organizational structure of Success Teams (who leads) is not the same in each IA. Does this matter? Who should lead? Who must be there? Who is optional? Who do the Success Teams Report to? What is the Deans role? Educating everyone about Success Teams is hard, but is beginning to work. Canvas Shells and Faculty Leads can really make the difference here. Faculty leads play a critical role as the bridge between the Success Team and all faculty including adjuncts. Otherwise, retention specialists do not get responses from faculty on students behalf. Retention Specialists play a critical role in the Success Team s ability to case manage student support, but they are also leading communications on behalf of the whole Interest Area. Is that clear? Retention Specialists can not be expected to organize events or other social occasions that help build community in the Interest Area It is hard to evaluate the effectiveness of Success Teams when the metrics or goals to measure implementation against/work towards are not clear

Student Survey Results

Results of Student Survey 102 Responses Human Behavior & Culture - 30% Art, Design & Performance - 22% Business - 18% Science & Health - 18% I'm not sure - 12% Most respondents have a limited understanding of Interest Areas 85% moderately familiar to not familiar at all Most respondents would like support connecting with faculty (64%) and other students (66%) Most respondents were not in a special program (52%) and were not in touch with their success team (62%)

Students generally connect in class and respondents prefer email correspondence 37 40 45 41 32 35 40 Percent Selecting 30 35 25 Percent Selecting 30 20 25 15 19 8 20 7 10 6 16 16 5 3 5 15 0 10 6 5 2 0

Summary: Key Take-aways While many of our flagship projects (Program Maps, Interest Areas, Success Teams, FYE) are not yet fully formed, the design and intent of each seems effective and needs more time to be effective The Interest Area groups do not really exist and there is a lack of awareness and faculty buy-in persists Retention Specialists are essential to the Success Teams and they need Counselors Faculty Coordinators provide a key touchpoint between Success Teams and Interest Area Faculty

Suggestions for Improvements Ensure that Interest Areas are a central organizing force for students and that all are clear on their purpose. The Steering Committee must get clearer more concrete about how to achieve these GP objectives (and how we will know if they are successful): Helping students enter and persist on clear academic pathways Aligning critical student support services with academic pathways Reaching all students especially part time students not already served by a special program Creating a sense of belonging and community for all (students, faculty and staff) as a means toward improving student persistence and completion The Steering Committee should clarify the roles and responsibilities of each person in an Interest Area group and its Success Team and help each group set goals and related metrics so they can achieve their purpose(s). Communication must be improved between the Steering Committee and the members of the Interest Area groups and their Success Teams; and between Interest Areas and students.