Insights into Disability Office Program Review Strategies

PROGRAM REVIEW OF THE DISABILITY OFFICE ANN KNETTLER, DELAWARE STATE UNIVERSITY JEAN ASHMORE, EMERITUS RICE UNIVERSITY AHEAD Conference 2019, Boston, MA

Concurrent Overview What is and rationale for Program Review Essential elements of Program Review Completing PR smoothly Using results to your advantage Continuous assessment and improvement Beginning to strategize for your own PR This session will include results from a recent member survey on PR. Full data will be available in session 5.13

What is Program Review Program Evaluation/Review a comprehensive internally or externally guided process which may include self-study, assessment data, input from DR stakeholders and institutional colleagues, benchmarking against other like institutions, and/or national data, concluding with an extensive report Definition used in recent AHEAD member survey on program review practices in Disability Resources

What PR is not A traditional annual report Typically a compilation of activities Includes numbers of individuals, contacts, programs, etc. Done yearly (generally) A public relations undertaking Quick and dirty PROGRAM REVIEW IS A DEEP OBJECTIVE DIVE INTO WHAT S WORKING AND WHAT S NOT IN THE DR OFFICE

Factors Influencing A Review The reasons behind the PR - internally or externally triggered accreditation driven trouble in DS paradise How the review will be undertaken - self study only (39%), external expert guided (22%), or hybrid Scope - complete office or a selected component such as alt format production Desired goals to achieve from and with a program review Timeframe available for planning and conducting review Whether this a one-off or regular undertaking Planned use of review results

Using Standards for PR? When embarking on a PR, what will guide your review, how will you evaluate, against what will you compare your results? Standards provide that framework - options AHEAD Program Standards & Performance Indicators CAS Disability Resources and Services (DRS) standards Has a companion Self Assessment Guide (SAG) - 43% used Student Learning Outcomes embedded - 55% included 2019 PR survey results show 37% used a combination of AHEAD and CAS for a review; 34% used CAS; 22% AHEAD

Standards Sections Most Cited in 2019 Member survey on PR practices AHEAD Professional Standards CAS Disability Resources Svcs Policies and Procedures 80% Program 64% Mission 55% Faculty/Staff Awareness 60% Diversity, Equity and Access 55% Program Administration and Evaluation 40%

Essentials for a Successful PR Plan early & well, expect delays, keep the faith Appoint a committed team, train together on evaluating evidence & resolving differences Establish a reasonable timeline 41% 2-3 months; 25% 4-5 months; 28% academic year Get buy-in from stakeholders Network with experienced colleagues on PR Conclude with comprehensive report, action plans

Objectives internal to DR What do you want to learn about your DR program? Strategic goal achievement from DR mission Stakeholder perspectives students, DR staff, faculty, administrators Internal & external comparisons Areas in need of improvement or creation Structural effectiveness does staffing/budget/overall organization support mission and program

What is your Mission? Do you have a mission statement that underpins the program? CAS Mission Statement for DRS Mission The primary mission of DRS is to provide leadership and facilitate equal access to all institutional opportunities for disabled students. CAS standards put forth three essential components of a DRS mission

3 Duties to accomplish DRS mission (per CAS Standards) Provide institution-wide advisement, consultation, and training on disability-related topics, including legal and regulatory compliance, universal design, and disability scholarship Collaborate with partners to identify and remove barriers to foster an all-inclusive campus Provide individual services and facilitate accommodations to students with disabilities 1. 2. 3.

Objectives external to DR What do you want others at your institution to learn about your department from the PR? DR s commitment to excellence, periodic assessment & use of professional standards Quantitative and comparative data used Rationale for requested resources Perspectives of those in the institutional community DR takes evaluation seriously and is an excellent campus partner on PR

Timeline things to consider Remember good work takes time Be conservative in establishing your expected timeline life happens Know that all participants are giving *extra* time to take on PR responsibility Time needs will expand with PR scope Evaluating a program component of DR (e.g. alt format)will be quicker than a full departmental review An academic year, semester?? Don t plan to be done quickly that may be self sabotaging with loss of support and participation

Sample Timeline By Feb. 15: Team Selection By March 15: Team Training Mar. 15-May 15: Compile and Review; Documentary Evidence May 15-June 30: Judging Performance July 30: Final Reports Due 66% 2-5 months; 28% academic year

Stakeholders Department s supervisor and her/his superior Must have buy-in, support for results & any resources the review may entail, e.g. admin time, research time, etc. DR staff members DR students Faculty Administrators Others specific to the institution When decisions made, timeline set, notify main campus partners of the upcoming PR

Your internal PR Team How many? Main Team 3 to 5 probably Who is leader? Probably the DR Director Develop a PR job description and PR outline that includes proposed benchmark dates Train team on reviewing evidence, ratings to ensure similar approaches by all members (Self Assessment Guide from CAS is great tool to help with this process)

Institutional Partners to Consider Engage institutional research/effectiveness staff as consultants on process, data analysis Colleagues in other student services departments who have completed a PR CAMPUS LABS if subscribe, the CAS DRS self assessment guide (SAG) may be available plus rubrics used by other depts, especially on student learning outcomes 28% reported that PR was completed by DR plus divisional or other institutional colleagues Your circle of buy-in is expanding!

Incorporating External Evaluators Following data collection, an external expert may be brought in to work with the PR team 22% A hybrid model self review augmented by Constituent surveys AHEAD has for low cost External expert reviews all collected information, visits campus, interviews stakeholders, evaluates data, provides recommendations, submits a comprehensive report External evaluators often valued by administration as they are viewed as objective

Executing your PR Define data needed Determine documentary evidence from which data can be obtained Print, program data, website information, survey results, campus stats, outcomes, year to year data Rate/evaluate your collected evidence Determine criteria for evaluating evidence to promote consistency across evaluators Remember the CAS SAG is a good tool Do you have data from program outcomes or student learning outcomes?

Data, data, and more data Data talks whereas stories are nice but not particularly supportive when appealing for resources Quantitative data vs. qualitative; use qualitative data sparingly perhaps exceptional events, occurrences In a PR you may want to incorporate data that: Compares salaries, staffing, case load to state or national data (AHEAD biennial member survey data - excellent sources) Contrasts grad rates, majors, GPAs of DR students to institutional data, regional, national Compares departmental processes (intake, appointment wait time, testing, etc) to other departments, national statistics Gives rating averages for effectiveness of print, web, policies as agreed on by PR team

Reporting your Results Produce professional report for history, sharing and staff review Present data/results with consistency for ease of interpretation by management/accreditors Develop a realistic Action Plan & include it in report Need review to be objective and to provide usable input for future directions (avoid being defensive) 63% report PR goals somewhat achieved; 35% fully This is a time to highlight strengths AND areas in need of improvement remember a negative is an opportunity

Using your PR Results Develop an executive summary of full report Use PR results to jump-start strategic planning Energize staff Adopt a continuous improvement perspective incorporate into your Action Plan PR can trigger adopting new perspectives and practices 83% report PR is internal tool for planning and to enhance practice perhaps Social Justice, learning outcomes

Why is self assessment & program review of DR so important? Gathering formative and summative data (metrics) is a primary means of: Understanding how our programs are working well (Program Outcomes) Evidencing impact on student learning (Student Learning Outcomes) Zeroing in on how to improve services* Justifying additional resources* Providing a common language for communication with colleagues and administration* * Rated highly as benefits of PR

Think through your hypothetical PR First steps with a partner or small group Answer initial questions Why will you undertake a program review? Identify 3 reasons What resources/data will be needed? Identify 5 resources to do this PR When is a good timeframe? Sketch out a proposed schedule for the PR

Executing your PR Answer these next questions Who will you bring into the process? Will this be internal to DR staff, with institutional partners (identify those), or include an external expert from your area, state or beyond? Will you use standards? If yes, which ones? If no, what will structure your PR? How will you proceed? Develop 3 steps to take to get the PR going

Concluding your PR Final questions How will your report be organized? Sketch out main chapters/sections of your report What are your hoped for outcomes? Identify 2 short term and 2 long term desired outcomes What do you think you ll do with your results? Suggest 3 ways you can expect to use the PR results

Contact Information Feel free to contact us about topics we ve discussed, professional standards, CAS Ann Knettler, Delaware State University aksmith@desu.edu AHEAD representative to CAS Jean Ashmore, Past President AHEAD jean@ahead.org AHEAD representative to CAS

AHEAD Survey citation Should you wish to cite findings from the 2019 AHEAD member survey on practices in program review, please feel free to do so using the following attribution/citation Ashmore, J., Knettler, A., Scott, S., & Washington, K. (2019). Survey of Program Evaluation and Review Practices by Disability Resource Professionals. Huntersville, NC. The Association on Higher Education and Disability

Program Review Steps Understand rationale for review Secure (or affirm) buy-in from superiors Determine scope for review Study professional standards to use Establish a workable timetable Appoint/solicit team members for PR Determine & secure evidence needed Train team on evaluating/rating evidence

More PR steps Establish how to engage stakeholders Determine how report of findings will be structured, written and by whom Incorporate an Action Plan with findings Distribute/present comprehensive report to key campus leaders do an Exec Summary Use results for strategic planning, resource requests, new programming, continuous improvement initiatives