Developing an Effective Outcome Monitoring System for Program Performance
Explore the process of establishing an outcome monitoring system for program performance, focusing on key questions, facilitators, barriers, and target audiences. Learn how to simplify the system, identify goals, use standardized measures, and provide timely feedback to enhance program effectiveness and reporting efficiency.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
State Data Systems to Track Program Performance State Data Systems to Track Program Performance Gary Bond Westat ASPIRE Leadership Training Sponsored by Office of Disability Employment Policy U.S. Department of Labor April 20, 2023
Starting Out: Initial Questions Starting Out: Initial Questions Why develop an IPS outcome monitoring system? What do we need to get started? What outcomes do we collect? Who collects the data? Who compiles the data into a data base? Who creates reports? What are the contents of the various reports? Who gets the reports and how often?
Facilitators and Barriers to Outcome Monitoring (Marty, 2008) Local Level State Level Agency leadership supports External reporting requirements Agency champion with know-how Technical assistance from experts Facilitators Easy-to-use method to collect data Available database Outcomes easy to measure No desire to collect data State data systems not user friendly Tracking outcomes seen as too hard Outcomes poorly defined Barriers Agency does not monitor outcomes State systems don't collect right data or collects too much data No leadership support Lack of resources State does not produce timely reports
Developing an IPS Outcome Monitoring System Keep it simple. Focus on a few main goals. Identify your target audiences. Start at your answer and work backwards. Use standardized measures whenever possible. Use two-track data collection: Internal=detailed; External=program-level In most cases, IPS team leader/local data person compiles data. In many cases, state data person compiles data and produces reports. Give frequent, timely, nonpunitive, individualized feedback to IPS team. Produce timely state reports synchronized to external deadlines. Summarize data in simple tables. Add graphs to help users visualize. Compare your data to normed data.
Who are your target audiences? At local level: IPS team and local agency Provide data at state level For developing and executing state plans For explaining IPS to managed care organizations and service providers who have not yet adopted IPS For grant applications For justifying IPS to other state agencies For convincing legislators to provide funding For communicating with the public
Planning Outcome Reports: Graphs and Summaries Knowing what you want to report helps you decide what to collect Sample questions for designing graphs: What are the trends over time? How does my state or program compare to benchmarks? At state level, how do programs compare with each other?
Minnesotas Annual IPS Report for 2022 Client Characteristics (Courtney, 2023)
Minnesotas Annual IPS Report for 2022 Outcomes (Courtney, 2023) Courtney, C. (2022). Individual Placement and Support for persons with serious mental illness in Minnesota. Report to the legislature as required by Minn. Stat. 268a.15. In: Minnesota Employment and Economic Development, Vocational Rehabilitation Services. Available at: https://www.lrl.mn.gov/docs/2021/mandated/210103.pdf.
Graph Showing Employment Rates Over Time (Fictitious Data) Number of IPS Clients Employed and Served in State xxx During 2022 3000 2614 2560 2494 2500 2000 1500 1310 1123 1000 806 500 0 Quarter 1 Quarter 2 Quarter 3 Employed 806 1123 1310 Served 2494 2614 2560 Employed Served Employment Rates: Q1 = 32%, Q2 = 43%, Q3 = 51%
Employment rate for last quarter of 2021: 46.3% (exceeds 41% benchmark for good outcome) 10
State level data: Number of clients served by quarter State-level data: Percentage of IPS clients working during last three quarters Other possible graphs: Number of new enrollees each quarter Number of terminations each quarter
Data Collection/Data Reporting Relationship Pitfalls Collecting data that are never compiled Collecting data that you may want to use in the future Reporting outcomes that are out of date Producing reports that are data dumps that do not adequately summarize information
Suggested Outcome Measures For IPS supervision IPS team leader usually maintains a spreadsheet Demographic sheet (pull from electronic record?) Milestone sheet, including: Date enrolled Date first employer contact Date first job start Date job ending first job start Date terminated Job start sheet Specifics of job (wage, hours worked, milestones achieved) For external audiences Quarterly employment form data aggregated quarterly
Outcome Measurement for Local Teams IPS team leader maintains a spreadsheet with two sheets: Demographic sheet (pull from electronic record?) Milestone sheet, including: Date enrolled Date first employer contact Date first job start Date job ending first job start Date terminated from IPS Job start details: Specifics of job (wage, hours worked, milestones achieved) IPS specialist assigned to client Spreadsheet is cumulative record, contains full record of job starts and endings used for IPS supervision, VR billing, and other purposes
Outcome Measurement for External Reporting One pragmatic strategy: Snapshot of status for a particular reporting period (for example, quarterly) Quarterly employment rate: Number of clients competitively employed at any time during last 90 days divided by: Number of clients on IPS caseload at any time during last 90 days Advantages: Easy to compile, benchmarked Disadvantages: Does not account for length of time enrolled in IPS, length of employment, etc.
IPS Supported Employment Sites - Quarterly Outcomes Report for the IPS Learning Community Report Period (check one): January March 2018 April June 2018 Agency Name: Agency Person Reporting: Date: Total Number of People on Caseload of IPS Supported Employment Staff. Include total number of clients that are on the assigned caseload of the IPS supported employment staff at anytime during the reporting quarter. Only include those people (unduplicated) who received at least one employment service. Number of People (unduplicated) from IPS Supported Employment Caseload Working Integrated Competitive Employment at Anytime During the Quarter.Integrated competitive employment is defined as a community-based job that pays at least minimum wage, is available to any person, belongs to the worker and does not have time limits determined by the rehabilitation/mental health agency. Number of People on IPS Supported Employment Caseload Enrolled in Education Programs During this Quarter. Education is defined as a credit-bearing educational program --- such as a technical school, college for which the person will receive documentation upon completion. Include GED prep classes. Number of People Working Successfully in Integrated Competitive Employment who Transitioned off the IPS Supported Employment Caseload during this Quarter. July September 2018 October December 2018 Number of New Enrollees Admitted to the IPS Supported Employment Program During this Reporting Quarter. This number is a subset of total number served on IPS supported employment caseload this quarter. Number of New Job Starts for All IPS Supported Employment Participants During the Quarter. Include all job starts. For example, one person starts three new jobs, which equals three job starts. Number of People on IPS Supported Employment Caseload who Enrolled in Education Programs During this Quarter. This number will be a subset of the total number enrolled in a credit-bearing education program this quarter. Number of People Not Working Who Transition Off the IPS Supported Employment Caseload during the Quarter. Staffing: Complete this section for the majority (over half) of the quarter. Number of People who are Employment Specialists with an IPS Caseload (excluding the supervisor). Total FTE Employment Specialists (excluding the supervisor) with an IPS Caseload - (FTE = full-time equivalent). Include full-time and part-time positions. For example, 2 employment specialists working 20 hrs/week = 1 FTE. Number of IPS Clients on Supervisor s Caseload (if any). January 26, 2018 Note: Individual Placement and Support (IPS) is the evidence-based approach to supported employment
Benchmarks for Quarterly Competitive Employment Outcomes in 151 IPS Programs 17 Benchmark for good outcome: 41%
Summary From the standpoint of ASPIRE, state leaders, and IPS programs, the goal is to have procedures that produce: Statistics on competitive employment outcomes for IPS clients The employment measures should be: Easy to collect and to understand State data systems are often cumbersome. A local agency s electronic medical record may be as well. In may cases it s best to independently collect and compile most IPS data at site level. IPS data collected at the local level can be aggregated and transmitted to a data monitor at the state level. But every state is different. State-level portals for data collection work well in some states.
