Independent Living Technical Assistance Evaluation
In this evaluation, the Independent Living Technical Assistance provided by AoD is examined to understand its effectiveness in supporting CILs, SILCs, and DSEs, aiming to enhance services for individuals with disabilities. Research questions, project overview, and key findings regarding TA strategy are discussed with insights for improvement.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Independent Living Technical Assistance Evaluation Discussion of Results Jennifer Howard, PhD Dustin Williams, MA RTI International September 21, 2023
Discussion Agenda Study Background Introduction Research Questions Project Overview Results by Research Question Methods Key Informant Interviews Document Review Literature Review Program Performance Report Data Analysis TA Survey Conclusions 2
Introduction The Administration on Disabilities (AoD) provides grants to support independent living (IL) services through the IL network which includes CILs, SILCs, and DSEs as part of its mission to give individuals with disabilities of all ages the opportunities, tools, and supports to lead the lives of their choice in their community. Under two cooperative agreements with AoD, the Independent Living Research Utilization (ILRU) and its partners provide technical assistance (TA) to support the CILs, SILCs, and DSEs. AoD contracted with RTI International to evaluate the TA strategy and how it might be improved to better meet the needs of IL grantees. 4
Research Questions 1. What are the elements of effective program TA? 2. What do the DSEs, SILCs, and CILs get with regard to TA? 3. What are DSEs , SILCs , and CILs needs when it comes to TA? 4. How is the current approach to TA meeting DSEs , SILCs , and CILs needs? 5. How is the TA not meeting their needs? What is additionally needed? 6. What are AoD s TA needs and priorities for the DSE, SILC, and CIL programs? 7. How is the current TA meeting AoD s needs? 8. What are the lessons learned that might be applicable to other AoD programs? 5
Project Overview To answer the research questions, we did the following: Interviewed IL grantees, TA providers, and AoD staff to give us information to help us develop survey questions Reviewed available literature and resources from various organizations about what works well in TA Reviewed documents to better understand the current TA approach Analyzed IL grantee PPR data Surveyed grantees about their TA experiences 6
WHAT ARE THE ELEMENTS OF EFFECTIVE PROGRAM TA? 8
What are the elements of effective program TA? Key Informant Interview (KII) Results What makes TA effective according to KII participants? TA provider is credible Information available is timely Information balances specificity and generality TA recipients are informed of TA availability TA is offered in a variety of formats Questions and comments are kept confidential Teaching style is engaging 9
What are the elements of effective program TA? Literature Review Results What are elements of effective TA according to the literature? Relationship between TA provider and recipient/TA provider is a SME Baseline needs assessment/establishing TA goals Accessibility/delivering TA in multiple formats Tailoring TA to recipient s needs Flexibility of TA provider Knowledge of the TA recipients context Working with staff at various levels within the TA recipient organization 10
TA Survey Participation RTI fielded the survey in June August 2023 and received 234 responses. Organization Type Number of Survey Responses Organizations on Frame Response Rate (%) CIL 180 286 62.6 SILC 27 56 44.6 DSE 27 55 50.1 Total 234 397 58.9 11
What are the elements of effective program TA? Survey Results What would make TA most effective for your organization? All Respondents (%) Convenience 74.8 Relevance 72.2 Personalized Content 50.9 Accessibility 50.4 Timeliness 44.4 Other 3.4 12
WHAT DO THE DSES, SILCS, AND CILS GET WITH REGARD TO TA? 13
What do the DSEs, SILCs, and CILs get with regard to TA? KII Results/Document Review Results Who provides the TA and what types of TA are received according to the KII and document review results? TA Providers Used ILRU APRIL and NCIL Types of TA Received Webinars & courses/rapid courses Peer-to-peer Conferences Intensive support Products (e.g., newsletters, directory updates) 14
What do the DSEs, SILCs, and CILs get with regard to TA? Survey Results (1 of 5) Number of Survey Responses Received ILRU TA N (%) Received non- ILRU TA N (%) Grantee Type CIL 180 130 (72.2) 108 (60.0) DSE 27 12 (44.4) 8 (29.6) SILC 27 26 (96.3) 24 (88.9) Total 234 168 (71.8) 140 (59.8) 15
What do the DSEs, SILCs, and CILs get with regard to TA? Survey Results (2 of 5) TA Type % Receiving TA Type Online, live trainings or webinars 91.9 On-demand (recorded) trainings or webinars 91.5 On-demand publications and resources 85.0 Other 82.1 Peer Group Technical Assistance Discussions 73.1 In-person trainings or webinars 67.5 One-on-one technical assistance or intensive support 65.8 Technical Assistance Office Hours 63.2 Peer-to-Peer Mentoring 62.4 16
What do the DSEs, SILCs, and CILs get with regard to TA? Survey Results (3 of 5) TA Topic % Receiving TA on Topic Other 91.9 COVID-19 and emergency preparedness resources 80.8 Disability, diversity, intersectionality, and outreach to underserved groups 79.5 Transition of youth 77.8 Independent living history and philosophy 76.1 Home and community-based services 74.4 Community and institutional transition or diversion 73.9 Individual or systems advocacy 73.9 SILC and DSE roles and responsibilities; State Plan for Independent Living 73.9 17
What do the DSEs, SILCs, and CILs get with regard to TA? Survey Results (4 of 5) TA Topic % Receiving TA on Topic Peer counseling and peer support 69.7 Performance measurement, program performance reporting, compliance reporting, and evaluation/outcome measurement 69.7 Transportation 68.4 Expanding housing options 67.5 Independent living skills training 67.5 Information and referral support 65.8 Succession planning, staff development, benefits, and remote working 65.8 18
What do the DSEs, SILCs, and CILs get with regard to TA? Survey Results (5 of 5) TA Topic % Receiving TA on Topic Board training and development 65.8 Business acumen, financial management, fundraising and grant writing 65.4 Consumer information files or consumer service records (for example, release authorizations, goal forms, and independent living plans) 64.5 Assistive device or technology support 61.5 19
WHAT ARE DSES, SILCS, AND CILS NEEDS WHEN IT COMES TO TA? 20
What are DSEs, SILCs, and CILs needs when it comes to TA? KII Results What are the IL Grantees TA needs according to KII participants? Programmatic Administrative or management New or upcoming regulation guidance IL philosophy or history Technology Innovative or hot topics Post-COVID 21
What are DSEs, SILCs, and CILs needs when it comes to TA? PPR TA Needs Data Analysis Results (1 of 2) What are the CILs reported TA needs? 2019 (N = 347) 2020 (N = 339) 2021 (N = 335) 1. Financial: Resource Development - Fee- for-Service Approaches (60) Advocacy/Leadership Development - Systems Advocacy (60) Innovative Programs - Best Practices (49) Data Collecting and Reporting - PPR/704 Reports (45) Financial: Resource Development - Diversification of Funding Base (42) Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (42) 1. Financial: Resource Development - Fee-for- Service Approaches (55) Innovative Programs - Best Practices (43) Financial: Resource Development - Diversification of Funding Base (42) Advocacy/Leadership Development - Systems Advocacy (42) Data Collecting and Reporting: PPR/704 Reports (38) 1. Financial: Resource Development - Fee-for- Service Approaches (50) Financial: Resource Development - Diversification of Funding Base (48) Advocacy/Leadership Development - Systems Advocacy (43) Innovative Programs - Best Practices (32) Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (32) 2. 2. 2. 3. 3. 3. 4. 4. 5. 4. 5. 5. 6. 22
What are DSEs, SILCs, and CILs needs when it comes to TA? PPR TA Needs Data Analysis Results (2 of 2) What are the SILCs reported TA needs? 2019 (N = 54) 2020 (N = 54) 2021 (N = 51) 1. Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (8) Advocacy/Leadership Development - Community/Grassroots Organizing (7) Financial: Resource Development - Fund- Raising Events of Statewide Campaigns (7) SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6) Evaluation - Community Needs Assessment (6) 1. Advocacy/Leadership Development - Systems Advocacy (8) Advocacy/Leadership Development - Community/Grassroots Organizing (7) Advocacy/Leadership Development - General Overview (7) Innovative Programs - Best Practices (6) SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6) Financial: Resource Development - Diversification of Funding Base (6) Applicable Laws - Medicaid/Medicare/PAS/waivers/ long-term care (6) 1. Advocacy/Leadership Development - Systems Advocacy (10) SILC Roles/Relationship to CILs - Development of State Plan for Independent Living (9) Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (7) Advocacy/Leadership Dev elopment - Legislative Process (6) SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6) 2. 2. 2. 3. 3. 3. 4. 4. 4. 5. 5. 6. 5. 7. 23
HOW IS THE CURRENT APPROACH TO TA MEETING DSES , SILCS , AND CILS NEEDS? 24
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? KII Results Overall Experience with TA Positive, especially for newer grantee staff Mixed experiences with timeliness of TA topics Experience Requesting TA Paula is well known as the go-to person for TA requests at ILRU Unclear whether IL grantees know how to request TA Grantee organizations refer members to ILRU for TA ILRU responds quickly to TA requests 25
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? Survey Results (1 of 5) How satisfied was your organization with the types of TA received? Neither Satisfied or Dissatisfied Very Very Satisfied TA Type Dissatisfied Satisfied Dissatisfied Peer-to-peer mentoring 2.9% 1.9% 7.8% 44.7% 42.7% 2.4% 1.2% 4.8% TA office hours 48.8% 42.9% Peer group TA discussions 3.1% 2.3% 7.7% 46.9% 40.0% 26
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? Survey Results (2 of 5) How satisfied was your organization with the types of TA received? Neither Satisfied or Dissatisfied Very Very Satisfied TA Type Dissatisfied Satisfied Dissatisfied One-on-one TA or intensive support 3.0% 2.0% 9.0% 41.0% 45.0% In-person trainings or webinars Online, live trainings or webinars 4.4% 0.9% 13.3% 46.0% 35.4% 2.8% 1.4% 6.9% 55.3% 33.6% 27
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? Survey Results (3 of 5) How satisfied was your organization with the types of TA received? Neither Satisfied or Dissatisfied Very Very Satisfied TA Type Dissatisfied Satisfied Dissatisfied On-demand (recorded) trainings or webinars On-demand publications and resources (e.g., fact sheets) 3.1% 1.0% 8.7% 52.8% 34.4% 2.8% 0.6% 7.7% 51.4% 37.5% 0.0% 0.0% 20.0% Other 20.0% 60.0% 28
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? Survey Results (4 of 5) Respondent rating of ILRU and non-ILRU TA in terms of helping organization fulfill its mission Very Good # Very Good % Grantee Type Poor # Poor % Fair # Fair % Good # Good % Excellent # Excellent % Total # 2 0.9 14 6.0 69 29.4 94 40.0 56 23.8 235 CIL 0 0.0 4 8.3 7 14.6 18 37.5 19 39.6 48 SILC 0 0.0 0 0.0 8 42.1 8 42.1 3 15.8 19 DSE All Grantees 2 0.7 18 6.0 84 27.8 120 39.7 78 25.8 302 29
How is the current approach to TA meeting DSEs, SILCs , and CILs needs? Survey Results (5 of 5) What are the benefits of TA for the organization? All Grantees (%) Improved operational efficiency 86.3 Improved performance on key outcomes 80.8 Improved financial performance 41.9 Improved outreach 41.9 Staff retention 39.7 Other 4.7 There are no benefits 1.7 30
HOW IS THE TA NOT MEETING THEIR NEEDS? WHAT IS ADDITIONALLY NEEDED? 31
How is the TA not meeting their needs? What is additionally needed? Survey Results (1 of 2) Barriers preventing organizations from using additional ILRU and non-ILRU TA CIL N (%) SILC N (%) DSE N (%) All Grantees N (%) 143 (60.1) 31 11 (4.6) 185 (77.7) No additional TA needed (13.0) Limited staff and could not make time for anyone to participate in TA 45 7 2 54 (18.9) (2.9) (0.8) (22.7) TA types did not occur frequently enough or at times my organization could attend 32 8 3 43 (13.4) (3.4) (1.3) (18.1) 27 5 2 34 Did not know about it (11.3) (2.1) (0.8) (14.3) 10 (4.2) 9 3 22 (9.2) Other (3.8) (1.3) 32
How is the TA not meeting their needs? What is additionally needed? Survey Results (2 of 2) SILC N (%) Barriers preventing organizations from using additional ILRU and non-ILRU TA CIL N (%) DSE N (%) All Grantees N (%) Past TA has not been appropriately tailored for my organization and was not relevant 11 (4.6) 3 1 15 (6.3) (1.3) (0.4) 5 2 1 8 TA was not timely (2.1) (0.8) (0.4) (3.4) 7 (2.9) 0 (0.0) 1 (0.4) 8 (3.4) TA was poor quality 3 0 1 4 (1.7) TA is not accessible for people with disabilities (1.3) (0.0) (0.4) 33
How is the TA not meeting their needs? What is additionally needed? KII Results Recommendations for TA improvements according to KII participants No need to change to TA (2) Direct a greater portion of available funds for direct support and peer-to-peer (2) Better identify grantees TA needs and desired approaches and tailor accordingly (2) Elevate the focus on disability (2) Require TA providers to work with national associations (1) Change and increase ILRU staff (1) Increase personal engagement and networking (1) Reduce TA information email volume (1) 34
METHODS 35
Key Informant Interview Methods Conducted 11 KIIs between October 2022 and January 2023 with representatives from AoD (2), CILs (3), SILCs (1), DSEs (2), and TA providers (3). Interviews covered the following topics: Types of TA received TA providers used Awareness of TA Experience with TA TA effectiveness TA needs Recommendations for improvements 36
Document Review Methods Document types reviewed and information pulled: Funding Opportunity Announcements (FOAs) TA requirements Performance reports time frame, performance metrics, and partners Contracts and related materials TA requirements and planned approach 37
Literature Review Methods Developed search strategies for both the peer-reviewed literature and organizations websites. Published in English since January 2012 Search terms technical assistance and service providers and disab* Search locations 7 databases Search locations 22 federal government agency, research organization, and service provider websites Team members reviewed the literature and determined that 40 publications met inclusion criteria. 38
Project Performance Report Data Analysis Methods Grantees were asked to select from 78 TA topics to answer the following: Please identify the CIL s and SILC s TA needs; the needs identified in this chart will guide the priorities set by ACL for the TA provided to CILs and SILCs. Choose up to 10 Priority Needs Rate items 1 10, with 1 being most important. Frequencies Number of grantees that responded to the question Number of grantees that reported a 1 3 for a TA topic 39
TA Survey Methods Developed a web survey about TA needs and effectiveness. Conducted six cognitive interviews and nine pilot surveys to test the survey. The full survey was distributed to 397 grantees, using a combination of email, mail, and phone outreach. Organizations were asked questions on: Their use or non-use of TA The benefits/utility of current TA types and topics Their TA needs Barriers to using TA Ways to improve future TA 40
Conclusions (1 of 2) There is a great deal of agreement in what constitutes effective TA. Grantees are getting TA from a mix of TA providers. Webinars are the most commonly used type of TA. Out of all the topics covered for TA received, the following topped the list: (1) COVID-19 and emergency preparedness, and (2) Disability, diversity, intersectionality, and outreach to underserved groups. These topics differed from the reported needs in the 2021 PPR, which indicated that the top TA needs focused on finances, including resource development related to Fee-for-Service approaches or diversification of funding base. These topics also differed from the reported needs by KII participants, which indicated that the top TA needs were related to programmatic, administrative or management, IL philosophy or history, technology, or post-COVID. Grantees are overall satisfied with the TA providers and the TA they provide. 41
Conclusions (2 of 2) Areas of opportunity may include: Providing TA offerings more frequently and with more flexibility to allow greater participation among grantees with limited staff availability Increasing grantees awareness of and engagement in TA Identifying grantees TA needs and tailoring TA accordingly 42
Thank you! 43
