Exploring the Role of Electronic Health Records in Children's Behavioral Health Implementation
Electronic Health Records (EHR) have the potential to enhance quality and fidelity in children's behavioral health implementation. Health Information Technologies (HIT) serve as crucial implementation strategies, requiring careful evaluation and usability testing. This study examines how a tool like FidelityEHR can promote efficient, high-quality implementation and positive outcomes in the field. Supported by the National Institute of Mental Health, the research hypothesis focuses on the facilitation of efficiency, fidelity, and positive results through HIT. The NIMH Small Business Technology Transfer Study explores the impact of transitioning to digital platforms like FidelityEHR on service provision and client outcomes in children's behavioral health.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Can an Electronic Health Record (EHR) Promote Implementation Quality and Fidelity in Children s Behavioral Health? OR: The Potential for Positive Impact of a Tool Designed to Promote Implementation Quality Probably Depends at Least Somewhat on the Usability and Quality of Implementation of that Tool Eric J. Bruns, PhD Alyssa N. Hook, BS Elizabeth Parker, PhD Aaron R. Lyon, PhD University of Washington / Wraparound Evaluation & Research Team Kelly L. Hyde, PhD FidelityEHR 4th Biennial Society for Implementation Research Collaboration September 8, 2017 Wraparound Evaluation & Research Team 2815 Eastlake Avenue East Suite 200 Seattle, WA 98102 www.depts.washington.edu/wrapeval FidelityEHR 2100 Calle de la Vuelta, C-202 Santa Fe, NM 87505 www.fidelityehr.com
Acknowledgments FidelityEHR Founder & CEO Kelly L. Hyde, PhD Mission: To support empowerment, engagement and healthy outcomes through innovations in technology for families and communities. This study funded by the National Institute of Mental Health (R42-MH95516; PI Bruns)
Health Information Technologies (HIT) increasingly Function as Implementation Strategies (Lyon & Lewis, 2016) Some subtypes of HIT support service quality monitoring and can be classified within quality management implementation strategies (Powell et al., 2012, 2015) HIT simultaneously function as practitioner-facing implementation strategies and client-facing intervention components.
Health Information Technologies (HIT) increasingly Function as Implementation Strategies (Lyon & Lewis, 2016) Despite their potential, HIT often require their own strategic implementation supports to be successfully applied (Cohen, 2015; Ruud, 2015) Evaluation of implementation outcomes (e.g., acceptability, feasibility) for digital implementation strategies is critical to ensure their potential for impact Can be conceptualized using hybrid trial approaches that simultaneously consider effectiveness and implementation (Curran et al., 2012)
Research Hypothesis: Health Information Technology (HIT) can facilitate efficiency, fidelity, positive outcomes
NIMH Small Business Technology Transfer (STTR) Study Three phases: Phase 1: Program elements of FidelityEHR Phase 2: User Experience Testing:Determine if FidelityEHR is feasible and usable Phase 3: Determine if transitioning from paper to FidelityEHR impacts Wraparound implementation by providers and outcomes for youth and families
Theory of Positive Impact EHR Components Impact on Staff/Teams Paths to Family Outcomes Outcomes Information management: e.g., family, team, plan, providers, services, billing Fidelity support: e.g., Workflow pane, reminders, alerts, supervisor reports Standardized assessment: clinical alerts, treatment recommendations Feedback of information via dashboard reports on fidelity, services, progress, outcomes Supervisor, manager, administrative reports: e.g., services, costs, satisfaction, fidelity, outcomes, placements Availability of information Transparency and efficiency Better collaboration and teamwork Adherence to elements of high- fidelity Wraparound More frequent progress review Decision-making based on objective data More focused, directive, data- informed supervision Staff more satisfied and self-efficacious Admin/manager- level accountability Families retained in services Greater social support Greater progress and reduction in top problems Reduced youth emotional and behavioral problems Improved youth functioning Reduced out of home/ community placement Reduced costs to systems Goal clarity Team communication and consensus Better problem- solving Greater treatment alliance Family and team better engaged, hopeful, and satisfied Shorter self- correction cycles More effective treatment Reduced staff turnover
FidelityEHR Highlighted Features Secure, web-based login User friendly interface Contact/Progress Notes, Critical Incident Tracking Secure Messaging and Scheduling Report Builder for program and system decision support High-Fidelity Wraparound-based Plan of Care Including family vision, team mission, individualized needs statements, strategies linked to needs, etc Workflow pane customized to Wraparound Care Coordination Idiographic Progress Monitoring plus Standardized Assessment Builder
FidelityEHR Record Navigation and and Workflow
Research Aims Is FidelityEHR feasible, acceptable, and contextually appropriate in the real world of wraparound implementation? Comparing care coordinators randomly assigned to EHR vs. wraparound service as usual (SAU), how does FidelityEHR affect: Wraparound supervision? Wraparound practice? Teamwork and Alliance? Wraparound Fidelity? Parent Satisfaction?
Study Flow (CONSORT Diagram) Randomization at the Care Coordinator (Facilitator) level
Facilitator Demographics EHR SAU n = 18 (%) n = 13 (%) Male 9 (39%) 2 (15%) Female 11 (61%) 11 (85%) White 12 (67%) 10 (77%) African American 5 (28%) 2 (15%) Hispanic 0 1 (8%) Other 1 (6%) 0 No significant differences at baseline
RESULTS: User Experience and EHR Acceptability and Appropriateness
EHR usability ratings in marginal range but slowly increased over time The System Usability Scale (SUS) provides a quick and easy understanding of a user s subjective rating of a product s usability 12 facilitators completed the SUS over the course of one year (Site 1) 3 facilitators completed the SUS at 6 months only (Site 2) Site 2 (n=3; 1 UX assessment) 100 Acceptable usability 90 80 70 63.9 Marginal usability Low: 50-62 High: 63-70 60 58.3 54.6 50 48.1 50.6 40 Site 1 (n=12; 4 waves of UX data) 30 20 Unacceptable usability 10 0 Field-Based Testing Sept. 2015 (n=7) Implementation Wave 1 Feb. 2016 (n=12) Implementation Wave 2 June 2016 (n=12) Implementation Wave 3 Jan. 2017 (n=12) SOURCE: Bangor, A., Kortum, P., & Miller, J. (2009). An empirical evaluation of the system usability scale.
The distribution of scores indicate a range of opinions on usability The distribution indicates more than half of the users (61%) rated FidelityEHR with Marginal or Acceptable usability after 6 months of use Distribution of SUS Scores for both agencies 7 6 6 Number of Users 5 4 4 4 3 2 2 1 1 1 0 0 0 0-20 21-30 31-40 41-50 51-60 61-70 71-80 81-100 Marginal usability Low: 50-62 High: 63-70 Unacceptable usability Acceptable usability SUS Score (n=18)
Facilitators newly hired and trained on system report higher usability ratings Facilitators trained on FidelityEHR as part of their onboarding process report higher ratings for usability than facilitators in the research study 100 Acceptable usability 90 80 70 62.2 Marginal usability Low: 50-62 High: 63-70 56.8 60 52.2 50 40 30 20 Unacceptable usability 10 0 Newly-Hired Facilitators (n=43) EHR Group Facilitators (n=14) SAU Group Facilitators (n=8)
Staff report EHR aligns well with Wraparound service setting System Acceptability & Appropriateness scale (SAAS) gauges satisfaction, utility, and fit with service context of technology Staff (n=18) rate the degree to which they agree with each item at 6 months Appropriateness 72% 28% Fits with approach to service delivery 67% 33% Fits with treatment modality 61% 39% Compatible with service setting 39% 61% Comfort interacting with system Acceptability 6% 78% 17% Satisfied with content of system 17% 72% 11% Satisfied with ease of use 17% 67% 17% Satisfied with current version 0% 20% 40% 60% 80% 100% Not at all Moderately Extremely
Qualitative feedback: Strengths of the system Can quickly pull reports Can more easily make changes on the fly Better direction of where to go in supervision More aware of looking at needs and progress Great to be able to work remotely Families are better at understanding their outcomes Overall, love the system compared to the old one Keeps us focused on particular needs & outcomes, more organized with monitoring Tasks flow from strategies which link to needs System is overall good just need to work out kinks
Qualitative feedback: Needs for system improvement Contact logs take a lot of clicks and we use it the most Team meeting reminders aren t consistent Core assessments don t all display in supervision Plan of Care is too long can t just print one page (e.g., assessments) need POC report builder Tedious to add and delete strategies Can t sort contact logs by dates
Qualitative feedback: Change is hard, and transition to EHR must be done strategically First weeks were hard challenging to have conflicting answers from supervisors hard because things weren t sorted out Hard to learn all at once had a lot of workarounds Would have been better to have earlier trainings, and a better user s manual Took a long time to transition couldn t breathe til March EHR was added to the CAFAS, Suicidal Ideation/BX assessment, assessments asking families at EVERY team meeting how they do and how they feel, Protective Risk Factors Survey, etc. there is too much we are overwhelmed with requirements Starting to get the hang of it but study data will be impacted because we weren t using the system to its maximum capacity just trying to get by
RESULTS: Changes in Practice
Supervisors report small differences in supervision activities by group 15.7% After six months of FidelityEHR use, Wraparound Supervisors report how much time they spent on certain activities in supervision with Facilitators Reviewing Plans of Care and Skills Coaching & Training take up approximately one-third of supervision Reviewing Plans of Care 13.8% 12.3% Skills Coaching & Training 15.8% 10.9% Youth & Family Engagement 9.6% 10.1% Natural Support Engagement 9.6% 10.1% Reviewing Progress Toward Needs 6.9% 8.2% Facilitator Personal Support 5.8% 7.7% Administrative Tasks 11.5% 7.2% Crisis Assessment/Management 8.1% 6.6% Supervisory Relationship 8.1% 6.1% Case Conceptualization 5.8% 5.0% Facilitator's Professional Role 5.0% 0% 5% 10% 15% 20% EHR SAU
Supervisors report more time reviewing progress toward needs for EHR staff (p<.01) EHR group spends more time reviewing progress toward needs compared to the SAU group 12% 10.1% Percentage of time spent in supervision 9% 6.9% 6% 3% 0% Reviewing Progress Toward Needs EHR SAU
Facilitators report shifts in practice throughout the course of EHR use The Current Assessment Practice Evaluation Revised (CAPER) was administered to facilitators on a biweekly basis for eight months to assess the degree to which their practice was influenced by reviewing assessment data
Attitudes toward standardized measures improved at 6 months but only for SAU At 6-month follow-up, SAU facilitators reported more positive attitudes about using standardized measures and opinions about their reliability and validity compared to EHR group 5 Positive Attitudes 4 3.6 3.4 3.4 3.3 3.3 3.2 3.2 3.1 3.0 2.9 2.8 2.8 3 2 1 Negative Attitudes 0 BCJ PQ PC BCJ PQ* PC* EHR (n = 18) SAU (n = 13) Baseline 6 Months * Indicates item is significantly different; p<.05
Fidelity to Wraparound Caregivers completed the WFI-EZ after four months of Wraparound services No difference found for total fidelity Sig difference in favor of EHR found for Strength/Family Driven 100 90 * 79.7 77.6 80 75.3 74.7 74.4 73.8 73.6 72.0 71.7 71.4 69.7 69.4 68.8 67.8 66.1 65.6 70 62.4 Fidelity Score 57.9 60 50 40 30 20 10 0 Effective Teamwork Natural Supports Needs-Based Outcomes-Based Strength & Family Driven Total Fidelity EHR (n=42) SAU (n=23) National Mean
Facilitator satisfaction with Wraparound practice is high Adapted Clinician Satisfaction Index measured facilitators feelings about usefulness and effectiveness of Wraparound with enrolled families Both groups report high satisfaction; no significant within- or between- group differences over time 70 High 62.15 Satisfaction 59.08 58.29 58.29 60 50 40 30 20 10 Low Satisfaction 0 EHR (n=17) SAU (n=13) Baseline 6 Months
RESULTS: Impact on Youth & Family Experiences
No significant differences in Caregiver Satisfaction Caregiver satisfaction with services at approximately the mean for WFI-EZ sites nationally 2.00 1.4 1.4 High 1.2 1.2 1.2 Satisfaction 1.2 1.1 1.1 1.0 1.0 0.9 0.9 1.00 0.00 C1. Satisfied with Wraparound C2. Satisfied with youth's progress C3. Family made progress toward needs C4. More confident about ability to care for youth -1.00 Low Satisfaction -2.00 EHR SAU National Mean
Caregivers in both groups report positive working alliance with facilitators Working Alliance Inventory (WAI) quantifies the degree to which team members work collaboratively and connect emotionally Caregiver ratings very high; no between- or within-group differences 26.6 28 26.0 High Alliance 25.7 25.4 24.4 24.2 23.8 23.6 24 20 16 12 8 4 Low Alliance 0 Goal Task Bond Total Working Alliance EHR (n=42) SAU (n=23) SOURCE: Hanson, W. E., Curry, K. T., & Bandalos, D. L. (2002). Reliability generalization of working alliance inventory scale scores.
Caregivers report a positive team climate for both groups The Team Climate Inventory (TCI) assesses team interactions and performance with items such as, We have a we are in it together attitude Both groups report positive team climate; no between- or within-group differences Max score: 20 20 Positive Team Climate 18.0 17.6 17.3 17.1 Max score: 15 15 13.1 13.0 13.0 12.9 10 5 Negative Team Climate 0 Vision Participative Safety Task Orientation Support for Innovation EHR (n=42) SAU (n=23)
Successful EHR Implementation is becoming a science in and of itself Studies of successful EHR implementation have consistently recognized the importance of thoughtful planning and training in the implementation process: Timing training to coincide with implementation Targeting training to users needs Providing knowledgeable on-site support
Limitations and Study Challenges Grant timeline required rapid training and implementation cycles Staff-level randomization within supervisors/ organizations caused disruptions to routines Supervisors having to supervise differently depending on staff Staff not able to all support one another System continued to be improved throughout study based on feedback
Discussion: User Experiences Staff report EHR aligns with Wraparound service setting Marginal usability reported overall User opinions ranged from low to high Typical patter of eager adopters vs laggers Staff saw strengths of the EHR, but also experienced multiple kinks during study to be addressed by development team
Discussion: User Experiences User experiences affected by study enrollment Facilitators trained on EHR as part of their job and/or trained during onboarding more satisfied Those who had to change practice and/or do different things from their colleagues less satisfied Usability scores increasing over time System improvements in response to feedback On the job learning made things easier
User experience over time during EHR adoption parallels typical Big Change trends Confidence and Usability starts high, typically declines, then increases again Adapted from Domenz, Bob, (2014) Leading Organizations Through Big Change , ABF Journal, 12(6)
User experience over time during EHR adoption parallels typical Big Change trends Confidence and Usability starts high, typically declines, then increases again 100 80 63.9 60 54.6 50.6 40 48.1 20 0 Field-Based Testing Implementation Wave 1 Implementation Wave 2 Implementation Wave 3
Discussion: Impact on Practice & Implementation Few significant findings: EHR group spends more supervision time reviewing progress toward needs EHR group had marginally better fidelity in one area (Strengths and Family Driven) Both groups demonstrated significantly improved use of assessment and feedback Side effect of investment in EHR agency-wide in these sites? SAU facilitators report more positive opinions about using standardized measures at 6 months
Discussion: Impact on Practice & Implementation No differences on caregiver satisfaction, working alliance, team climate Still analyzing youth emotional/behavioral outcomes and fidelity as assessed by document reviews
Implications Rigorous study provided opportunity for substantial improvements FidelityEHR System Staff viewed system as appropriate to wraparound context, but change was hard and improvements were needed Modest but positive shifts in some proximal outcomes (supervision, use of data, fidelity) and lack of negative impact on satisfaction, teamwork, staff job satisfaction could be viewed favorably given the challenges Wraparound-specific EHR in wraparound worthy of continued development and research
Discussion: Next Steps Complete analysis on youth and family outcomes More rigorous study with: Updated FidelityEHR system featuring revamped responsive design More time / resources for implementation support Longer follow-up
For more information: wrapeval@uw.edu www.wrapinfo.org info@FidelityEHR.com www.FidelityEHR.com Wraparound Evaluation & Research Team 2815 Eastlake Avenue East Suite 200 Seattle, WA 98102 www.depts.washington.edu/wrapeval FidelityEHR 2100 Calle de la Vuelta, C-202 Santa Fe, NM 87505 www.fidelityehr.com
