Accreditation Requirements & Analysis for CME Programs
Accreditation criteria 11, 12, and 13 outlined by David Pieper focus on evaluating learner outcomes, analyzing program effectiveness, and implementing necessary improvements. The process involves assessing changes in competence, performance, and patient outcomes, aligning activities with the CME mission, and planning enhancements to meet educational goals. Examples and responses from a medical conference demonstrate how providers can evaluate and enhance their programs to ensure quality educational interventions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Criteria 11, 12 and 13 David Pieper (dpieper@med.wayne.edu) with help from Brooke Taylor and Rebecca Daniel
ACCME ACCME Accreditation Requirements and Descriptions Accreditation Requirements and Descriptions Criterion 11: The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions. Criterion 12 The provider gathers data or information and conducts a program based analysis on the degree to which the CME mission of the provider has been met through the conduct of CME activities/educational interventions. Criterion 13 The provider identifies, plans and implements the needed or desired changes in the overall program (eg, planners, teachers, infrastructure, methods, resources, facilities, interventions) that are required to improve on ability to meet the CME mission.
Criterion 11 Criterion 11 The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions.
Criterion 11: Example of Citation Criterion 11: Example of Citation The evidence presented in the self-study report and in the activities reviewed demonstrated that the provider evaluated learner satisfaction and whether learning objectives were met. The provider did not present data related to, or an analysis of, changes in physician competence, performance, and/or patient outcomes
Majority of WSU Activities Assess Competence Majority of WSU Activities Assess Competence
Example of Evaluation of Competence Example of Evaluation of Competence Based on what you learned in this session, will you make changes in your practice? Yes or No If yes, please describe the changes you intend to make What barriers to change do you anticipate? What strategies or mechanisms will you apply to overcome the barriers?
Example of Responses Example of Responses WSU/DMC WSU/DMC Antithrombosis Antithrombosis Conference BASED ON WHAT YOU LEARNED TODAY, WILL YOU MAKE CHANGES IN YOUR PRACTICE? YES: 101 NO (will keep the same): 21 Conference NO (Retired): 2 If Yes, please describe the changes you intend to make: will encourage our cardiologists to prescribe anticoagulation meds in stable non stented CAD patients no ASA without assessment of patient will look at CHADS2 VASC2 score along with CHADS establish TSOAC monitoring procedure duration of therapy of anticoagulants in first event DVT/PE => more than 6 months is beneficial making recommendations for candidates for anticoagulation therapy I intend to implant formed INR goals on referring physicians to eliminate unwarranted goals CHADS2VASC score initiated evaluate bridging protocol for warfarin interruptions for patients with old (> 612 months) PE/DVT and factor check Leiden (didn t realize this doesn t increase risk of recurrent VTE) more confident to defend the role of warfarin over NOAC. I have more understanding of that data recommend wider use of NOACs for VTE
Example of Responses Example of Responses WSU/DMC WSU/DMC Antithrombosis Antithrombosis Conference Conference What barriers to change do you anticipate? insurance prior authorizations and cost of new meds none DMC will implement changes none time away from clinic to discuss insurance coverage, or lack of patient preference to DOT and MD preference of DOT physician resistant/threat to autonomy physician pushback physician resistance to new agents education; physician buyin time cost of new anticoagulants (No GEQ) time and resources billing for service
Example of Responses Example of Responses WSU/DMC WSU/DMC Antithrombosis AntithrombosisConference Conference What strategies or mechanisms will you apply to overcome these barriers? follow policies DMC speak to leadership assistance programs (?) describe evidence support longer treatment of AC in first event of DVT/PE change referrals, enforce new rules persistence delegate to resident assignments refer programs to help patients pay for meds patient assistance programs use of EMR, resource allocation/shift, costavoidance justification, costbenefit justification present primary literature discussed/presented today change note template in chart small sections at a time (of learning information) back up change in studies from program
Case Conferences / Tumor Boards Case Conferences / Tumor Boards Example from WSU/DMC Tumor Board Example from WSU/DMC Tumor Board No Patient Identifiers Nodule Size Status or Comments Recommendations Nodular density 18mm RML & scarring; subcm LUL subpleural nodule Mediastinal lymphadenopathy with subcarinal 15 mm & R hilar 19 mm In the board's opinion these findings are: Consistant with malignancy. At this time it is recommended that patient is referred to CT surgery. New PET/CT done 1/30 for review. Case #1 In the board's opinion these findings are: Inconsistant with malignancy. At this time it is recommended that the patient continue surveillance per oncology. In the board's opinion these findings are: highly suspicious for malignancy. The board agrees with primary pulmonary team plan to refer patient to thoracic surgery and obtain PFTs. Hx NSCLC & melanoma. PET/CT done 2/4 for review. Case #2 10mm spiculated nodule PET/CT done 2/3 to review. PFT's scheduled for Pt at NH. Case #3 RUL mass In the board's opinion these findings are: Consistant with malignancy. The board recommends: the patient should be referred to CT surgery. 24x21mm & nodule 17x13mm; LUL nodule 10x7mm; RLL infiltrate Hx of Renal Cell CA. New CT from 1/27 to review. Case #4
Example from DMC VTE Prophylaxis Example from DMC VTE Prophylaxis Performance Improvement Project Performance Improvement Project DMC CY2013 Baseline Measure Name Target Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 CY YTD Tenet Clinical Operations Dashboard Stroke 89.66% 78 87 100.00% 66 66 88.89% 8 9 83.33% 5 6 96.72% 59 61 96.61% 57 59 86.11% 31 36 98.78% 81 82 94.83% 96.25% 77 80 98.39% 61 62 88.89% 8 9 100.00% 1 1 98.33% 59 60 94.23% 49 52 83.33% 30 36 100.00% 76 76 96.01% 92.94% 79 85 98.51% 66 67 100.00% 7 7 75.00% 6 8 96.88% 62 64 98.04% 50 51 90.48% 38 42 97.50% 78 80 95.54% Pediatric Outpatient Process Measure 96.10% 74 77 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 54 76 64 66 76 54 76 64 66 76 66.67% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 2 6 8 10 2 3 6 8 10 2 62.50% 100.00% 80.00% 100.00% 100.00% 100.00% 5 4 4 5 2 8 4 5 5 2 100.00% 97.37% 100.00% 100.00% 96.05% 47 74 62 64 73 47 76 62 64 76 100.00% 96.55% 97.96% 98.11% 100.00% 100.00% 100.00% 100.00% 100.00% 48 56 48 52 47 48 58 49 53 47 95.74% 93.88% 100.00% 95.65% 100.00% 45 46 29 44 49 47 49 29 46 49 94.81% 97.80% 100.00% 100.00% 100.00% 73 89 76 76 87 77 91 76 76 87 96.40% 97.39% 99.20% 99.25% 99.30% 97.00% 97 100 98.80% 82 83 100.00% 100.00% 82 82 98.86% 87 88 100.00% 100.00% 100.00% 63 70 63 70 97.35% 954 980 99.74% 754 756 95.71% 67 70 86.21% 50 58 98.07% 712 726 98.33% 588 598 94.71% 448 473 98.81% 910 921 97.84% 91 91 74 74 VTE Prophylaxis (STK1) 96.9% 69 69 48 48 53 53 55 55 Antithrombotics at DC (STK2) 99.2% 5 5 1 1 5 5 5 5 Anticoag Tx for Afib/Aflutter (STK3) 93.8% 75.00% 3 4 97.92% 47 48 100.00% 100.00% 4 4 93.88% 46 49 4 4 7 7 Thrombolytic Tx (STK4) 88.2% 98.95% 100% 100.00% 68 68 100.00% 51 51 Antithrombotics at by Day 2 (STK5) 97.8% 56 56 33 33 44 44 48 48 DC on Statin (STK6) 95.8% 95.45% 42 44 97.62% 82 84 98.80% 96.88% 31 32 100.00% 100.00% 100.00% 60 64 60 64 98.96% 99.07% 100.00% 100.00% 33 33 30 30 Stroke Education (STK8) 90.9% 68 68 Rehab Assessment (STK10) 98.1% 100.00% Composite Pending 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 14 24 25 52 71 33 14 24 25 52 71 33 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 14 24 25 52 71 33 14 24 25 52 71 33 100.00% 100.00% 96.00% 100.00% 100.00% 100.00% 14 24 24 52 71 32 14 24 25 52 71 32 100.00% 100.00% 98.67% 100.00% 100.00% 100.00% 100.00% 639 639 100.00% 639 639 98.11% 624 636 99.37% 46 46 83 83 138 138 71 71 46 46 36 36 CAC-Relievers for Inpt Asthma 100% 98.95% 46 46 83 83 138 138 96.32% 131 136 98.79% 71 71 46 46 36 36 CAC-Corticosteriods for Inpt Asthma 100% 100% 89.13% 41 46 96.38% 100.00% 83 83 100.00% 98.59% 70 71 99.53% 100.00% 100.00% 46 46 100.00% 100.00% Home Management Plan of Care (HMPC) Document Given to Pt./Caregiver - (CAC 3) 36 36 Pending Composite N/A
Other Evaluation Methods Other Evaluation Methods Survey Monkey 3 Month follow-up Audience Response Simulation Mock scenarios Hands-on performance
C11 Example from ACCME Focus Group Reports (for sessions held March 2015June 2015) Facilitator: Dr. Dre 7 participants from conference sessions (including 3 attendings, 2 nurse leaders, 1 resident, and 1 genetics counselor) All representatives shared new strategies for managing early diagnosis and consultation regarding surgery Identified systembased obstacles for implementing some strategies (e.g. scheduling delays for genetic counselors, poor communication between clinical and social work staff). Shared that it would be helpful to share this feedback with the group as a whole. (for sessions held July 2015September 2015) Facilitator: Ms. Black 8 participants (2 surgeons, 1 medical records, chief quality officer, 1 attending, 2 nurse practitioners, and 1 medical scribe) Sessions on medical informatics were interesting, but didn t provide actionable next steps. Participants didn t know how to apply the information. Suggestions to include a tool (maybe a checklist) for future sessions (end of year) Facilitator: Dr. Long 9 participants (2 surgical staff, 1 pathology attending, 1 path student, 1 case manager, quality manager, 1 pharmacist, and 2 oncology residents) The coffee is terrible. Can we go back to having Starbucks?! Participants said it was helpful to have the team participate this past quarter had great discussions how to implement changes to our shared processes. Participants appreciated seeing their data Great to see how survivor data is improving. and suggest that we should bring more data into conferences to provide feedback on how we re improving.
Beaumont Health Criterion 11 Evaluate Changes in Competence Immediate post-conference survey: Based on information received at this CME activity, what primary action will you take to change your practice? Seek additional information Implement new information/skill(s) into my practice Do nothing, current practice reflects activity recommendations Do nothing, system barriers prevent me from changing my practice Other, specify Give an example of at least 1 action you plan to take to change your practice:
Beaumont Health Example of Responses
Beaumont Health Criterion 11 Evaluate Changes in Performance 6 Week follow-up survey:
Beaumont Health Example of Reponses
Beaumont Health Criterion 11 Evaluate Changes in Performance Have you noticed an improvement or change in your patient outcomes since implementing this change in practice? Yes please describe No Have not tracked
Beaumont Health Example of Reponses
Beamont Health Examples for C11 Competence what will you do? Performance what did you do?
Beaumont Health Criterion 11: Activity Analysis of Data 60% 50% 40% 62% 30% 60% 59% 20% 36% 27% 26% 10% 5% 0% Seek Additional Information Implement New Information/skill(s) into Practice Will Not Make Changes due to System Barriers Will Not Make Changes - Current Practice Reflects Recommendations Immediate Post-Conference Evaluation 60-Day Post-Conference Follow-Up Evaluation
Beaumont Health Beaumont Health Criterion 11: Activity Analysis of Data Criterion 11: Activity Analysis of Data Of the learners who successfully implemented a change in practice, 50% indicated they had noticed an improvement in their patient outcomes. Patients happier Shortened wait times Better outcomes with bladder control Improved headache control in selected patients
C11 Example from SJMAA Dermatology RSS Yes Areas of assessment No CME activity met all objectives 100 Material was relevant to the learner s practice 100 Learner intends on utilizing content of activity to make a practice change 98 2 Presentation was free of commercial bias 100 Learner feels this was a valuable activity and would attend again 100 Ways in which this activity changed competence, performance or patient outcomes Increased knowledge base to take care of cong nevi, eruptive KA, Improved ability to utilize broad approach and perspective in management, awareness of morphine metabolism to enable better dosing, improve outcomes in NICU, facilitate early diagnosis of histiocytosis, helped improve my decision making with melanoma management. Topical dermatology, excellent review Main discussion points during the presentation Comments or suggestions for this educational activity Made me smart!
Evaluation Challenges Evaluation Challenges Low percent of evaluations returned Meaningless responses ( All 4 syndrome ) Information not summarized or utilized Speakers need individual ratings for P&T
Criterion 12 Based on data and information gathered, provide your program based analysis on the degree to which the expected results component of your CME mission has been met through the conduct of your CME activities/educational interventions.
WSU Reached Target Audience (Attendance Data) 35000 30000 25000 20000 15000 10000 5000 0 2010 2011 2012 2013 2014 Blue bars = Physicians Open bars= NonPhysicians
WSU: Produced the types of activities stated in WSU: Produced the types of activities stated in the mission the mission:
DMC: Expected Results DMC CY2013 Baseline Measure Name Target Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 CY YTD Tenet Clinical Operations Dashboard Stroke 89.66% 78 87 100.00% 66 66 88.89% 8 9 83.33% 5 6 96.72% 59 61 96.61% 57 59 86.11% 31 36 98.78% 81 82 94.83% 96.25% 77 80 98.39% 61 62 88.89% 8 9 100.00% 1 1 98.33% 59 60 94.23% 49 52 83.33% 30 36 100.00% 76 76 96.01% 92.94% 79 85 98.51% 66 67 100.00% 7 7 75.00% 6 8 96.88% 62 64 98.04% 50 51 90.48% 38 42 97.50% 78 80 95.54% Pediatric Outpatient Process Measure 96.10% 74 77 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 54 76 64 66 76 54 76 64 66 76 66.67% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 2 6 8 10 2 3 6 8 10 2 62.50% 100.00% 80.00% 100.00% 100.00% 100.00% 5 4 4 5 2 8 4 5 5 2 100.00% 97.37% 100.00% 100.00% 96.05% 47 74 62 64 73 47 76 62 64 76 100.00% 96.55% 97.96% 98.11% 100.00% 100.00% 100.00% 100.00% 100.00% 48 56 48 52 47 48 58 49 53 47 95.74% 93.88% 100.00% 95.65% 100.00% 45 46 29 44 49 47 49 29 46 49 94.81% 97.80% 100.00% 100.00% 100.00% 73 89 76 76 87 77 91 76 76 87 96.40% 97.39% 99.20% 99.25% 99.30% 97.00% 97 100 98.80% 82 83 100.00% 100.00% 82 82 98.86% 87 88 100.00% 100.00% 100.00% 63 70 63 70 97.35% 954 980 99.74% 754 756 95.71% 67 70 86.21% 50 58 98.07% 712 726 98.33% 588 598 94.71% 448 473 98.81% 910 921 97.84% 91 91 74 74 VTE Prophylaxis (STK1) 96.9% 69 69 48 48 53 53 55 55 Antithrombotics at DC (STK2) 99.2% 5 5 1 1 5 5 5 5 Anticoag Tx for Afib/Aflutter (STK3) 93.8% 75.00% 3 4 97.92% 47 48 100.00% 100.00% 4 4 93.88% 46 49 4 4 7 7 Thrombolytic Tx (STK4) 88.2% 98.95% 100% 100.00% 68 68 100.00% 51 51 Antithrombotics at by Day 2 (STK5) 97.8% 56 56 33 33 44 44 48 48 DC on Statin (STK6) 95.8% 95.45% 42 44 97.62% 82 84 98.80% 96.88% 31 32 100.00% 100.00% 100.00% 60 64 60 64 98.96% 99.07% 100.00% 100.00% 33 33 30 30 Stroke Education (STK8) 90.9% 68 68 Rehab Assessment (STK10) 98.1% 100.00% Composite Pending 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 14 24 25 52 71 33 14 24 25 52 71 33 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 14 24 25 52 71 33 14 24 25 52 71 33 100.00% 100.00% 96.00% 100.00% 100.00% 100.00% 14 24 24 52 71 32 14 24 25 52 71 32 100.00% 100.00% 98.67% 100.00% 100.00% 100.00% 100.00% 639 639 100.00% 639 639 98.11% 624 636 99.37% 46 46 83 83 138 138 71 71 46 46 36 36 CAC-Relievers for Inpt Asthma 100% 98.95% 46 46 83 83 138 138 96.32% 131 136 98.79% 71 71 46 46 36 36 CAC-Corticosteriods for Inpt Asthma 100% 100% 89.13% 41 46 96.38% 100.00% 83 83 100.00% 98.59% 70 71 99.53% 100.00% 100.00% 46 46 100.00% 100.00% Home Management Plan of Care (HMPC) Document Given to Pt./Caregiver - (CAC 3) 36 36 Pending Composite N/A
Survey Responses: What do you like about the CME program at WSU/DMC? Accessibility Tracking my CME Readily available Variety of activities Easy, convenient Variety of speakers at Grand Rounds. Easily available, Practical. Close by, incorporated in clinical schedule Well organized Ability to present and discuss new information It is collegial Very active Available and high quality Cell phone sign in Broad coverage
Survey Responses: What would you like to see improved with regard to the CME program at WSU/DMC? Website listing upcoming CME activities, including those coming in months ahead, so that I can plan my Schedule a few months in advance More use of teleconferencing to allow physicians that aren't in proximity to participate More timely documentation of participation Improved access to my updated CME hours Online / SMART phone effort Online versions for credit if unable to attend in person Interdepartmental CME courses Easier access, web based learning without blackboard hassels
WSU Survey Responses: What are the areas of your practice that are most in need of improvement? Continuity for current molecular diagnostics Continued medical knowledge improvement Work life balance Safety Community Resources in Wayne County (Free / Low Cost Clinics; etc.) Use of rating scales to measure change Patient scheduling Practice organization Automatic access to current practice guidelines and updates preparing for ICD 10 implementation
Beaumont Health Criterion 12: Program- based Analysis Types of Activities what formats did you certify? Hours of Instruction did you offer more than just AMA credit? Target Audience who attended your activities? Internal staff? External? Learners who were your learners? Physicians? Others? Content Areas did you offer content in all areas you hoped? Are you satisfied with the results? How did this year compare to last year?
Beaumont Health Criterion 12 2010 2011 2012 2013 2014 2015 52 93 263 96 104 125 3 20 41 0 4 2 0 0 0 151 221 431 Number of Activities 151 132 86 1 0 370 122 147 87 4 1 361 138 175 94 8 1 416 Seminars Regularly Scheduled Series Web-based Enduring Web-based Live Learning from Teaching Total Activities 400 350 300 250 200 150 100 50 0 Courses RSS Internet (enduring) Internet (Live) Learning from Teaching Total 2011 2012 2013 2014 2015
Beaumont Health Criterion 12: Program- based Analysis Expected Outcomes Learners are expected to gain competence in their approach to clinical problem solving, apply newly acquired strategies in their practice, and change their performance in order to provide evidence-based and patient-centered care. 100% 93% 90% 80% 70% 60% 54% 50% 97% 40% 73% 70% 63% 30% 23% 20% 25% 10% 7% 0% Competence Performance Patient Outcomes Educational Planning Immediate Post-Conference Evaluation National Average
(WSU) Areas for Improvement Plan(s) for Implementation Plans for measuring impact of improvements on WSU s ability to meet its CME mission Steps Taken and Changes Implemented C13 C13 The provider The provider identifies, plans identifies, plans and implements and implements the needed or the needed or desired changes desired changes in the overall in the overall program ( program (eg planners, planners, teachers, teachers, infrastructure, infrastructure, methods, methods, resources, resources, facilities, facilities, interventions) interventions) that are required that are required to improve on to improve on ability to meet ability to meet the CME mission the CME mission Need to integrate CME with quality improvement and GME Joined AAMC AE4Q. Implemented large QI/PICME projects with QI and GME. Developed new template for web based CME using CME Tracker Improved patient care CMS Core measures for VTE prophylaxis showed marked improvement Need for online CME activities A number of CME activities are now available on web using the new web portal infrastructure (CME Tracker) New web template established and a variety of activities now available on web for CME eg, , Need to increase number of RSS with QI related topic Plan more RSS based on QI related topics Increase number of RSS based on QI related topics Organized Grand Rounds for Internal Medicine and Surgery on QI topics About 30% of RSS now on the call in system Need to improve efficiency of attendance data entry for RSS Implemented Cell Phone call in attendance system Gradually increasing number of RSS on the new call in attendance system Policies and forms developed and implemented Need to give WSU Faculty ability to receive credit for Learning from Teaching Developed policy and forms for Learning from Teaching Need to revise the existing policy and forms to make the expectations easier to understand Working hard to make sure speakers do not use trade names in their presentations Need to improve compliance of presentations with regard to trade names Continue to review presentations and work with faculty to get them to understand policy Considering ABMS Portfolio sponsorship System is working but is cumbersome and requires a lot of diligence Need to integrate MOC Part IV projects into QI type CME programs Would move WSU SOM more into demonstrable patient care quality improvement We have participated in several quality improvement projects that were accredited for CME and were also awarded MOC Part IV credits
ACCME Example C13 ACCME Example C13 In its self-study report, County Medical Hospital has described areas where it identified, planned and implemented needed or desired changes to its CME program. These included, incorporating an audience response system, getting a better database to store and analyze participant data and records, and making changes in the evaluation forms to collect more information from learners about future topics.
Answer Has the provider described improvements to the overall program that will help their ability to meet their CME mission per the expectation of Criterion 13?
ACCME Example C13 The Accredited Provider conducts an overall program review and finds that they ve met their mission . During the conduct of that overall program review, however, they have identified organizational challenges with regard to managing the planning and evaluation of CME activities. They describe implementation of a new CME tracking database as a solution to overcome these obstacles. Has the provider described improvements to the overall program that will help their ability to meet the CME mission to the expectation of Criterion 13?
ACCME Example C12 and C13 The accredited provider s self-study narrative includes a description that it achieved some of the expected results (i.e., changing performance) in its mission statement, but has fallen short of goals to change patient outcomes. As a result, the provider described implementing strategic planning with its Executive Committee to identify ways to better align education with quality and safety for better patient outcomes. The provider reported at the interview that early results are showing progress in addressing some patient care issues like falls and poor HCAHPS (patient satisfaction) scores. Please indicate your findings for the provider s performance for Criterion 12 and Criterion 13: A. Compliance with both C12 and C13 B. Compliance with C12, but not C13 C. Compliance with C13, but not C12 D. Noncompliance for both C12 and C13
