Assessing Impact of Optional Enrichment on Students' Exam Performance in Pharmacy Law Course

Slide Note

Explore the impact of optional enrichment materials on students' exam performance in the Pharm 548 Pharmacy Law course. Presented by Michael C. Brown, PharmD, this study delves into course overview, curriculum placement, competency mapping, assessment design, and implications for student learning. The importance of pharmacy law, MPJE competencies, and related regulatory processes are also discussed.

mmyl Follow

Uploaded on Sep 24, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

ASSESSING THE IMPACT OF OPTIONAL ENRICHMENT MATERIALS ON STUDENTS EXAM PERFORMANCE Michael C. Brown, PharmD Professor, Department of Pharmacy Practice Associate Dean for Academic Affairs Concordia University Wisconsin School of Pharmacy

OUTLINE Course Overview Place in Curriculum Course Development through Competency Mapping Class Time Design Assessment Design Enrichment Design Results Implications, Limitations, and Transferability Brown MC, Faculty Laureate Presentation Apr 20, 2023

COURSE OVERVIEW Phar 548 Pharmacy Law 2 credit class Tuesday and Thursday from 1:10-2:00 pm P3 year (third of four years in the Doctor of Pharmacy program) Covers federal and state law United States Codes and related regulations Controlled Substances Act and related regulations Wisconsin Statues 450 and 961 Pharmacy Examining Board Regulations Phar 1 through Phar 19 Brown MC, Faculty Laureate Presentation Apr 20, 2023

PHARMACY LAWS (522) Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLE: 21 CFR 1306.25 https://www.ecfr.gov/current/title-21/chapter-II/part-1306/subject-group-ECFRe4ae2bfb4eae102/section-1306.25 Accessed 4 18 23 Brown MC, Faculty Laureate Presentation Apr 20, 2023

WHYIS LAW IMPORTANT Governs pharmacy practice Regulations on pharmacy practice Regulations on personnel and physical space Regulations on prescription medications Regulations on controlled substances Pharmacy law licensure exam Multistate Pharmacy Jurisprudence Examination (MPJE) Written by the National Association of Boards of Pharmacy (NABP) 120 questions 2 hours, 30 minutes in length https://nabp.pharmacy/programs/examinations/mpje/ Accessed 4 18 23 Brown MC, Faculty Laureate Presentation Apr 20, 2023

MPJE COMPETENCIES At the time of the course, there were three MPJE Competency Areas Area 1: Pharmacy Practice (83% of exam) Area 2: Licensure, Registration, Certification, and Operational Requirements (15% of exam) Area 3: General Regulatory Processes (2% of exam) https://nabp.pharmacy/programs/examinations/mpje/competency-statements/ Accessed 4 18 23 Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLE: COMPETENCY 1.4 Brown MC, Faculty Laureate Presentation Apr 20, 2023

THE CHALLENGES Pharmacy Law involves a lot of memorization Lecturing alone does not help pharmacy law stick Pharmacy Law is 2 credits of a 16-20 credit semester MPJE performance is solid and has room for improvement Nationally ranked in the top third (37 out of 129 Schools) in 2022 Overall pass rate was 83% on first attempt NABP under some criticism in the profession for difficulty/relevance of the exam Brown MC, Faculty Laureate Presentation Apr 20, 2023

THE OPPORTUNITIES Improve design of the course relative to competencies Improve engagement with the course content Maximize course exam performance Improve MPJE performance Brown MC, Faculty Laureate Presentation Apr 20, 2023

COURSE DESIGN 22 modules of content Tied directly to MPJE Competencies 1.1 through 3.1 Laws mapped to MPJE Competencies Competencies are NOT mapped to related laws by NABP Mapped 526 total laws to the 22 modules 160 (30.4%) related to Controlled Substances 203 Federal and 323 State Median 23 laws mapped to each module Range 11 to 46 17 of 22 sessions had both Federal and State laws Brown MC, Faculty Laureate Presentation Apr 20, 2023

CLASS TIME DESIGN Lecture days Presenting content Presenting cases Use of practice questions Pre-Exam Review days Use of practice questions Use of games (ex: Bingo-like game) Post-Exam Review days Guided review of exam questions and exam performance to find patterns that might inform future study strategies Brown MC, Faculty Laureate Presentation Apr 20, 2023

ASSESSMENT STRUCTURE Post-lecture homework assignments 213 total questions; 6-18 per module Every question is mapped to an MPJE Competency Multiple attempts allowed prior to due date Some responses included feedback 1/3 of course grade Brown MC, Faculty Laureate Presentation Apr 20, 2023

ASSESSMENT STRUCTURE (COND) Exams 5 total exam (4 midterms and a cumulative final exam) No questions were repeated Every question is mapped to an MPJE Competency Format and pace similar to MPJE 2/3 of course grade Exam 1 Exam 1 Exam 2 Exam 2 Exam 3 Exam 3 Exam 4 Exam 4 Final Exam Final Exam Number of Modules Number of Questions 4 7 6 4 22 28 49 42 27 80 Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLE: MULTIPLE CHOICE QUESTION Question: At what number of days of anticipated continuous absence shall a managing pharmacist delegate the supervisory responsibility to another pharmacist? a) 15 or more b) More than 15 c) 30 or more d) *More than 30 e) For any anticipated continuous absence regardless of duration MPJE Competency 1.1 Reference: WI 450.09(1)(b) Use: ASSIGNMENT Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLE: MULTIPLE SELECT QUESTION Question Which of the following persons could prescribe controlled substances while being exempt from DEA registration? (Select ALL that are correct) a) A pharmacist prescribing under a collaborative practice agreement with a physician b) *A physician in the US Public Health Service c) A physician previously registered with the DEA in private practice d) *A staff physician employed by the hospital prescribing under the DEA registration of the hospital e) None - no one can prescribe without DEA registration MPJE Competency 1.3.5 Reference: 21 CFR 1306 Use: ASSIGNMENT Brown MC, Faculty Laureate Presentation Apr 20, 2023

ENRICHMENT STRUCTURE Enrichment 1 466 Answer-in-the-form-of-a-question practice questions Linked to same competencies and laws as assignment and exam questions Review material provided when answer revealed Enrichment 2 Ability to repeat homework after due date prior to exams Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLE: ENRICHMENT 1 MATERIAL MPJE Competency 1.1 Reference: Phar 7.14 Brown MC, Faculty Laureate Presentation Apr 20, 2023

EXAMPLES: ENRICHMENT 1 MATERIAL MPJE Competency 1.1 Reference: Phar 7.14 Brown MC, Faculty Laureate Presentation Apr 20, 2023

RESULTS Brown MC, Faculty Laureate Presentation Apr 20, 2023

NUMBER (%) HOMEWORK ASSIGNMENT REPEATED UNTIL FULL CREDIT EARNED 1000 800 1058 (94.3%) 600 400 64 200 (5.7%) 0 Category 1 Full Credit Less than Full Credit Brown MC, Faculty Laureate Presentation Apr 20, 2023

PERFORMANCEON EXAMS 255 TOTAL EXAMS AMONG 51 STUDENTS 100% Overall average 79.9% 83.2% 82.3% 76.2% 79.6% 78.4% 80% 60% 40% 20% 0% Exam 1 Exam 2 Exam 3 Exam 4 Final Exam Brown MC, Faculty Laureate Presentation Apr 20, 2023

PRE-EXAM ENRICHMENT ENGAGEMENT (N=255) Gameshow-style and repeating archived assignment enrichment 85 (33.3%) No optional enrichment 80 (31.4%) Repeating archived assignment enrichment only 23 (9.0%) Gameshow-style enrichment only 67 (26.3%) Brown MC, Faculty Laureate Presentation Apr 20, 2023

PERFORMANCEON EXAMSBY PRE-EXAM ENRICHMENT ENGAGEMENT 100% 86.5% 82.0% 80% 73.8% 71.2% Control Control p=0.728 p=0.728 p < 0.001 p < 0.001 p < 0.001 p < 0.001 60% 40% 20% 0% No optional enrichment (N=80) Repeated archived assignment enrichment only (N=23) Gameshow-style enrichment only (N=67) Both repeated archived assignment and gameshow-style enrichment (N=85) Brown MC, Faculty Laureate Presentation Apr 20, 2023

MEDIAN GAMESHOW-STYLE ENRICHMENT USEAND RELATED EXAM PERFORMANCE Range of Use When Used Before An Exam (N=152) 3 124.5 Median 765 Maximum Minimum 100% 86.8% 82.2% 80% 60% 40% p = 0.005 20% 0% Below Median Use Above Median Use

USEOF GAMESHOW-STYLE ENRICHMENT: NUMBEROF ENRICHMENT QUESTIONS REVIEWED PRIORTO EXAM* Minimum Minimum Median Median (per Question) (per Question) 0 (0) Maximum Maximum Non-users 0 0 Quartile 1 3 45.5 (1.1) 116 Quartile 2 117 131 (3.1) 145 Quartile 3 146 167 (4.0) 201 Quartile 4 206 268.5 (6.4) 656 *Normalized to median exam (42 questions) Brown MC, Faculty Laureate Presentation Apr 20, 2023

PERFORMANCEON EXAMSBY PRE-EXAM GAMESHOW-STYLE ENRICHMENT ENGAGEMENT p = 0.018 100% 87.5% 85.4% 85.2% 79.9% 80% 73.2% p = 0.01 vs 1st 60% p < 0.001 vs 2nd, 3rd, 4th 40% p > 0.8 20% 0% None 1st Quartile 2nd Quartile 3rd Quartile 4th Quartile Brown MC, Faculty Laureate Presentation Apr 20, 2023

IF THESE EXAMS WERETHEMPJE when ANY Gameshow- Style Enrichment used when NO Gameshow- Style Enrichment used 45.6% passed 85.5% passed A score of 75% is passing on the MPJE Brown MC, Faculty Laureate Presentation Apr 20, 2023

IMPLICATIONS, LIMITATIONS & TRANSFERABILITY Brown MC, Faculty Laureate Presentation Apr 20, 2023

WHAT THIS WORK DEMONSTRATES When students elect to use enrichment material, particularly the gameshow-style enrichment, they better learn the material When students elect to engage more with enrichment material, they do better than students who engage less with the same material Brown MC, Faculty Laureate Presentation Apr 20, 2023

LIMITATIONS Students do not elect to do enrichment randomly Control group is self-selected The reasons for engagement and non-engagement in the enrichment are not clear Is there an enrichment design element that improves/discourages engagement by different cohorts? Very time intensive to develop and maintain Brown MC, Faculty Laureate Presentation Apr 20, 2023

WHAT IS LEFT UNANSWERED (FORNOW) Can we get overall course exam performance above 80%? 90%? Do these approaches lead to better MPJE exam scores? Brown MC, Faculty Laureate Presentation Apr 20, 2023

WHATIS LEFT UNANSWERED (FORNOW) If enrichment helps to this extent, should it be required? If enrichment because required, how does that integrate? If enrichment is required, would it still help? Brown MC, Faculty Laureate Presentation Apr 20, 2023

TRANSFERABILITY Mapping technique Not just law any competency set could benefit Enrichment improves engagement Particularly memorization-heavy course content Brown MC, Faculty Laureate Presentation Apr 20, 2023