Trainee Survey 2022 Findings: Assessment Strategy Review
Based on 420 trainee responses, the survey revealed key insights. A survey response rate of 10% from 4000 trainees highlighted preferences for face-to-face delivery in assessing skills. Gender and ethnic origin distributions among respondents were also analyzed, providing valuable data for future improvements in assessment strategies. Key findings point towards the importance of formative feedback, personalized assessments, and the need for standardized evaluation methods.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Assessment Strategy Review Trainee Survey 2022 Based on 420 responses
Key findings Survey response rate was approximately 10% (survey was sent to 4000 trainees) 53% of respondents were Core trainees, 47% were higher trainees. 42% had not taken the CASC. 30% had taken the online CASC. 25% had taken face-to-face CASC. 3% had taken both. CASC Respondents thought that face-to-face delivery was better to assess Examination (66%) and Communication skills (56%) and History taking (38%). Face-to-face delivery scored highest in the following categories: Networking (63%), Candidates support for each other (56%), Being sure I will complete (44%) Reliability (42%), standardisation (40%), exam conditions (37%), Invigilation (37%), standardisation (60%) Digital scored highest in the following categories: Accessibility (57%), Sustainability (53%), Cost (46%), Anxiety management (43%), Health and Safety (41%) Reliability (68%), completing the exam (66.5%) demonstrate skills (63.7) and standardisation were considered very important factors 45% of respondents preferred face-to-face delivery, 24% online delivery, 19% combination of online and face-to-face, 12% not sure For CASC feedback, free text personalised feedback scored most highly, followed by actual score achieved on individual stations WPBA A combination of formative and summative assessment were the preferred option for the assessment of all clinical skills. Formative scored highest for Physical Examination (43%), Legal and Ethical Management (32%), Developing a personalised holistic management plan (31%), Collateral history (27%), Clinical decision-making skills (63%), Advanced communication skills (44%) and formulation of complex management plans (42%) were considered very important capabilities and knowledge to assess Collaborative working across disciplines/teamwork - scored the highest in terms of gaps in capabilities (28% very important and 49% quite important) Providing training for supervisors on how to provide formative feedback scored highest, making it mandatory for free text comments to be added by supervisors scored second highest, closely followed by providing descriptors/anchor statements to provide more standardised feedback
How would you describe your gender? Number of responses 160 Male Female 250 Prefer not to say 4 Non-binary 6 Other 0 Male Female Prefer not to say Non-binary Other
How would you describe your ethnic origin? Number of Responses Ethnic Origin Prefer not to say White English/Welsh/Scottish/Northern Irish/British 174 Any other Ethnic Group 9 White Irish Other - Arab White Gypsy or Irish Traveller 1 Any other Mixed/Multiple ethnic background 41 Any other White background Mixed - White and Black African 41 Asian or Asian British Indian Mixed - White and Black Caribbean 25 Asian or Asian British Pakistani Mixed - White and Asian 4 Asian or Asian British Bangladeshi Any other Black/African/Caribbean background 8 Asian or Asian British Chinese Black or Black British - African 12 Any other Asian background Black or Black British - Caribbean Black or Black British Caribbean 1 Any other Asian background Black or Black British African 60 Asian or Asian British - Chinese Any other Black/African/Caribbean background 1 Asian or Asian British - Bangladeshi 4 Mixed White and Asian Asian or Asian British - Pakistani Mixed White and Black Caribbean 1 Asian or Asian British - Indian Mixed White and Black African 1 Any other White background 8 Any other Mixed/Multiple ethnic background White - Gypsy or Irish Traveller 16 Other Arab White - Irish 5 Any other ethnic group White - English/Welsh/Scottish/Northern Irish/British 8 Prefer not to say 0 50 100 150 200
Which country do you live in? In which country did you take your primary medical qualification? 400 Other 13 South Africa 1 350 341 Romania 3 Pakistan 22 Nigeria 46 300 Zimbabwe 2 United Arab Emirates 1 Turkey 3 250 Sudan 2 Oman 1 200 Palestine 1 Nepal 1 Malta 1 150 Italy 4 Ireland 3 Jordan 1 100 India 25 Greece 1 55 Barbados 50 2 Bangladesh 6 10 9 Egypt 5 2 0 Bahrain 2 England Scotland Wales Northern Ireland Other UK 253 300 250 200 150 100 50 0
What is your current level of training? 196 224 Core Trainee (C1-C3) Higher Trainee (ST4-ST6)
Which exam(s) have you taken? (select all that apply) 231 388 317 MRCPsych Paper A MRCPsych Paper B CASC
Which versions of the CASC exam have you taken? 100 171 124 12 Face-to-face Online Both Neither
Thinking about the CASC exam, how well do you think clinical skills can be assessed through the different exam delivery methods? Face-To-Face Is Better Digital Is Better Don t Know / N/A Same/Similar History Taking (including risk assessment) 37.9% 34.5% 9.4% 18.1% Examination (mental state, cognitive, physical and capacity assessment 66.2% 12.7% 5.5% 15.6% Clinical Management 28.5% 44.8% 10.3% 16.5% Communication Skills 55.6% 20.3% 7.8% 16.3%
Based on your perception and/or experience with digital and face-to-face exam delivery for the CASC exam, how do the following factors compare? Face-To-Face Is Better 41.7% 8.7% 37.5% 12.4% 37.4% 27% Digital Is Better 7.5% 45.8% 26.6% 40.8% 10.4% 11.7% Don t Know / N/A 30.8% 30.9% 23.8% 29.5% 32.2% 36.2% Same/Similar 19.9% 14.6% 12.2% 17.3% 20% 25.1% Reliability of the exam Cost Exam conditions Health and safety Invigilation Information and support before and during exam Ability to manage anxiety and stress before and during exam 18.9% 15.2% 43.3% 22.6% 11.4% 10.9% 57.3% 20.3% Accessibility 44.4% 18.4% 14.1% 23.1% Being sure that I will be able to complete the exam Examiners and role players can meet to standardise the station effectively 40.8% 21.3% 8.9% 29% 63.5% 4.4% 7.4% 24.7% Candidates can undertake professional networking 56% 11.4% 10.2% 22.4% Candidates can support each other 11% 11.3% 52.8% 25% Sustainability
How important are the following factors to you? Not Not that important Very important at all Neutral Important important The exam enabling candidates to demonstrate 1% 0% 2.7% 32.7% 63.7% Reliability of the exam 0.2% 0% 2% 30.2% 67.6% Cost 1.2% 2% 6.9% 35.1% 54.8% Exam conditions 0.5% 1.2% 6.7% 40.7% 50.9% Health and safety 5.2% 9.5% 25.1% 34.8% 25.4% Invigilation 1.2% 4.2% 20.4% 45.8% 28.4% Information and support before and during exam 0.3% 1.5% 10.5% 46% 41.8% Accessibility 1.7% 4.2% 16.4% 38.6% 39.1% Being sure that candidates and examiners will be able to complete the exam Examiners and role players can meet to standardise the station effectively Examiners can undertake continuing professional development 0.2% 0% 3.5% 29.8% 66.5% 0.2% 2.5% 8.7% 37.4% 51.1% 15.9% 22.6% 25.9% 20.6% 14.9% Examiners and candidates can undertake professional networking with colleagues 5.5% 17.4% 28.3% 29.5% 19.4% Sustainability 3.2% 4% 23.4% 40.4% 28.9%
What would be your personal preference regarding the most appropriate delivery method for the CASC exam? 50 80 184 98 Face-to-face Online remote delivery Combination of online and face-to-face Not sure
Online CASC positive comments Minimises anxiety and stress from travel / strange environment Sustainability Cost No need to worry about health impact, infections No need to worry about exam being cancelled Time off work / away from family Better for international candidates to access Reality of new ways of working is virtual, need to find way to access Blend of both reflects modern working practices Visa and currency exchange challenges Less nerve-wracking Avoids the anxiety of understanding and navigating the venue Timing is easier online No need to take study leave to travel to exam Not surrounded by other anxious candidates 'holding pen' / intimidating Difficult to focus on stations f2f as could hear other stations around one Easier to practice with colleagues online Difficult of sleep in strange hotel room Less discriminatory for autistic doctor space to prepare undisturbed Face to face venue not able to hear very well, echos and other candidate noise
Face to face CASC positive comments Many candidates emphasised benefits of face to face in respect of: Building rapport and empathy Evaluating mental state, physical examinations, cognitive testing more effectively Countertransference Easier to maintain attention No need to worry about technical infrastructure and wifi access Ability to demonstrate skills more effectively Being much more reflective of an actual clinical environment
Trainees in the focus groups have indicated that they would like to have more feedback from the CASC exam so that they can improve their performance. What sort of feedback do you think we should provide? Please rank these options by putting your favourite at the top and least favourite at the bottom. (%) First Choice Last Choice 60 50 Rank 1. Free text feedback personalised for each candidate 2. Actual scores achieved on individual stations 3. Feedback in respect of specific clinical areas 4. Descriptions of areas for improvement clearly linked to syllabus 40 30 20 10 0 Free text feedback personalised for each candidate 52.8 23.9 13.1 10.3 Descriptions of areas for improvement clearly linked to syllabus 12.3 14.1 27.6 46 Feedback in respect of specific clinical areas Actual scores achieved on individual stations First Choice Second Choice Third Choice Fourth Choice 9.5 29.4 42.2 18.8 25.4 32.7 17.1 24.9
We are developing a new syllabus for the CASC exam and reviewing the use of workplace based assessments across core and higher psychiatry training. Please indicate if you think the skills listed below should be assessed by summative assessment (such as the CASC exam), or by formative assessment (such as workplace based assessments), or by both. Summative Only Formative Only Both Assessment Types Summative Only Formative Only Both Assessment Types Biological management 7.5% 18% 74.6% History taking from patient 7% 11.5% 81.5% Psychological management 7.1% 22.6% 70.2% Collateral history 7.9% 27.1% 65% Developing a personalised holistic management plan 7.4% 30.9% 61.7% Mental State Examination (psychotic symptoms) 8.7% 10.2% 81.1% Safety planning 6.9% 21.4% 71.8% Mental State Examination (non-psychotic symptoms) 8.6% 10.1% 81.3% Legal and Ethical Management 7.4% 32.1% 60.5% Mental State Examination (cognitive function) 8.2% 19.2% 72.6% A personalised and respectful approach to communication 6.4% 17.9% 75.8% Capacity Assessment 8.4% 18.9% 72.6% Recognising and responding effectively to cues 7.9% 16.8% 75.3% Physical Examination 12.4% 42.6% 44.9% Empathy 7.1% 21.9% 71% Risk Assessment 7.1% 12% 80.9% Providing clear, understandable information 6.3% 9.1% 84.6% Formulation Skills 6.9% 35% 58.1%
Below are listed some suggested gaps in the capabilities and knowledge currently evaluated by formative assessment (WPBAs) and summative assessment (exams). Please indicate how important you believe it is that we should evaluate these capabilities and knowledge in some form of assessment. Not Not that important Quite Important Very important at all Neutral Important Advanced verbal, non-verbal and written communication skills 0.3% 1.8% 13.3% 40.8% 44% Advocacy and personal health impact 1.5% 6.1% 30.4% 43.8% 18.2% Clinical decision-making skills 0.3% 0% 4.1% 32.4% 63.3% Cultural differences in clinical encounters 2.3% 3.8% 16.3% 48.7% 28.8% Digital literacy and e-consultations 5.8% 12.4% 30.1% 41% 10.6% FLIPAT Flexibility, Leadership, Initiative, Prioritisation, Adaptability and Timekeeping 5.3% 11.1% 31.6% 37% 14.9% Formulation of complex patient management plans, managing complexity and uncertainty 0.8% 1.8% 8.9% 46.8% 41.8% Personalisation of care plan, patient journey from admission to discharge 1% 4.8% 14.9% 45.8% 33.4%
Below are listed some suggested gaps in the capabilities and knowledge currently evaluated by formative assessment (WPBAs) and summative assessment (exams). Please indicate how important you believe it is that we should evaluate these capabilities and knowledge in some form of assessment. Not Not that important Quite Important Very important at all Neutral Important Training, mentoring, peer-to-peer support 3.8% 11.9% 27.7% 37.1% 19.5% Collaborative working across disciplines, teamwork 1.3% 6.1% 16.3% 48.7% 27.6% Research, audit, and quality improvement 10.9% 19.8% 30.5% 27.9% 10.9% Doctor s relationship with local communities 10% 17.7% 36.2% 25.4% 10.8%
Tell us about any skills that you think are missed by the present assessment system but that are not listed above. Psychodynamic formulation skills trauma informed care Emotional literacy Neurostimulation, developmental assessment, statistical reasoning Written communication Ability to spot errors Psychopharmacology Leadership Management Deviating from guidelines nuanced care Research Keeping cool when faced with challenging behaviour Interface between psychiatry, physical health and substance misuse Complex patients Chairing meetings Motivational skills Educational skills Safeguarding Medicolegal reports Rehabilitation
Trainees in the focus groups have raised concerns about the quality of feedback they receive from their supervisors or WPBA assessors on how they can improve their performance. How could we improve the quality of this feedback? Please rank these options by putting your favourite at the top and least favourite at the bottom. First Choice Last Choice Rank 45 1. Provide training for supervisors on how to provide formative feedback 40 35 30 2. Make it mandatory for free text comments to be added to WPBAs and end of year/placement evaluations 25 20 15 3. Provide descriptors/anchor statements to enable supervisors to provide more standardised evaluations 10 5 0 Provide Make it mandatory for free text comments to be added to WPBAs and end of year/placement evaluations 21.5 18.9 21.2 18.4 10.9 9.1 Reduce the Likert scale evaluation to 'not at level expected'/'at level expected' in conjunction with free text comments 15.7 13.6 14.6 16.4 24 15.7 descriptors/anchor statements to enable supervisors to provide more standardised evaluations 19.7 21.2 19.4 23 9.6 7.1 Provide training for trainees on how to receive and benefit from formative feedback Provide training for supervisors on how to provide formative feedback Remove the Likert scale evaluations and ask only free text comments 4. Reduce the Likert scale evaluation to only not at level expected / at level expected in conjunction with free text comments 5. Remove the Likert scale evaluations and ask only for free text comments First Choice Second Choice Third Choice Fourth Choice Fifth Choice Sixth Choice 26 24.5 14.4 14.6 17.7 2.8 14.1 12.9 12.1 12.6 23.2 25 3 8.8 18.2 14.9 14.6 40.4 6. Provide training for trainees on how to receive and benefit from formative feedback
If the College was to provide training for supervisors, what skills should that training address? 48% of respondents said Feedback Other responses: How to standardise evaluation Constructive and personalised, individual, specific, not generic praise Support trainees especially those in difficulty How to protect time for supervision Standard setting with other supervisors Train trainees to know what to ask for! Portfolio training Bias elimination, EDI Empathy Leadership How to adjust expectations for neurodiverse / disabled trainees Time management Give them framework of free texts to use Make ES and CS responsible for completing reports on time Understanding of what trainee should have accomplished by each point in time Compassion How to adapt for people outside of British culture Cultural barriers and how to overcome Update on curriculum Communication skills Conflicts of interest between supervisor and line manager role How to engage trainees with their training Free text feedback Their obligations as trainers Mentoring
