Portfolio Committee on Basic Education
Umalusi's mandate and regulatory framework for ensuring the quality of assessments, the framework for quality assurance of assessment, and the processes followed in the 2023 examination are outlined in this briefing. It covers areas of improvement, challenges faced, and recommendations for enhancing the standardization and results of the National Senior Certificate Examination.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Portfolio Committee on Basic Education Briefing on the November 2023 National Senior Certificate Examination 6 February 2024
Presentation Outline 1. Umalusi mandate and regulatory framework 2. Framework for Quality Assurance of Assessment 3. The quality assurance processes undertaken in 2023 3.1 Scope of the 2023 quality assurance of assessment 3.2 Areas of improvement 3.3 Directives for compliance and improvement 3.4 Recurring areas of non-compliance 4. Standardisation and Resulting 5. Challenges 6. Recommendations 7. Conclusion 2
UMALUSI MANDATE AND REGULATORY FRAMEWORK 3
Umalusi mandate and regulatory framework Umalusi derives its mandate for quality assurance of assessment from the: National Qualifications Framework (NQF) Act No. 67 of 2008 Sections 27 (h) and 27 (i) and The General and Further Education and Training Quality Assurance Act as amended in 2008. Section 17 (A) which state that: (3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions (4) The Council may adjust raw marks during the standardisation process and (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has satisfied the conditions that warrant such an approval. 4
FRAMEWORK FOR QUALITY ASSURANCE OF ASSESSMENT 5
Framework for Quality Assurance of Assessment Quality assurance of assessment is conducted to ensure that assessment leading to the award of certificates in schools, adult education centres and technical and vocational education and training colleges is of the required standard. This is in order to ensure that the certificates issued by Umalusi are credible. This is achieved through: Moderation of examination question papers, Practical Assessment Tasks (PAT) and Common Assessment Tasks in the case of Life Orientation Monitoring and moderation of School Based Assessment (SBA) Audit of the State of Readiness to conduct examinations; Audit of appointed markers; Monitoring of the conduct, administration and management of assessment and examination processes Standardisation of marking guidelines Monitoring and verification of marking Management of concessions and examination irregularities Standardisation of assessment outcomes Approval of release of results 6
National Senior Certificate The National Senior Certificate examinations are administered by three Assessment bodies (Department of Basic Education (DBE), Independent Examinations Board Comprehensive Assessment Institute (SACAI). All three assessment bodies are quality assured and certificated by Umalusi. (IEB) and South African Overview of candidates registered to write the NSC examinations Assessment Body DBE Full-Time (FT) DBE Part-Time (PT) TOTAL IEB Full-Time (FT) IEB Part-Time (PT) TOTAL SACAI Part-Time (PT) 2021 2022 2023 733 198 163 965 897 163 12 857 752 003 168 631 920 634 12 581 717 377 181 143 898 520 13 968 1 218 15 186 5 826 968 955 13 825 3 998 13 536 4 951 TOTAL 3 998 4 951 5 826 7
QUALITY ASSURANCE PROCESSESS UNDERTAKEN in 2023 8
Overview of the Quality Assurance of Assessment and Examination Processes: DBE Umalusi moderated and approved 146 NSC question papers for this examination in 59 subjects. The other 16 question papers for six subjects were sourced from the bank. Umalusi sampled 11 NSC subjects for Phase 1 school-based assessment (SBA) moderation, and nine NSCsubjectsfor Phase 2 SBA moderation across the nine Provincial Education Departments (PED). The DBE sstate of readiness to conduct the November 2023 NSC examinations was reported on 13 October 2023. Umalusi monitored a sample of 275 examination centres and 21 marking centres. 9
Overview of the Quality Assurance of Assessment and Examination Processes Umalusi participated in the marking guideline standardisation meetings of 56 subjects comprising 135 question papers for the November 2023 examinations. Umalusi approved and signed-off marking guidelines of all these question papers. Umalusi verified the marking of 37 subjects across the nine PED. DBE presented a total of 66 subjects for the standardisation of the November 2023 National Senior Certificate (NSC) examinations. All irregularities that were identified were managed in accordance with the Regulations Pertaining to the Conduct, Administration and Management of the National Senior Certificate Examinations by the Provincial Examination Irregularities Committees (PEIC) and ratified by the National Examinations Irregularities Committee (NEIC). A detailed draft report, covering all the quality assurance of assessment processes, was shared with the Department of Basic Education. 10
MODERATION OF QUESTION PAPERS 11
Moderation of Question Papers Directives for Compliance and Improvement The DBE must ensure that the examining panels of the question papers that did not comply with the following criteria are supported through intensive training: Technical details; Text selection, types and quality of questions; Language and bias; Accuracy and reliability of marking guidelines; as well as Overall impression 12
Moderation of Question Papers Recurrent Non-compliance The following areas with recurrent non-compliance were noted: a. The recurring low performance levels in the five criteria: technical details; text selection; types and quality of questions; language and bias; accuracy and reliability of marking guidelines; as well as overall impression; and b. The inability to attain 100% compliance with the criteria for internal moderation, content coverage, cognitive skills, conformity with question paper and predictability is still a concern. 13
MODERATION OF SBA, PAT AND ORAL ASSESSMENT 14
Scope of Moderation of SBA Umalusi conducted two phases of SBA moderation. Phase 1: Quality and standard of assessment tasks Phase 2: Learners files (portfolios of evidence) List of subjects Phase 1 Accounting Business Studies Economics English FAL History Life Orientation Life Sciences Marine Sciences Mathematical Literacy Mathematics Physical Sciences Phase 2 Accounting Business Studies Economics Geography History Life Orientation Life Sciences Marine Sciences Mathematics 15
Scope of Moderation of SBA Mathematics Life Sciences Mathematic Accounting Geography Orientation Economics English FAL al Literacy Physical Business Sciences Sciences Marine Studies History Total Life PED Eastern Cape x x x x 4 Free State x x x x 4 Gauteng x x x x 4 Kwa-Zulu Natal x x x 3 Limpopo x x x 3 Mpumalanga x x x x 4 Northern Cape x x x 3 North West x x x 3 Western Cape x x x x x 5 Total 2 2 6 1 4 3 3 3 2 2 4 1 33 16
Moderation of School Based Assessment (SBA) Areas of Improvement Umalusi noted the following areas of improvement: a. Useful comments made by the provincial and national moderators were included in the teacher files; Constructive feedback was provided by provincial moderators in the schools sampled for the moderation of Economics (Eastern Cape); and Marking was of good quality and there was appropriate application of consistent accuracy in marking in all 11 moderated schools in Mathematics (North West). b. c. 17
Moderation of School Based Assessment Directives for Compliance and Improvement The DBE must ensure that: a. Internal moderation processes are strengthened, and teachers are upskilled on moderation practices to improve the implementation of the SBA process (recurring directive from 2022); and Schools adhere to subject policies and procedures and implement current Programmes of Assessment (PoA) in all subjects. b. 18
Moderation of School-Based Assessment Recurrent Non-compliance The following areas with recurrent non-compliance were noted: a. Administration of assessment tasks which are not CAPS compliant. b. Poor quality of internal moderation or lack thereof. 19
Monitoring the State of Readiness Areas of Improvement The following areas of improvement were noted: a. A strategy by the Western Cape PED to mitigate the shortage of markers, by appointing markers per subject and not per paper; and b. Appointment of the State Security Agency by the DBE to provide an external, independent audit of the national key point where question papers are printed and stored. 21
Monitoring the State of Readiness Directives for Compliance and Improvement The DBE must ensure that: a.Sustainable and achievable mitigating strategies to address staff shortages are in place; b.Clear timelines are set for all PED to strengthen security in the storage points, with the intention that non-negotiable, major criteria, as determined, are fully complied with, nationally; and c. A clearly defined contract management solution is set between the DBE and the State Security Agency to accelerate the vetting of staff involved in live handling of question papers across the nine PED. 22
Monitoring the State of Readiness to conduct examination Recurrent Non-compliance The following areas with recurrent non-compliance were noted: a. Inadequate provision of the national information relating to granted examination accommodations; b. Notable shortage of staff at examination and assessment directorates across the Provincial and district levels; and c. Slow progress made for the vetting of staff who are appointed to handle live question papers at provincial printing precinct. 23
AUDIT OF APPOINTED MARKERS 24
Audit of Appointed Markers Areas of Improvement Six (Northern Cape, KwaZulu Natal, Limpopo, North West, Mpumalanga and Western Cape) out of nine PED submitted minutes of meetings of their Marker Selection Committees. 25
Audit of Appointed Markers Directives for Compliance and Improvement The DBE must ensure that: a. The PED adhere to the ratio stipulated for the appointment of senior markers to markers and deputy chief markers to senior markers; b. The PED adhere to the appointment of 15% of novice markers and, where deviations are inevitable, approval to deviate should be sought from the Director-General or HoD as per the PAM requirement; and c. The PED adhere to a threshold of a minimum of three years' experience as an educator in the specific subject or a related area, along with at least two years teaching or other curriculum-related experience within the last five years at Grade 12 level in the respective subject when appointing markers across subjects. 26
Audit of Appointed Markers Recurrent Non-compliance The following area of non-compliance was noted: Non-adherence to the ratio of 1:5 in the appointment of senior markers to markers and deputy chief markers to senior markers. 27
MONITORING OF THE WRITING OF THE EXAMINATIONS 28
Monitoring of Writing of the Examinations Areas of Improvement Scanners were used to detect cell phones and any metal- related material prior to candidates entering the examination rooms. 29
Monitoring of the Writing of Examinations Directives for Compliance and Improvement The DBE must ensure that: a. All invigilation teams are continuously evaluated, to strengthen the implementation of regulation 30 and Annexure I, as provided for in the regulations pertaining to the conduct, administration and management of the NSC examination; and b. Sustainable and continuous innovative security measures are used to make sure an irregularity-free examination is conducted. 30
Monitoring of the Writing of the Examinations Recurrent Non-compliance The following areas of non-compliance were noted: a. Inconsistencies in the execution of the roles and responsibilities prescribed for invigilators. b. Pockets of Behavioural Offences and Acts of Dishonesty (BOADS) that occur during writing: Reported cases of unathourised materials used in the examination sessions; and Reported cases of group copying. 31
MARKING GUIDELINES STANDARDISATION 32
Marking Guidelines Standardisation Areas of Improvement The following areas of improvement were noted: a. An improvement in the pre-marking of 20 scripts requirement by the provincial internal moderators and chief markers was recorded; and b. Northern Cape recorded full compliance with the pre- marking requirement in 2023. 33
Marking Guidelines Standardisation Directives for Compliance and Improvement The DBE must ensure that: a. All pre-marking provincial reports are submitted timeously; b. Provincial delegates receive the requisite number of pre- marking dummy scripts; and c. The PED examination officials do not interfere with the professional duties of the provincial delegates. 34
Marking Guidelines Standardisation Recurrent Non-compliance The following area of non-compliance was noted: While there is an improvement in general adherence to the notion, there were inconsistencies in the pre-marking quota as some delegates marked the same scripts and others received fewer than the required 20 pre-marking scripts from their PED in preparing for the marking. 35
MONITORING OF THE MARKING 36
Monitoring of the marking of examinations Area of Improvement The WCED adopted a strategy in which a pool of appointed markers in selected subjects marked Paper 1 and Paper 2 scripts in History; and Paper 1 in English Home Language, as a way to address too few applications recruitment. received during marker 37
Monitoring of the marking of examinations Directives for Compliance and Improvement The DBE must ensure that: a. There is sufficient supply of water at the marking centres and the boarding facilities; b. Security personnel is trained on how to control security at the entrance to marking centres. 38
Monitoring of the Marking of the Examinations Recurrent Non-compliance The following area of non-compliance was noted: a. Inconsistencies in the management of access control at marking centres by the security firms. 39
VERIFICATION OF MARKING 40
Verification of Marking Areas of Improvement The areas that showed improvement were noted as follows: a. There was marked improvement in the application of the approved marking guidelines in all marking centres; and b. Improvement at all levels of moderation was observed in most marking centres. 41
Verification of Marking Directives for Compliance and Improvement The DBE must ensure that: a. Compliance with the 1:5 ratio for the appointment of deputy chief markers to senior markers is adhered to in all PED; and b. The quality of marking and internal moderation across question papers and PED is strengthened. 42
Verification of Marking Recurrent Non-compliance The following areas of non-compliance were noted: a. Non adherence to the 1:5 ratio for the appointment of deputy chief markers to senior markers; and b. Poor quality of marking and internal moderation especially in SASL HL Paper 2 and Paper 3. 43
STANDARDISATION AND RESULTING 44
What is Standardisation? A process used to eliminate the effect of factors other than the learners knowledge, abilities and aptitude on their performance. 45
Why Standardise Results? To ensure that learners are not advantaged disadvantaged by factors other than their knowledge of the subject, abilities and aptitude. To achieve and consistency of the results from one year to the next. or comparability 46
Standardisation Principles In the case of an individual candidate, the adjustment effected should not exceed half of the raw mark obtained by the candidate No If the After considering qualitative and quantitative reports, Umalusi formulates positions on each subject adjustment should exceed 10% of the historical average in either direction (upward or downward) distribution of the raw marks is below or above the historical average, the marks may be adjusted either way subject to limitations 47
Results Standardised by Umalusi per Qualification National Certificate (Vocational) General Education and Training Certificate N2-N3 Engineering Studies Department of Higher Education and Training (DHET) National Senior Certificate Department of Basic Education (DBE) National Senior Certificate General Education and Training Certificate Independent Examinations Board (IEB) National Senior Certificate General Education and Training Certificate South African Comprehensive Assessment Institute (SACAI) 48
Standardisation Decisions DBE NSC Description 2019 2020 2021 2022 2023 Number of 67 65 67 66 66 instructional offerings standardised Raw marks 47 (70%) 48 (74%) 35 (52%) 47 (71,2%) 49 (74,2%) Adjusted (mainly 13 (19%) 9 (14%) 27 (40%) 16 (24,2%) 6 (9,1%) upwards) Adjusted (mainly 7 (11%) 8 (12%) 5 (8%) 3 (4,5%) 11 (16,7%) downwards) 49
Serious Challenges Observed The following challenges were noted: Challenge How it was resolved Printing errors and errors in question papers. The impact of these errors was mitigated during the marking standardisation meetings preparation for marking. guideline held in Group copying. The results of the affected/implicated candidates were blocked pending the DBE investigations. Loss of candidates' scripts that could not be traced. Missing script principle was used to calculate marks for the lost scripts for qualifying candidates in line with the applications for assessment concessions received from the DBE. 50
