Revolutionizing Education Assessment in Southern Africa
Explore the challenges and imperatives in education assessment as discussed at the 2018 conference by the Southern Africa Association of Education Assessment (SAAEA). Delve into the importance of redefining traditional assessment methods with a focus on school-based assessments for a more comprehensive evaluation of learners' capabilities.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
SOUTHERN AFRICA ASSOCIATION OF EDUCATION ASSESSMENT (SAAEA) 2018 CONFERENCE 13 16 MAY 2018
Getting the Moderation of School-Based Assessment Right 2
The Educational Imperative Learners who can think critically and creatively, use evidence to support their solutions to complex problems, and communicate clearly. (National Curriculum Statement Grades R -12) The Assessment Imperative To evaluate that learners are on track to achieving this goal? (Curriculum and Assessment Policy Statement)
The Big Assessment Challenge? In examination dominated systems, learners respond to a test or examination and the assessment fails to confirm that they can think critically and creatively?
School Based Assessment Curriculum developers and examiners, across developed and developing countries have come to realise that real change will not take place in schools if traditional paper and pencil tests, are not replaced by more valid and authentic ways of assessing which have been introduced into school syllabus over recent years (Torrance, 1994) 2
The Case for SBA (a) ..school-based assessment is often put in place to collect evidence on what students have achieved, especially in important learning outcomes that do not easily lend themselves to pen and paper tests (Izard, 2001 and Raivoce and Pongi ,2001) . (b) it is now being accepted more widely that internal assessment for qualifications can include an even wider range of learning outcomes, even those traditionally assessed by an external examination . (Lusby, 2004) (c) broader evidence about student achievement than what can be obtained from examinations administered in a brief period (Taylor, 2003) (d) opportunities to assess knowledge and skills that are difficult to assess via standardized large scale tests/examinations (Taylor. 2003) 2
The Assessment Conundrum 2 Low Validity High Reliability Examinations School Based Assessment High Validity Low Reliability
Quality and standard of Tasks Score Variation Teacher Capacity THE CHALLENGES OF SBA Support Teacher and Parent assistance Mimic Exams Quality Assurance
SBA in South Africa (a) Fragmented system pockets of good practice, widespread sub- standard assessment tasks, inflated marks. (b) Assessor capacity - assessment task development and assessing learner evidence. (c) Inadequate subject advisory support shortage of person power and lack of appropriate skills register. (d) Failure to comply to policy requirements. (e) Dominated by tests and examinations. (f) Lack of rigorous internal moderation systems. (g) Limited coordination at provincial and district level. (h) Lack of internal monitoring (i) Clarity of roles and responsibilities. 2
Quality Assurance at the different levels Umalusi Moderation and Statistical Moderation National Province Moderation District School 2
Moderation of SBA (School, Cluster, District, Province, National) (a) Moderation of the Assessment Task (b) Monitoring of the Administration (c) Moderation of the Assessment Evidence (d) Feedback No Mark Adjustments
Statistical Moderation (Umalusi) (a) External Exam mark is used as the anchor. (b) External exam is adjusted based on historical norms. (c) Raw SBA marks are adjusted within a certain range of the adjusted examination mark. (d) Orals and Practical Assessment Task (PAT) are combined with the exam mark SBA Marks Adjusted
Understanding the Statistical Moderation by Umalusi The Principles relating to the adjustments based on Mean Comparison: Principle 1: If the mean of the SBA marks for the centre are between 5 10% above the adjusted examination mean, the SBA marks of that centre will be accepted as is. Adjusted Exam Mean Raw SBA Mean Final SBA Mark 50 57 57 45 55 55
Understanding the Statistical Moderation by Umalusi The Principles relating to the adjustments based on Mean Comparison: Principle 2: If the mean of the SBA marks for the centre are less than 5% above the adjusted examination mean, the mean of the SBA marks must be brought up to 5% above the adjusted examination mean. Adjusted Exam Mean Raw SBA Mean Final SBA Mark 50 53 55 50 48 55
Understanding the Statistical Moderation by Umalusi The Principles relating to the adjustments based on Mean Comparison: Principle 3: If the mean of the SBA marks for the centre are greater than 10% and less than or equal to 15%, above the adjusted examination mean, the SBA marks will be adjusted. Adjusted Exam Mean Raw SBA Mean Final SBA Mark 50 61 59 50 62 58 50 63 57 50 64 56 50 65 and above 55
Understanding the Statistical Moderation by Umalusi The Principles relating to the adjustments based on Standard Deviation (SD) (a) Standard deviation is an indication of the spread of the marks. (b) A low SD means marks are bunched at a particular mark. (c) A (SD) less than 5% and less than of the exam SD; Indicates the following: (a) The SD is very small, implying that assessor was not able to distinguish the abilities of the candidates (bunching of marks). (b) Distribution of SBA marks lower than exam marks- greater bunching of SBA marks. However: When SBA marks are rejected, a mark is generated which is equivalent to 5% above the examination mark (hence learner gets the minimum of 5%).
Provincial Comparative Analysis 2016 (SBA and Examination) %Rejected Rejected % >15% %<5% >15% <5% ALL EC 10942 420 3.8 1457 13.3 5297 48.4 4190 63 1.5 181 4.3 2654 63.3 FS GP 12510 104 0.8 682 5.5 8613 68.8 KZN 19325 540 2.8 2581 13.4 9500 49.2 14818 913 6.2 1845 12.5 6804 45.9 LP MP 6799 209 3.1 712 10.5 3523 51.8 1602 34 2.1 171 10.7 886 55.3 NC NW 4481 163 3.6 315 7.0 2594 57.9 WC 6042 28 0.5 228 3.8 3933 65.1 NATIONAL 80690 2474 3.1 8172 10.1 43804 54.3
THE LIMITATIONS OF THE STATISTICAL MODERATION MODEL
Limitations of the Current Model (a) Undermining the development of skills that are attained through other forms of assessment (b) Expert judgment moderation and statistical moderation are carried out as separate methodologies (c) All components of SBA are regarded as measuring the same construct as examination. Hence all SBA components are aggregated and brought to within a certain range of the adjusted examination mark. (d) Consider the formal SBA relating to Life Sciences: - 3 tests - Mid-Year Examination - Preliminary Examination Exam Like - 3 selected practical tasks - Environmental Studies Fieldwork Non- Exam like
Elements of the Quality Assurance Model (a) Focus on both Statistical Moderation and Expert Judgment Moderation in an integrated approach. (b) Identify the components of SBA that are exam-like and those that are not exam-like Statistical Moderation Expert Judgment Moderation Approach Statistical Moderation supplemented by sample review of learner s work Expert supplemented techniques judgment moderation statistical with Characteristics of SBA Component This method is adopted for subjects in which the SBA and the public examination share a substantial portion of common assessment objectives This method is adopted for subjects with a small candidature or which involve outcomes that are very different from those assessed through the public Examination Determination of the mean with reference to review of samples of Learner s work (expert judgment), assisted with statistical techniques Key Features Determination of the mean in the exam, supplemented with review of samples of learner s work
Elements of the Quality Assurance Model (c) Statistical moderation uses the mean and standard deviation of the class group. (d) Expert Judgment follows the following procedure: select a representative sample of learner evidence. moderate the assessment task and the learner evidence determine the group mean and SD from the sample that has been reviewed. compare group mean of the reviewed sample to the group mean of the population. if the two are broadly comparable, accept the scores of the population. if the scores are not comparable, adjust the mean and SD of the population to the reviewed scores
Quality Assurance at the different levels Umalusi External Moderator Moderator National Curriculum Head Province Subject Advisor District School HoD 2
Elements of the Quality Assurance Model (e) Expert Moderation Judgment must adopt a linked sample approach: Each level must take responsibility for the level below The focus must be on the custodian of the standard at that level. If a district is divided into clusters: the cluster leader must take responsibility for the schools in his/her cluster. all the schools in that cluster comply with a particular standard. moderation of the cluster by the district subject specialist, will entail a moderation of a sample of schools in the cluster. moderation of the district by the province, will entail a moderation of a sample of schools from selected clusters. Moderation of the province by National, will entail a moderation of a sample of clusters from a district
Conclusion (a) SBA assesses skills and knowledge application that cannot be assessed under controlled examination conditions. (b) Hence, SBA must not be compromised by assuming that SBA and exams measure the same construct, and hence the assessment outcome should be the same. (c) Distinguish the constructs measured in SBA, and treat like with like. (d) Integrate statistical moderation and expert moderation judgment. (e) Adopt a sampling technique that links the school, to the cluster, to the district, to the province and finally to the DBE and Umalusi. (f) Focus at each of these levels must be the assessment custodian.
