Importance of Quality Assurance Programmes in Population Censuses
Quality assurance programmes play a crucial role in ensuring the accuracy and reliability of census data. These programmes help in identifying and rectifying errors at various stages of the census process, ultimately leading to high-quality outcomes that are valuable for data users. Emphasizing on the importance of incorporating quality assessment throughout all census phases, the programmes aim to enhance the overall quality of census operations by preventing errors and undertaking necessary corrective measures. By establishing quality assurance and improvement systems as an integral part of census programmes, countries can enhance the usefulness and credibility of their census data.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
United Nations Regional Workshop on the 2020 World Programme for Population and Housing Censuses for Arabic-speaking countries 5-8 December 2022 Algiers, Algeria Session 4 Quality assurance in population and housing censuses Seiffe Tadesse United Nations Statistics Division Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Overview Importance of a quality assurance programme o Institutional and organizational preconditions o The role of managers o The quality assurance circle Dimensions of quality Quality assurance by census phase/process/major activity Evaluation o o Processes Data products Quality reports Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Importance of a quality assurance programme P&R recommends that Each country must have a quality assurance and improvement programme in place to measure the quality of each stage of the census (P&R 2.169) A major objective of a quality assurance programme is to ensure that quality assessment is consistently incorporated in all phases of the census, focusing on efforts in controlling the occurrence of errors and taking remedial actions to ensure the highest quality of both the processes and their outcomes. A quality assurance programme should also be viewed as a quality improvement programme, and without such a programme, the census data when finally produced may contain many errors, which can severely diminish its usefulness The quality assurance and improvement system should be developed as part of the overall census programme -- integrated with other census plans, schedules and procedures -- and established at all phases of census operations, including pre enumeration, enumeration, and post-enumeration phases Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Importance of a quality assurance programme Quality is the outcome of processes, and deficiencies in quality (for example, delays in processing or lack of accuracy in the results) are usually the result of deficiencies in process rather than the actions of individuals working in that process. Quality is relative, and is based on what is acceptable to data users, or fit for the purpose, rather than on a concept of absolute perfection The key to quality assurance and improvement is the ability to regularly measure the timeliness and accuracy of a given process so that errors are prevented from reoccurring, to detect errors easily and inform the workers so that they do not continue Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Institutional and organizational preconditions To produce good quality statistics several institutional and organizational preconditions are needed, including: o Mandate and responsibility for producing statistics is clearly specified through a legal arrangement (Statistical Acts and other arrangements) Standards and policies are in place to promote consistency of methods, processes and results, including processes for monitoring and checking quality Sufficient resources (human, financial, ICT infrastructure, skills and capacity) Coordination among data-producing and data-sharing agencies Measures are in place to ensure data confidentiality, and data are used only for statistical purposes Professionalism and ethical standards to assure user confidence in agency and statistical outputs (UN Principles of Official Statistics) Establishment of census quality teams (for various phases/processes; involving survey managers, methodologists, subject-matter specialists, etc.) o o o o o o Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
The role of managers Managers have a vital role in establishing quality their main roles include: o Establishing a culture within the census agency that has a focus on quality issues and to obtain the commitment of staff to strive to achieve high quality goals o Creating an environment in which everyone has the opportunity to contribute to quality improvement o Ensuring that clients expectations are known and that these expectations are built into planning objectives and into the systems that are to deliver them o Ensuring that processes for implementing quality assurance programmes are documented and such documentation provide information on: how quality is going to be measured who is involved in identifying root causes of problems with quality, how the process improvements are going to be implemented Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality assurance circle The quality assurance circle (simple feedback loop) is a schematic representation of the iterative process by which quality is improved Measure quality Quality assurance circle is particularly applicable to tasks that are highly repetitive such as the processing phase of the census Implement corrective action Identify most important problems It is less applicable in processes that are one-off or time-constrained (eg. enumeration) as there is less opportunity to measure performance, identify problems and implement corrective actions Identify causes of problems The emphasis of the quality circle is on improving the process that caused the error , which may be any of the cost, timeliness or accuracy attributes falling below specified levels Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality Quality is a multidimensional concept Outputs of any statistical exercise should possess some or all of the following six main attributes: o Relevance o Accuracy o Timeliness o Accessibility o Interpretability o Comparability Some of these dimensions are inter-dependent and involve trade-off (eg. timeliness and accuracy) Additional dimensions of quality: coherence, completeness Quality indicators and targets which make description of quality more informative and comparable -- could be constructed guided by these dimensions of quality Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Relevance The relevance of statistical information is the degree to which it meets the needs of users -- and suggests the need to avoid the collection and production of data for which there is no significant use o This dimension is important in census content development and dissemination o Relevance is a qualitative assessment of the value of the census data produced, including in terms of meeting the mandate of the agency, legislated requirements, user needs Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Accuracy The accuracy of statistical information is the degree to which those data correctly estimate or describe the quantities or characteristics that the statistical activity was designed to measure o It is usually characterized in terms of errors in statistical estimates o Examples of indicators of quality: coverage rate, sampling error, unit non- response, item non-response, imputation rate, average size of data revision, etc Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Timeliness Timeliness refers the length of time between the census reference day and the date on which the information becomes available o It represents the degree to which information is released in a time period that still permits the information to be of value to users o It often involves a trade-off with accuracy o Census results are often made available over several release dates---so to provide an assessment of timeliness, major information releases should have specified publication dates in the dissemination schedule o Examples: time-lag of first results, time-lag of final results Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Accessibility The accessibility of statistical information refers to the ease with which it can be obtained o Takes into account the suitability of the form in which the information is available to users, the media of dissemination o Availability of metadata o Where data products are provided at cost, the affordability of the information to users also affects accessibility o Example of qualitative quality indicators: consultations with users on metadata, data tables Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Interpretability The interpretability of statistical information reflects the availability of supplementary information and metadata necessary to interpret and use it o Usually it covers the underlying concepts, definitions, classifications used, the methodology of data collection and processing and indications of the accuracy of the information Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Dimensions of quality: Comparability The comparability of statistical information reflects the degree to which statistical information is comparable across countries, regions within a country, and time o Usually underlying concepts, definitions, classifications used, the methodology of data collection and processing provide information on comparability o Example: length of comparable time-series Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Additional dimensions of data quality Coherence Coherence reflects the degree to which the census information can be successfully brought together with other statistical information within an integrated framework over time o The use of standard concepts, definitions and classifications possibly agreed at the international level - promotes coherence Completeness an extension of relevance Completeness reflects the degree to which statistics serve the needs of users as completely as possible, taking limited resources and respondent burden in to account Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality assurance by census phases/processes/major activities Measurement of the quality of key census processes is critical for continuous quality improvement, and for generating information for quality indicators and quality reports Quality assessment can be applied to the entire census cycle with: Performance in the previous phase being evaluated at any given level of detail; Problems with quality ranked in order of importance; Root causes identified and corrective action implemented Process quality can be assessed with a set of information such as process variables, process descriptions, and quality indicators, which may include: Methodological soundness Soundness of implementation processes (sequencing of activities, use of appropriate tech.) Cost-effectiveness, resource and time used Response rates (managing respondent burden) Error rates (editing) Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality assurance by census phase/process: Methodological soundness The application of international/regional/national standards, guidelines and practices in the production of statistical outputs enhances comparability Standards may refer to: o Concepts, definitions and classifications o Design of questionnaires o Data collection methods o Piloting methods o Editing and imputation methods o Analysis of data whether or not they follow accepted standards, guidelines and good practices Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality assurance by census phase/process/major activity Key census processes critical for census data quality -- that could benefit from quality assessment include: Development of census questionnaires and manuals Consultation with users/stakeholders to ensure relevance to users needs and legislative requirements Testing each census question and the design of the form Involving key internal stakeholders in design process: dissemination team, subject matter specialists, data capture and processing teams, etc. Mapping Quality checks of EA maps, boundaries, features, print quality/readability Ensure administrative units and codes reflect what s on the ground Enumeration Establishing criteria and procedures for recruitment and training of field staff Developing standard training materials, monitoring and supervision of training Checking the work of enumerators - coverage/content (observing; re-interview; checking form completion, etc) Monitoring non-response and follow-up rates Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality assurance by census phase/process/major activity Data processing Developing data processing procedures with a view to minimizing the risk of erroneously cancelling, losing or artificially creating households during all phases of data processing Developing procedures for monitoring the quality of each phase Repeating certain procedures based on the sample of batches/records and comparing two datasets Identifying systematic errors Dissemination Ensure relevancy and timeliness of data release within agreed cost constraints Consistency between tables/products Metadata included in reports/products Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Evaluation The P&R recommends that: a complete evaluation takes place and is documented at the end of each phase of the census -- particularly for phases such as planning, enumeration, data processing and dissemination -- for identifying strengths and weaknesses of census phases and for ensuring organizational learning is carried forward to the next census A comprehensive evaluation programme should include: o Evaluation of census processes o Evaluation of data quality The results of evaluations quality (of processes and data) should be made available to stakeholders in the form of quality reports Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Evaluation Operational aspects/processes Operational assessments provide valuable information on strengths and weaknesses of past operational procedures which should be carefully reviewed prior to the development of the next census Operational assessments should: o document operational errors o explain the effectiveness of operations and procedures and their likely impact on overall quality of census Census evaluation with all dimensions of quality requires a comprehensive evaluation programme for assessing and documenting the outcomes of each process using appropriate and customized methodologies these methodologies should be planned well in advance, in the planning phase of the census Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Evaluation Operational aspects/processes P&R recommends that the census evaluation programme should be undertaken by subject specialists according to the agreed goals and methodologies covering all possible dimensions of quality Some areas for evaluation include: o Identification of the deficiencies and achievements in data capture, coding and editing; o Relevance of census data to user needs and satisfaction of users with dissemination tools and products (based on information collected through user consultation); o Achievements and difficulties in use of new technologies and methodologies and identification of improvements for the next census; o Realization of the census calendar, including the calendar of releasing census results, and, in the case of changes to the calendar, the reasons and consequences. Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Evaluation Data quality P&R recommends that Evaluation of the accuracy of the census data should be undertaken, to the extent possible, by conducting a post enumeration survey for measuring coverage and content errors, by comparing the census results with similar data from other sources (surveys and administrative records in a similar time frame & previous census results) and by applying demographic analysis The purposes of evaluating the accuracy of the data are to inform users on the quality of the current census data and to assist in future improvements Evaluation of data accuracy may enable the identification of any problem areas that have not been previously detected through the quality management processes in earlier phases of the census Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Evaluation Data quality Coverage errors Errors in the count of persons or housing units resulting from cases having been missed or counted erroneously Omissions: EAs not covering whole country, EAs having unclear boundaries, enumerator mistakes made in canvassing assigned areas Duplications: overlapping of enumerator s assignments, errors during pre-census listing and delineation Erroneous inclusions: housing units, households and persons enumerated in the wrong place Content errors Errors in the characteristics of persons or housing units resulting from the interview operation (enumerators/respondents), coding, editing, etc. Occurs due to mistakes made by enumerators and respondents during the interview; as well as due to errors made during the coding and editing operations Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Quality reports Important for communicating about the quality of statistical processes or data products Written from perspectives of data producers or data users Focus on either quality of processes or quality of data, or both Typically examine and describe quality in terms of the dimensions of quality adopted Should be produced by considering user needs and perception of quality Such reports are valuable for improving processes and data products, as well as for implementing a continuous quality improvement programme as an integral part of a statistical agency s working practices Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
Conclusions Quality assurance and improvement systems should be developed as part of the overall census programme, and integrated with other census plans, schedules and procedures The systems should be established at all phases of census operations, including in the pre enumeration, enumeration, and post-enumeration phases Quality management procedures for each phase/major activity of the census should be assessed with appropriate methods and quality indicators Statistics Division Demographic and Social Statistics Branch Demographic Statistics Section
