Optimised Methods for Collecting Socioeconomic Information in the Pacific Region
This presentation explores the optimised methods used in the Republic of the Marshall Islands Household Income and Expenditure Survey (RMI HIES) to collect socioeconomic data. It covers the objectives and history of HIES in the Pacific, the standardised methodology implemented since 2012, and details of the HIES questionnaire modules.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
RMI HIES Experiment: optimised methods to collect socioeconomic information Michael Sharp Economic Statistics and Microdata Specialist Statistics for Development Division michaels@spc.int
HIES objective and use Multi-purpose and multi-user survey with the objective to: Update the household component of GDP Rebase the CPI Conduct poverty analysis Conduct general demographic and socioeconomic analysis More recently, conduct food security and nutrition oriented analysis Directly populate 17 of the 132 Pacific subset of SDG indicators (13%) Contributes data to an additional 6 indicators (17% of the total subset)
History of HIES in the Pacific Figure 1 12 12 9 9 70 HIES conducted since 1943 (Fig1) with 60% since MDG era Excluding Pitcairn, all PICTs conducted at least 1 HIES (Fig2) 3 PICTs with only 1 HIES 3 PICTs with most recent HIES pre- 2010 Desirable to have more, but costly (NSO, HH, TA provider, donor) 6 5 4 3 3 3 2 1 1 13 Figure 2 5 4 4 4 4 4 4 3 3 3 3 3 2 2 2 2 2 1 1 1 PNG TUV KIR NCL FJI SLB TON NIU NRU WLF MNP PLW ASM VUT PYF FSM WSM GUM MHL COK TKL
The regionally standardised HIES methodology
Pacific standardised HIES Implemented in the Pacific since 2012 Developed under the guidance of a technical working group Standardised: Survey instruments Classifications Field operations schedule Data capture and processing system Report and data structure
HIES questionnaire Module 1 - Demographic information: demographic profile; activities last week; education; health; communication; and previous household members. Module 2 - Household expenditure: dwelling characteristics; dwelling tenure; utilities and communication; land and dwelling; goods and assets; vehicles; private travel; services; cash contribution to special occasions; provision of financial support; insurance and taxes; and loans. Module 3 - Individual expenditure: education; health; clothing; communication; luxury items; and alcohol, betel nut, kava and tobacco; (optional: deprivation).
HIES questionnaire Module 4 - Household and individual income: wages and salaries; agriculture and forestry; fishing, gathering and hunting, livestock and aquaculture; handicrafts and home processed foods; non-subsistence business; property, transfer income; and remittances and cash gifts. Two-week diary: cash expenditure on food and non-food items; cash expenditure on services; goods and services received for free; home produced items today (home consumption, to give away and to sell). Main methodological issues surround the use of the diary.
Field schedule Rolling 12-months sample (seasonality) Field teams: Supervisor (supervision and CAFE data entry) and two enumerators Two-week diary requires field teams to be in an EA for three-weeks 16 HIES rounds main constraint Seven to eight visits per household per round Six to seven households per enumerator per round (~100 per enumerator in total)
Challenges with the regionally standardised HIES methodology
Methodological challenges Despite the regionally standardized HIES meeting the multi-user needs, there are a number of inherent methodological issues, including: High cost Apparent respondent and/or enumerator fatigue Diary versus recall Food away from home Food acquisition as a proxy for consumption Non-standard units Partakers Imputed rents and consumption of fixed assets
High cost Last HIES Regionally standardised? Total cost (USD) Cost per sampled household (USD) Cost per capita (USD) Next HIES ASM 2015 No - - - 2025 COK 2015 Yes 195,000 244 13 Unknown FJI 2013 Partially - - - 2019 FSM 2013 Yes 416,000 250 4 2019 GUM 2005 No - - - 2018 KIR 2006 No 625,510 313 5 2019 MHL 2002 No 640,000 800 12 2019 MNP 2016 No - - - Unknown NCL 2008 No - - - 2018 (ongoing) NIU 2015 Yes 42,250 264 28 Unknown NRU 2012 Yes - - - 2019 PLW 2014 Yes - - - 2019 PNG 2009 No - - - 2019 PYF 2015 No - - - Unknown SLB 2012 Yes 3,000,000 651 4 2020 TKL 2015 Yes 32,500 271 23 2019 TON 2015 Yes 321,200 176 3 2020 TUV 2015 Yes 170,400 237 17 2021 VUT 2010 No 786,922 262 3 2019 WLF 2005 No - - - 2019 WSM 2013 Partially 570,000 285 3 2018 (ongoing) Mean 341 10
Fatigue Literacy Time poor Seven to eight enumerator visits recommended Diary drop off in transactions over the two-week diary period (ref: Fig) Well managed diary considered to be gold standard
Diary versus recall Diary value Recall value Difference SBD SBD 1,280 750 28,000 165,090 135,120 263,830 56,250 56,950 TOP TOP 660 270 680 7,800 16,800 2,210 COICOP Division 2 (alcohol, tobacco and narcotics) May be underreporting in the diary as COICOP Division 2 is already captured in the modules, but clear differences in expenditure when comparing diary and recall data sources Solomon Islands 2012 HIES Spirits Wine Beer Tobacco Betel nuts Tonga 2015 HIES Spirits Wine Beer Tobacco Kava % 143% 13% 490% 95% 1% % 446% 93% 462% 115% 184% 3,110 850 3,600 520 3,820 6,270
Food away from home Food acquired and consumed away from the dwelling Anecdotally a significant calorie source for Pacific Islanders (feasts, takeaway) No dedicated section in HIES questionnaire Ref fig: Proportion of total food expenditure from COICOP Group 11.1 Solomon Islands pocket money
Non-standard units of measurement Common methodological issue across all HIES (or its equivalent) Ref fig: Proportion of food items reported in NSU Statistical methods available, but need critical mass Market and producer surveys over time and geographical domain
Other methodological challenges Food acquisition as a proxy for estimating consumption Very high and very low food acquisition Home production for consumption today or total home production Opening and closing stocks (Consumption = opening stock + food acquisition closing stocks) Partakers Food sharing is a major component of the Pacific culture Partakers not well captured to account for sharing of food in consumption estimates Imputed rents No baseline rental market House value and age not collected Use value of assets Age and value of durable goods not collected
The RMI HIES Experiment There were three main components to the experiment: Test the use of new technology in the form of using tablets to collect data through computer assisted personal interview (CAPI) and the potential use of new data backup protocols in small island states where internet connectivity is limited; Test the inclusion of complementary survey modules to increase the production of complementary data through HIES so as to avoid the need for stand-alone surveys; and Test new methodologies to collect consumption data through HIES.
New technology The RMI HIES experiment is testing the use of computer assisted personal interview (CAPI) to collect HIES data. Given the complexity of the HIES questionnaire and the significant period of time that field teams spend in areas where internet connectivity and electrification is limited, it is necessary to test whether CAPI is an appropriate means for the capture of HIES data. The experiment is testing the use of a satellite phone for data backup in remote island locations where there is no internet.
Complementary survey modules In order to meet demand for more dynamic data and to eliminate the need to conduct stand- alone surveys, the below new survey modules are being tested for inclusion in the core HIES questionnaire. Food away from home: recall and diary-based modules are being tested to encourage more comprehensive reporting of food acquired and consumed away from the dwelling. Partakers: recall and diary-based modules are being tested to capture partakers in meals in order to improve consumption estimates. Stocks: collection of opening and closing food stocks before and after the two-week food acquisition diary is implemented this will allow for estimation of apparent consumption, rather than just acquisition. Labour module: newly designed module in accordance with the resolution concerning the International Classification of Status in Employment (ICSE-18). Food Insecurity Experience Scale (FIES): to measure experiences of moderate-to-severe food insecurity and to report against SDG indicator 2.1.2. Shocks: household exposure to shocks through adverse environmental, economic and social change. Disability: addition of the Washington Group six questions to identify vulnerable populations. Improved modules: enhanced health and fisheries modules and improved means to capture information to better estimate household consumption of fixed assets.
Methods for the collection of consumption data There are five experimental arms, which are used to determine the efficacy of collecting consumption data through diary and 7-day recall means. The experimental arms are described below. ARM 1 - Recall CAPI: household members are asked to recall if they acquired and consumed food and non-food items in the last seven days; they are asked the number of partakers in each meal in the last seven days; and they are asked if the household members acquired food away from home in the last seven days. These answers are all provided by the respondent and entered into the tables by the enumerators. ARM 5 - Bounded recall CAPI: this is the same as Recall CAPI, however the enumerator visits the household seven days before conducting the recall interview to ask the household to try to remember what was acquired/consumed over the seven day period leading up to the interview. ARM 2 - Diary CAPI high monitored: household members are asked to keep a 14-day diary of all food and non-food items that the household acquired each day; they are also asked to record if they acquired and consumed food away from home; the enumerator will visit the household every second day to enter the diary data into the tablet and to ensure that the household is completing the diary comprehensively. ARM 3 - Diary PAPI high monitored: this is the same as Diary CAPI high monitored, except the diary data is not entered into the tablet during the visit to the household; it is entered into a data entry system by the national statistical agency after each round; the enumerator will still visit the household every second day to ensure that the diary is being completed. ARM 4 - Diary PAPI low monitored: this is the same as Diary PAPI high monitored, however the enumerator will only visit the household to drop the diary off on day one, then pick up the first week diary on day seven and to drop the second week diary, then pick up the second week diary on day 14; this is a poorly managed diary without regular monitoring of the household to ensure the diary is being completed.
Research Questions The results of the RMI HIES experiment have guided the development of an updated regionally standardised HIES, generally answering the following questions: Is CAPI an appropriate data capture means for HIES? Do the new survey modules produce the data required by the user? Does diary or recall produce more comprehensive consumption data and which implementation protocol will result in the production of high quality consumption data?
Location RMI was selected as the location to conduct the HIES project as RMI is scheduled to conduct an official national HIES in 2019, so the HIES project will serve to build capacity of Economic Policy, Planning and Statistics Office (EPPSO) in conducting complex statistical collections in advance of the 2019 HIES. From a regional perspective, the geographical context of RMI will allow the testing of methods and technologies in remote island locations, which is key to determining field protocols in low- and-no connectivity environments, as well as a densely populated urban areas, such as in Ebeye.
Analysis of results Survey Monkey: post-experiment survey of EPPSO and field staff Empirical analysis Complementary survey modules CPI National accounts Dietary energy consumption Poverty
Survey Monkey Important to ensure that the new technology, survey modules and methodologies can be operationalized by Pacific statistical agencies and that they are fit for purpose in the cultural and logistical context of the Pacific. The online survey was implemented using SurveyMonkey and conducted in April, 2019. The questionnaire covered the following general themes: Demographic profile of the respondents; Use of technology, including electrification, internet accessibility and use of a satellite phone to access the internet to synchronize data; Implementation of the different experimental arms to collect consumption data; and Inclusion of the complementary HIES questionnaire modules. In total, 20 responses to the online questionnaire were received (93% response rate from field staff).
Survey Monkey results: New Technology General assimilation of the use of tablets to conduct HIES. CAPI v PAPI: Use of CAPI is easier than PAPI CAPI (Survey Solutions) v CAPI (Other): Use of Survey Solutions CAPI software was equally as easy if not more than experience using other CAPI data capture software. Electrification: Did not appear to be a significant challenge in the RMI HIES Experiment. Power banks needed. Internet connectivity: somewhat problematic, particularly in rural atolls. The trial of the use of the satellite phone to connect to the internet was seemingly unsuccessful due to unreliable connection resulting in field staff not feeling confident to use the technology to synchronize their data. Satellite phone data use
Survey Monkey results: complementary modules The complementary survey modules did not appear to be problematic. The feedback indicates that more training is required for the following modules: Economic activity; Food insecurity experience scale; and Fisheries.
Survey Monkey results: consumption modules Preferred methodologies for the collection of consumption data are, in order of preference, Arm 1 (7-day food consumption recall), followed by Arms 2 and 3 (highly monitored two week diary, on CAPI and PAPI respectfully). Some concerns about the respondent s ability to accurately recall consumption were raised about the use of 7-day recall. Concerns about respondent fatigue through multiple visits and limited response to the diary were raised about Arms 2, 3 and 4. Whilst the feedback from the field teams should be given due consideration, the performance of each arm should be evaluated empirically. Notwithstanding, the preference for Arm 1 (7-day recall) by field staff should be duly noted as operationalization aspects and respondent participation are of high importance to successfully conducting HIES.
Non-consumption related complementary modules Labour force: produced required data according to international recommendation. Food Insecurity Experience Scale: poor training and implementation, but important module for inclusion in HIES. Washington Group Short: highly complementary data for poverty disaggregation. Health: produces complementary data, but training resources necessary. Fisheries: produces complementary data, but training resources necessary.
Consumption-related complementary modules: Partakers Partakers (last 7- days) HH size Current Partakers Average household size (persons) arm_1 CAPI recall arm_2 CAPI diary arm_3 PAPI diary arm_4 PAPI diary arm_5 CAPI recall bounded Total 4.49 5.81 5.39 5.46 4.80 5.03 4.38 5.39 5.31 5.28 4.71 4.88 4.59 5.72 5.58 5.48 4.87 5.10 4.54 4.84 4.69
Partakers: impact on dietary energy consumption Average dietary energy consumption (kcal/cap/day) HH size Current Partakers Average dietary energy consumption (kcal/cap/day) arm_1 CAPI recall arm_2 CAPI diary arm_3 PAPI diary arm_4 PAPI diary arm_5 CAPI recall bounded Total 3,315 3,616 3,393 2,714 2,866 3,077 3,371 3,877 3,428 2,824 2,898 3,158 3,231 3,462 2,942 2,618 2,783 2,950
Consumption-related complementary modules: Stocks Performance of the food stocks module Low response (below table; number and percentage of households responding to the opening and closing food stock) Destocking (average net destocking (opening closing) over the 14 day diary period (kcal/cap) Effect on dietary energy consumption (next slide) Arm 2 Arm 3 Arm 4 OpeningFoodStock_Arm2&3&4 ClosingFoodStock_Arm2&3&4 Sample % no response closing stock % no response opening stock 61 59 64 58 48 75 114 93 181 63% 51% 14-days Daily net Average opening Average closing Average net 1,282 1,406 1,052 1,179 arm_2 CAPI diary arm_3 PAPI diary arm_4 PAPI diary Total -783 -1,032 -614 -742 498 374 438 438 36 27 31 31 95% 92% 77% 64%
Food stocks Mean DEC (kcal/cap/day) changes with and without stocks Recommendation that, should diary be used when conducting HIES, stocks are routinely collected; and the collection of stocks should be efficient (i.e., target main calorie sources and bulk-purchased food) and comprehensive for both opening and closing stocks With stocks Without stocks p-value arm_2 CAPI diary arm_3 PAPI diary arm_4 PAPI diary 3,462 2,942 2,618 3,018 2,594 2,178 0.00 0.00 0.00
Consumption-related complementary modules: Food Away from Home Yes it does matter as impact on the average DEC is quite significant . But not homogenous throughout the 5 arms . How FAFH is captured really matters
Food away from home If compare to ARM 2 our baseline, PAPI low supervision diary tends to under estimate* the average amount spent by households. Unbounded recall tends to better capture FAFH compare to bounded recall * After deletion of three extreme values observed in ARM 4
Food away from home Lunch seems to be the meals contributing the most to FAFH Recall seem to under report the overall share of FAFH in total energy consumed Well supervised diary performs equally well either when data is collected through PAPI or CAPI Bounded recall does not seem to perform better compared to not bounded recall ARM 1 ARM 2 ARM 3 ARM 4 ARM 5 Average contribution of FAFH to total energy consumed 9.3% 13.7% 12.7% 11.5% 6.4%
National accounts The diary-based survey asks open questions about purchased food, service items and home-produced products. This potentially allows for a more detailed classification but leads to a large workload to chart the results. In order to facilitate the recall of consumed products in past periods CAPI uses closed questions which do not allow for a grade of product details as found in the diary. For national accounts purposes both questionnaires, the recall as well as the diary, allow for a sufficient degree of detail according to COICOP, typically used in GDP and foreign account compilation. The same goes for Supply-and-Use tables used for balancing supply and demand side of economies.
Consumer price index The move to a full recall questionnaire inevitably involves the loss of some detail in the expenditure aggregates. However, we conclude that the granularity of the data produced by the recall questionnaires is more than sufficient for the purpose of rebasing the CPI. Furthermore, we find that there is no (obvious) evidence that the recall questionnaires significantly and consistently over or underestimate expenditures on specific product groups. We conclude therefore that the use of expenditure data for deriving CPI weights does not represent an obstacle for the adoption of recall-based expenditure surveys in the Pacific. However, we note that there is significant variation in the expenditure shares generated by the different surveys. We do not believe that this is caused by the different survey methodologies and think that instead it is largely the result of the comparatively small sample sizes utilised for the various experimental survey arms. Nevertheless, countries that use the survey data to rebase the CPI may want to choose to forego some of the savings involved from the adoption of the recall methodology to increase the sample size of their survey.
DEC FAFH has a significant impact on average DEC and it is recommended to add an individual section on FAFH in the HIES. Recall bounded does not seem to improve quality of the data collected (but results are subject to how ARM 5 was administered). Use of CAPI reduces non response and variability in the distribution. With more visits the average amount of energy reported is higher than when diary is less supervised.
Poverty: diary fatigue 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 high monitored PAPI low monitored PAPI high monitored CAPI mean mean mean
Poverty: stocks Noodles/pasta Ketchup and sauce Canned mackerel Oil Soya sauce Eggs Flour Chicken leg Sugar Rice Canned tuna -80% -70% -60% -50% -40% -30% -20% -10% 0%
Poverty: rates mean CI mean CI raking weights 10% 10% 8% 8% 6% 6% 4% 4% 2% 2% 0% 0% high low high CAPI 1 visit recall CAPI 2 visit recall high low high CAPI 1 visit recall CAPI 2 visit recall monitored PAPI diary monitored PAPI diary monitored CAPI diary monitored PAPI diary monitored PAPI diary monitored CAPI diary -2% -2%
Financial implication of diary v recall Diary-keeping surveys costs ranged from USD 551 to USD 3,261 per household in urban and rural locations respectively. The most expensive is the CAPI diary (Arm 2), because of the interviewer time taken to transcribe transactions from paper diaries into tablets. In contrast, single-visit recall cost USD 289 and USD 577 and a two-visit recall costs $USD 380 and USD 850 per household in urban and rural locations respectively. Thus, the two status quo survey designs (Arms 3 and 4), with pen and paper diaries, are between 1.9 and 5.2 times as expensive as a recall survey (Arm 1), while a diary with transcription into CAPI would be 5.6 times as expensive. Total cost per enumerati on area $4,626 $10,248 $8,331 $19,564 $7,624 $18,847 $7,918 $19,184 $6,031 $14,876 Total cost per household Indexed $289 $577 $1,666 $3,261 $1,220 $3,016 $551 $1,163 $380 $850 Arm Survey Type Location Urban Rural Urban Rural Urban Rural Urban Rural Urban Rural 1.00 1.00 5.76 5.65 4.22 5.22 1.91 2.01 1.31 1.47 1 1-visit 7-day recall, CAPI 2 7-visit 14-day diary, CAPI 3 7-visit 14-day diary, PAPI 4 3-visit 14-day diary, PAPI 5 2-visit 7-day recall, CAPI
Use of CAPI in HIES PSMB agreed that: CAPI is an appropriate means to capture data through HIES. Switching from paper to tablet-based collections constitutes a methodological break. The switch from collecting data using paper to using tablet-based forms therefore requires careful consideration. Should other methodological changes be made (e.g. changing from diary to recall to collect consumption data), they should be made once only and at the same time. PSMB recommended: Major methodological changes, such as switching from paper to tablet-based collection occur infrequently and that once the decision to make a major methodological change is made, the implementing PICT should maintain the new methodology for subsequent surveys so as not to break trends. Frequent methodological changes are discouraged.
Complementary survey modules PSMB recommended: Stocks: a Stocks Module is implemented when using a food acquisition diary in HIES; need for improved development, training and implementation. Partakers: a Partakers Module is implemented as a core HIES module. Food away from home: an individual-level FAFH Module is developed and implemented as a core HIES module. Economic activity: all PICTs note and manage series breaks and ensure that national-level reporting requirements and data needs are met; ILO and SPC develop a minimal version of this Labour Force Module as soon as possible, to be reviewed by PSMB, for inclusion in the 2020 round of population censuses in the Pacific. WG-6: module be included as a core HIES and census module. FIES: incorporated as a core HIES module; consideration of cultural adaptation of the module and development of Pacific-oriented resource materials, and improved training of field staff in implementing the module. Use value of durables: include questions in HIES to collect estimations of the use value of durables, given the application of this information to poverty measurement and national accounts. Anthropometric: identify alternative sources of anthropometric data; the inclusion of a module that collects anthropometric information is the decision of the implementing PICT.
