Understanding Big Data: Sources, Analysis, and Applications
Explore the diverse sources of big data, its analysis techniques, and its potential for statistical purposes, including SDG indicator development. Learn about the main sources of big data and the procedures involved in harnessing them effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
ECA United Nations Development Account 10th Tranche Statistics and Data Sub-Regional Workshop on Integration of Administrative Data, Big Data and Geospatial Information for the Compilation of SDG Indicators Analysis of Country Responses Analysis of Country Responses to the Self to the Self- -Assessment Questionnaire: Assessment Questionnaire: Big Data Sources Big Data Sources Haile Mulualem, ECA Addis Ababa, Ethiopia 23-25 April 2018
Big Data Big data are non-sampled data, characterized by the creation of databases from electronic sources whose primary purpose is something other than statistical inference (W. Horrigan, 2013). Analysis of big data often relies on techniques such as machine learning and data mining. Big data is characterized not only by its large Volume, but also by its Variety and the Speed at which it is generated.
Sources of Big Data In general, big data sources can be classified as follows:- Sources arising from the administration of a programme E.g., electronic medical records, hospital visits, insurance records, bank records and food banks. Commercial or transactional sources arising from the transaction between two entities E.g., credit card transactions and online transactions (including transactions from mobile devices)
Sources of Big Data Sensor network sources E.g., satellite imaging, road sensors and climate sensors. Tracking device sources E.g., tracking data from mobile telephones and the Global Positioning System (GPS). Behavioral data sources E.g., online searches (about a product, a service or any other type of information) and online page views. Opinion data sources E.g., comments on social media.
Procedures to investigate the potential of big data sources for statistical purposes, including for the development of SDG indicators Only 6 (of 22) countries responded that some procedures are in place to investigate the potential of big data sources for statistical purposes, including for the development of SDG indicators.
Main sources of Big Data Scanner data 1 Social media data 1 Smart meter electricity data 1 Road sensor data 1 Credit card data 1 Web scraping data 2 Public transport usage data 3 Health records 3 Mobile phone data 4 Satellite imagery or aerial imagery data 5
Involvement of NSOs in big data project relevant for compiling and/or supporting the measurement of SDG indicators 7 NSOs have involved in Big Data projects relevant for compiling and/or supporting the measurement of SDG indicators
Main/intended outcomes of ongoing big data project More in depth analysis for official statistics, in particular development of SDG indicators To analyze call record details of mobile phone users for the purpose of monitoring SDGs To add new data and complement the existing ICT statistics To drive development of data-based innovations To establish data governance institutional framework Improvement of data quality for locational and geo-referencing of statistics.
Established/Planned NSO Partnerships for big data projects 6 6 5 4 3 International Organizations Technology Partners Governmnet Institutes Research / Academic Institues Others
Estimation methods or a methodological framework for use of Big Data Sources Only 4 NSOs developed new estimation methods or a methodological framework specifically related to the use of big data sources. Compiling price data for CPI creation (Pilot phase) No further detail information provided from responding NSOs 3 of those NSOs use Traditional statistical methods while 1 NSO use Data visualization methods
Technologies and tools used in Big Data processing projects Data visualization tools , 1 Data mining tools , 1 GIS , 3 Spreadsheet , 2
National challenges in the use of big data sources in the production of official statistics, including for the development of SDG indicators 6 PERCEPTIONS OF BIG DATA BY STATISTICIANS 7 ACCESS COSTS TO DATASETS 8 LIMITED ACCESS TO DATASETS 11 INAPPROPRIATE OR LIMITED LEGAL FRAMEWORK 11 METHODOLOGICAL CHALLENGES 13 LACK OF TOOLS AND METHODOLOGIES HUMAN CAPACITY (NO APPROPRIATE SKILL TO ACCESS AND MANAGE BIG DATA 14
Plans to improve data collection from big data sources for statistical purposes Further exploitation on how to collect big data Planning on the use of big data in the NSS during NSDS preparation. Consultants supporting the work, with finance both from Governments and development partners Capacity building / trainings on big data Considering big data sources as additional/complementary source of data to be used for monitoring and evaluating development programmes.
ECA Thank You
