Managing Data Integration: A Management Perspective

The data universe is expanding rapidly, leading to challenges in organizing and extracting valuable information. Dr. Patricia Staab discusses the importance of integrating data using SDMX at the SDMX Global Conference in Budapest. The approach involves moving towards an application-driven architecture, integrating data of high relevance through coordination, ontologies, and standardization. This data-centric strategy enables logical centralization and a uniform language for describing and treating data, enhancing standardization and integration efforts.

• Data integration
• Management perspective
• SDMX
• Data universe
• Information management

utes Follow

Uploaded on Sep 13, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Measuring the data universe: A management perspective on data integration using SDMX SDMX Global Conference , Budapest, September 2019 Dr. Patricia Staab, Statistical Information Management, Deutsche Bundesbank

2. The data universe is exploding Data amount is growing constantly and rapidly Automatic recording of process data (sensors, IoT) Social networks, smart phones and tablets Growing "numbermania More computing power, new analysis techniques However: Data is not information *) Yawning Data Gaps despite Collectomania Using IT not Possible Without Content-Related Expertise The Data Universe lacks Order Source: www.stratio.com Vision: A well ordered map of the starry sky of information Measuring the Data Universe *) Data is not information, information is not knowledge, knowledge is not understanding, understanding is not wisdom. Clifford Stoll 17/09/2019 2

3. The approach so far: Moving towards an application driven architecture Silo of BI Product C Silo of Silo of Data Science A Data Science D Silo of BI Product A Silo of BI Product B Silo of Data Science C Silo of Data Science B Source: R. Stahl, P. Staab, Measuring the Data Universe. Springer; 1st ed. 2018 (28. Mai 2018) Measuring the Data Universe 17/09/2019 3

4. A different, data centric approach: Integrating the data of high relevance Coordination Ontologies, Global IDs Standardization SDMX, DDI Semantic harmonization IT, Technology DWH, BI Projects Uniform data modeling method Order system The concepts, methods and codelists used for the classification of the data are the same. Logical Centralization Thus linking the data, the actual integration of content, becomes possible. A uniform language (the same concepts and terms) is used to describe the data. Thus a rule-based (and automatable) treatment of the data becomes possible. The data is stored (physically or virtually) in a common system. Common procedures can be used for administration, authorization and access. Ready to be linked Increasing degree of standardization Source: R. Stahl, P. Staab, Measuring the Data Universe. Springer; 1st ed. 2018 (28. Mai 2018) Measuring the Data Universe 17/09/2019 4

5. A different, data centric approach: Integrating the data of high relevance intelligent Data Warehouse simple Data Warehouse Data Lake Source: R. Stahl, P. Staab, Measuring the Data Universe. Springer; 1st ed. 2018 (28. Mai 2018) Measuring the Data Universe 17/09/2019 5

6. Bringing it all together: Data and systems landscape Datei:EZB-Geb ude in Frankfurt (Main).jpg A beautiful house by the lake A beautiful house by the lake Source: https://de.wikipedia.org/wiki/Datei:EZB-Geb%C3%A4ude_in_Frankfurt_(Main).jpg Measuring the Data Universe 17/09/2019 6

7. Bringing it all together: Data and systems landscape Datei:EZB-Geb ude in Frankfurt (Main).jpg Casual users Data Warehouse eg Bundesbank House of micro data Raw data from internal systems Business analysts Standardization eg SDMX Data Lake Big Data applications, advanced analytics Data science, research External data sources Company Data Center Measuring the Data Universe 17/09/2019 7

8. Example: Deutsche Bundesbank Central Statistics Infrastructure Bundesbank Central Statistics Infrastructure Data Content (February 2019) 160 mio time series (150 mio internal) in 450 data sets (210 internal) No. of time series (mio) 500 500 Integration Pipeline for House of Microdata in 2019 ESCB Centralised Securities Data Base: 350 mio time series German Securities holdings statistics: 12 mio time series Other 400 300 200 160 100 37 Over 1.500 active users of which 200 per day 26 10 3.5 0 10.000 downloads per day 1 mio time series downloaded per day Multiple sources (statistics, supervision, markets, cash, ) International organisations, commercial data Bundesbank House of Microdata Measuring the Data Universe 17/09/2019 8

9. SDMX for Microdata - Experiences of ECB & Bundesbank Measuring the Data Universe 17/09/2019 9

10. Workstream SDMX for Microdata from the SDMX Roadmap 2020 Resulting document: Design of data structure definitions for microdata Report of Experiences from the European Central Bank and Deutsche Bundesbank General challenges of Microdata (Volume, Confidentiality, Master Data, Reference Metadata, Back Data Revision Mechanisms) DSD specific challenges (Multiple Measures, un-coded concepts, exploding code lists, groups) DSD Design Principles for Microdata (keeping the same approach as for macrodata, balancing number of DSDs regarding optimum fit vs. redundancy and integrity) Easy-To-Use Formats (especially SDMX-CSV, SDMX-JSON) Use Cases (Bundesbank House of Microdata, AnaCredit) Measuring the Data Universe 17/09/2019 10

11. Example 1: Use Case House of Microdata Money Market Statistical Reporting Key dimensions Reporting agent Market segment Reference date Transaction identifier Frequency describe Money Market Statistical Reporting Nominal amount of the transaction Around 25 attributes with detailed information on the transaction E.g. interest rate, Proprietary Transaction Identifier (PTI), Legal Entity Identifier of the counterparty (LEI), maturity Measure Attributes Measuring the Data Universe 17/09/2019 11

12. Example 2: Use Case AnaCredit (Collection of microdata on credits on a loan-by-loan basis from Euro NCBs) BBK Internal BI-System SDMX-CSV SDMX-ML (Flat format) SDMX-ML (Flat format) Reporting Agent BBK AnaCredit ECB AnaCredit ECB uses the SDMX 2.1 flat format (where all dimensions appear at observation level) Bundesbank follows this approach for the domestic Bank s primary reporting without using a DSD reporting agents can manage their reporting obligations without having to handle SDMX concepts for internal interface to the BI Systems use of SDMX-CSV format Measuring the Data Universe 17/09/2019 12