Guidebook for Managing Data from Emerging Technologies in Transportation
This guidebook explores the challenges and benefits of managing data from emerging technologies in transportation. It discusses the significance of big data, the need for a modern approach to data management, and offers a roadmap for agencies to transition towards this data management strategy. The project aims to develop a framework for effectively incorporating data from connected and automated vehicles, as well as new mobility initiatives, into decision-making processes. Additionally, it addresses the struggles faced by transportation agencies in handling large, diverse, and rapidly changing data sets, highlighting the importance of adapting to cloud-based architectures and hiring skilled data management professionals.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Guidebook for Managing Data from Emerging Technologies for Transportation Transportation Research Board Transportation Research Board National Cooperative Research Program Project 08 National Cooperative Research Program Project 08- -116 116 March 2020 March 2020
Overview of Presentation Overview of Presentation Project background and objectives Challenges to managing data from emerging technologies What is big data? Why should agencies move toward the modern approach to data management? How can agencies make the move toward this approach? Roadmap to Managing Data from Emerging Technologies for Transportation Supporting tools Guidebook for Managing Data from Emerging Technologies for Transportation 2
Project Background and Objectives Project Background and Objectives Research Objectives: Background: Develop a framework for managing data from emerging technologies, including data from connected and automated vehicles and data linked to new mobility initiatives. New, big, and varied datasets are available to transportation agencies at an increasing pace. These data have tremendous potential to offer new insights to transportation agencies. Outline a process for applying this framework to help agencies incorporate these data into the decision-making process. The volume, speed, and granularity of these data are unprecedented and will fundamentally alter the transportation sector. Guidebook for Managing Data from Emerging Technologies for Transportation 3
Transportation Agency Challenges to Managing Transportation Agency Challenges to Managing Data from Emerging Technologies Data from Emerging Technologies Reliance on traditional database management systems. Data from emerging technologies are too large, too varied in nature, and will change too quickly to be handled by these traditional data systems. Experience a loss of control to vendorsover data, technology, and service agreements. Are unequipped to handle this level of big data at an organizational level. Struggle to break down business unit and data silos. Do not fully recognize the value of big data or the eminent need to ready for it. Our big data issues are straightforward, we don t have the technology, money, or skills. CITY DOT Do not fully understand the uses and benefits of cloud- based architecture conducive to handling data from emerging technologies. Have difficulty hiring and retaining modern data management professionals. Guidebook for Managing Data from Emerging Technologies for Transportation 4
Big Data is More Than a Buzzword or Simply Lots of Data Big data may refer to data sets that are so vast and complex that they require new and powerful computational resources to process.1 Big data may encompass all the non-traditional strategies and technologies needed to gather, organize, process, and generate insights from large datasets.2 Big data is an approach to generating knowledge in which advanced techniques are applied to the capture, management, and analysis of very large and diverse volumes of data data so large, so varied, and analyzed at such speed that it exceeds the capabilities of traditional data management and analysis tools.3 What is Big Data? What is Big Data? Big data is a term that describes the large volume of structured and unstructured data that inundates a business on a day-to-day basis. But it s not the amount of data that s important. It is what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.4 Big data is a new attitudeby businesses, non-profits, government agencies, and individuals that combining data from multiple sources could lead to better decisions.5 1 Big Data. Dictionary.com, 2019. 2 Ellingwood, J. An Introduction to Big Data Concepts and Terminology, Digital Ocean, 2016. 3 Burt, Cuddy, Razo. Big Data s Implications for Transportation Operations: An Exploration, USDOT, 2014. 4 What is Big Data? What is Big Data, SAS, 2019. 5 Press, G. 10 Big Data Definitions - What's Yours? Forbes, 2014. 5
Traditional vs Modern Big Data Management Traditional vs Modern Big Data Management Managing data from emerging technologies requires a complete paradigm shift. These data cannot be handled simply by adding more hardware or processing power. The nature of the data demands an updated approach. Guidebook for Managing Data from Emerging Technologies for Transportation 6
With increased connectivity between vehicles, sensors, systems, shared-use transportation, and mobile devices, unexpected and unprecedented amounts of data are being added to the transportation domain at a rapid rate. These new data offer the potential to uncover insights to drive better decision- making at all levels of transportation agencies in a way that is simply not happening now. Why Should Why Should Agencies Move Agencies Move Toward the Modern Toward the Modern Approach to Data Approach to Data Management? Management? The potential value of these new data cannot be easily or efficiently extracted by traditional methods; the complexity of the task requires new big data tools and techniques. As data sources become more varied and change more and more rapidly, the traditional approach cannot cope with the complexity and cannot be re- designed quickly or cost-effectively enough to handle frequent data and business requirements changes. Modern big data methods to collect, transmit, store, aggregate, analyze, apply, and share these data need to be adopted by transportation agencies if they are to be utilized to facilitate better decision-making. 7
Provides a modern big data management framework that introduces new concepts and methodologies, best practices, and 100+ recommendations for managing data in a modern, flexible, scalable, and sustainable way. Lays out a roadmap on how to begin to shift technically, institutionally, and culturally toward effectively managing data from emerging technologies. This Guidebook This Guidebook Can Help Can Help Agencies Shift Agencies Shift Toward the Toward the Modern Data Modern Data Management Management Approach Approach Provides examples and references of transportation agencies currently exploring or already navigating the implementation of big data, including their challenges and successes. Discusses common misconceptions within the transportation industry regarding big data management. Laying the Foundation Contrasts traditional vs. modern approach for 11 characteristics of data systems Presents modern big data architecture Supporting Resources & Tools NCHRP 08-116 Research Report Data Management Capability Maturity Self-Assessment (DM CMSA) Data Sources Catalog Tool Big Data Governance Role & Responsibilities Frequently Asked Questions (FAQ) Supporting Resources & Tools Modern Big Data Management Framework 100+ recommendations across the data lifecycle Roadmap to Managing Data from Emerging Technologies 8-step process toward organizational change 8
Big Data Management Lifecycle and Framework Big Data Management Lifecycle and Framework The lifecycle defines the four major components of managing data throughout the entire lifecycle including the creation of data, storage of data, use of data, and sharing of data. Share Create The framework builds from these data management components to include big data industry best practices and over 100 associated recommendations for managing big data across the lifecycle. Use Store The framework should be applied throughout each step in the roadmap. Big Data Lifecycle Guidebook for Managing Data from Emerging Technologies for Transportation 9
Roadmap to Managing Data from Emerging Roadmap to Managing Data from Emerging Technologies for Transportation Technologies for Transportation Step 1 Develop an understanding of big data Step 2 Identify a use case and an associated pilot project Step 3 Secure buy-in from at least one person from leadership for the pilot project Step 4 Establish an embryotic big data test environment/ playground Step 5 Develop the big data project within the playground Step 6 Demonstrate the value of the data to other business units Step 7 Demonstrate the value of the data to executive leadership Step 8 Establish a formal data storage and management environment Guidebook for Managing Data from Emerging Technologies for Transportation 10
Supporting Resources & Tools Supporting Resources & Tools NCHRP 08-116 Final Research Report Framework for Managing Data from Emerging Technologies to Support Transportation Decision-Making, provided under separate cover, documents the research activities and provides supplemental information for reference to support implementation of the guidebook. Data Management Capability Maturity Self-Assessment (DM CMSA) offers over 100 questionsto allow agencies to gauge their data management practices, as well as identify areas for improvement. Data Sources Catalog Tool a tool to catalog existing and potential data sources. Big Data Governance Plan Template provides a list of recommendations to consider when developing a modern data governance approach, a description and frameworks for big data governance, and a tool for tracking the big data governance roles and responsibilities within an agency. Frequently Asked Questions (FAQ) responses to frequently asked questions regarding big data implementation, management, governance, use, and security. Guidebook for Managing Data from Emerging Technologies for Transportation 11
Step 1 Step 1 - - Develop an Understanding of Big Data Develop an Understanding of Big Data Step 1 includes information on the following: What is big data? Big data characteristics Big data concepts When to pursue big data Common misconceptions regarding big data Case study The Importance of Understanding Big Data Additional resources 12 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 2 Step 2 - - Identify a Use Case and an Associated Identify a Use Case and an Associated Pilot Project Pilot Project Step 2 includes guidance on the following: Selecting of a use case and pilot project that align with business unit, leadership, and organizational goals, including examples of drivers for change, example big data sources of interest, and associated example use cases and pilot projects Engaging others in the cause, including those internal to the business unit, cross- business unit, junior and mid-level staff, and external partners A case study on the Portland Urban Data Lake Pilot Project (PUDL) 13 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 3 Step 3 Secure Buy Secure Buy- -In from at Least One In from at Least One Person from Leadership for the Pilot Project Person from Leadership for the Pilot Project Step 3 includes guidance on the following: Establishing and communicating the value proposition for the pilot project, including example projects, value propositions, and questions to assist in developing the pitch Ways to create a sense of urgency and a fear of missing out (FOMO) De-risking the decision by identifying and communicating risks and other potential barriers up front Knowing how and when to make the pitch 14 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 4 Step 4 Establish an Embryotic Big Data Establish an Embryotic Big Data Test Environment or Playground Test Environment or Playground Step 4 includes guidance on the following: Establishing buy-in from IT, including understanding potential challenges and barriers, as well as the pros and cons of on-premise versus cloud storage Establishing the playground, including both the data storage layer and the data processing layer Taking ownership and responsibility for analytical projects Common misconceptions regarding big data storage A case study on storing data on-premise vs in the cloud 15 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 5 Step 5 Develop the Pilot Project Within the Develop the Pilot Project Within the Big Data Test Environment/Playground Big Data Test Environment/Playground Step 5 includes guidance on the following: Developing/ensuring the availability of the right expertise, including the pros and cons of various options (e.g., training/hiring in-house staff, trusted contractors and university partners, and big data experts/consultants) Developing the project by applying a data science perspective (e.g., collecting raw data, processing and cleaning the data, performing exploratory data analyses, building data science pipelines) Iteratively developing and improving the project and the associated outputs/data products Case studies on negotiating technical contracts for data services and building data knowledge 16 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 6 Step 6 Demonstrate Value of Data to Other Demonstrate Value of Data to Other Business Units Business Units Step 6 includes guidance on the following: Building support for the data and project across the organization, including other mid-level/branch managers that may have an interest in the data, project, and data products (or similar products) for their own business areas Using the data to tell the story of success by crafting a compelling story using understandable and persuasive visualizations that tie the insights uncovered in the data to the ability to address an issue or solve a problem of the business unit Getting others involved in sharing and using their data within the test environment, including iteratively expanding the use of the data to improved, enhanced, and new use cases A case study on iterative success and growth of big data within a transportation agency 17 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 7 Step 7 Demonstrate Value of Data to Executive Demonstrate Value of Data to Executive Leadership Leadership Step 7 includes guidance on the following: Presenting the success stories/business case to executives Continuing to build support, foster data sharing, and grow iteratively and incrementally Pushing for organization change/adoption of a formal big data environment A case study on buy-In from executive leadership 18 Guidebook for Managing Data from Emerging Technologies for Transportation
Step 8 Step 8 Establish a Formal Data Storage and Establish a Formal Data Storage and Management Environment Management Environment Step 8 includes guidance on the following: Establishing a clear vision and goals Making data accessible yet secure Integrating at the data level Using data to make decisions Merging existing projects into the same data infrastructure Continuing to seek input from other stakeholders and to iterate on evolving data governance plans and procedures Seeking continuous improvement by periodically reviewing and revising datasets, technology, processes and procedures, documentation, security and privacy protection, metadata catalog, etc. A case study on one transportation agency s continued room for growth 19 Guidebook for Managing Data from Emerging Technologies for Transportation
In Closing In Closing Whether an agency: Is starting from scratch with a new technology data set Is trying to make the business case for emerging technology data Is already working on a big data project Has an issue or problem that might be solved with emerging technology data Is looking for a new enterprise data management solution The steps and guidance outlined in this document are designed to walk them through the necessary data management policies, procedures, and practices to fully meet the needs of data from emerging technologies. Guidebook for Managing Data from Emerging Technologies for Transportation 20
Contact Information Contact Information For further information about the project, please contact: Kelley Klaver Pecheux, PhD Principal Investigator Kelley.Pecheux@aemcorp.com 21 Guidebook for Managing Data from Emerging Technologies for Transportation
