Enhancing Data Quality: NDBC's Improvement Project
NDBC is undertaking a project to overhaul its quality control processes, algorithms, and station setups to align with technological advancements and support increased observations. The project aims to produce, disseminate, and archive high-quality observations, improve consistency and transparency, collect necessary metadata, manage quality assurance procedures, and enhance the usability of NDBC observations. Specific objectives and deliverables align with overall project goals, with the first phase focusing on correcting data quality issues and implementing corrections to improve system metadata.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dawn Petraitis & Karen Grissom NOAA National Data Buoy Center May 22, 2018
Observation Growth (2005- 2017) NDBC: 16% IOOS: 181% NOS: 172% O&G: 13%
To stay aligned with changes in technology & support increasing number of obs, NDBC requires an overhaul of its QC processes and procedures a data quality improvement project. The project shall revise, and in some cases redesign, the NDBC quality control system, including computer program and algorithms, QA/QC procedures, and station setup. The improvement will directly benefit NDBC systems of record, and indirectly benefit NDBC partner stations IOOS and NOS.
1. Produce, disseminate and archive high quality observations as defined by NDBC requirements for observation accuracy and resolution. 2. Ensure consistency and transparency of processing methods. 3. Collect, record and archive all metadata necessary to realize the full potential of the observations. 4. Establish process for managing quality assurance, quality control procedures and algorithms and archiving procedures. 5. Improve confidence and usability of NDBC observations.
The expected deliverables of this project fit into one of two categories, each will have its own specific objectives: Adjustment to existing observations and metadata to ensure the integrity of the observation; Improvements and enhancements to our processes, procedures, and algorithms. o o
Business Case/Problem Statement Business Case/Problem Statement Project Goals Project Goals Deliverables Deliverables Specific of the project that align with the overall project goals. Specific objectives are developed with each phase of the project that align with the overall project goals. objectives are developed with each phase Requirements (Problem or Enhancement) Requirements (Problem or Enhancement) Implementation Implementation Objectives Objectives Testing Testing Production Production Documentation Documentation
The objectives for the first phase of the project (FY18) were chosen based upon need (most urgent), ability to implement, and contribution to overall goals of the project. Using these criteria, four objectives were selected for completion during the FY18 phase: 1. Correct and prevent problems associated with lower quality data included in historical files. 2. Implement correction to Smart Module timestamp irregularity. 3. Ensure metadata for SCOOP system is included Weather Buoy Archives process. 4. Scope requirements for FY19 phase (Weather Buoy Archive processes and develop capacity to manage eulerian observations at the same level as existing NDBC observations).
Developed a weekly report on the auto QC results Takes the place of the old IOOS/NOS manual QC report Provides a summary of flags for each IOOS RA Shows details of what was flagged for each station http://www.ndbc.noaa.gov/ioosstats/ioos_auto_qc.shtml
Monthly report sent to RA directors and staffs detailing the data availability based on GTS messages Restricted to met parameters and wave height/period
Monthly report containing the number of observations sent to the GTS General categories of met and wave Met consists of wind speed, wind direction, sea level pressure, air temperature, dew point temperature, and water temperature Wave consists of wave height and dominant period http://www.ndbc.noaa.gov/ioosstats/
Monthly and quarterly reports on the number of web hits each partner station receives Pages, Data Requests, and RSS Requests Utilizes statistics from Webalyzer Planned future work includes stats from Google Analytics Allow for filters and groups More granularity on web metrics http://www.ndbc.noaa.gov/partnerstats/
Questions? Questions? Dawn Petraitis dawn.petraitis@noaa.gov or 228-688-2940 Karen Grissom karen.grissom@noaa.gov or 228-688-1325 Thank you Thank you
