Streamlining Data Submission and Tracking for Glider Data Assembly Center
Simplify the process of submitting and tracking glider data sets at the Glider Data Assembly Center (GDAC) by acquiring a data provider account, obtaining a WMO ID, registering the data set, and ensuring compliance with metadata standards. Check the status of data sets, access real-time data via ERDDAP, and utilize improved tracking features provided by gliders.ioos.us.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
TRACKING IOOS GLIDER DATA ASSEMBLY CENTER (GDAC) DATA SETS John Kerfoot1, Ben Adams2, Don Moretti2, Kathleen Bailey3, Leila Bagdad-Brahim2, Matt Grossi4 1Rutgers University, New Brunswick, NJ, USA 2 RPS Ocean Science, South Kingston, RI, USA 3 IOOS Office, Silver Spring, MD, USA 4NOAA National Centers for Environmental Information, Stennis Space Center, MS, USA
DATA SET SUBMISSION PROCESS Acquire a data provider user account from glider.dac.support@noaa.gov Acquire a WMO ID for real-time data sets NDBC New rules allow for the same WMO ID to be used regardless of WMO region Persistent mapping of glider platform name to WMO ID Data sets with WMO IDs are released on GTS unless the data provider objects NC_GLOBAL:gts_ingest = True Register the data set: https://gliders.ioos.us/providers Process raw data stream to DAC-compliant individual profile NetCDF files Upload (ftp) the NetCDFs to the deployment data set end point
TRACKING DATA SET STATUS Is the data set registered? Is the data set available via ERDDAP? Contents Glider profile positions Geophysical (and other) variable? Is the data set metadata compliant? Has the data set been archived by NCEI and assigned an accession record? Data set marked as archivable by NCEI Data set marked as complete (i.e.: no more files to be uploaded)
TRACKING THE OLD WAY Request a WMO ID for new platforms Register the data set Submit files Wait and hope . Email glider.dac.support@noaa.gov or kerfoot@marine.rutgers.edu if the ERDDAP end point is missing
IMPROVED TRACKING https://gliders.ioos.us/status/datasets/
DAC DATASET STATUS WEBPAGE FEATURES Current Dataset Inventory Real-Time, Delayed Mode & Missing Dataset Status # Profiles Days Deployed Time Coverage Dataset Inspection & Visualization NCEI Archiving and Accession Records
DATASET SELECTIONS Dataset ID Search Narrow down selection results using all or pieces of the dataset id By data provider By glider/platform By WMO ID Identify missing/problem data sets Optional pagination
NCEI ACCESSION RECORDS Quick and simplified access to accession records Selection by data provider Selection by glider/platform Missing accession records One-click to the NCEI accession record/dataset Multiple end points for programmatic access to the accession record & search Highlight potential reasons for missing accession records
API Programmatic access (JSON responses) Optional pagination Searchable dataset status Daily averaged (smaller) and full resolution GeoJSON tracks NCEI accession records Dataset counts by data provider, glider/platform or WMO ID
RESOURCES FOR IMPROVING DATASET METADATA & DISCOVERY DAC NetCDF specification - https://ioos.github.io/ioosngdac/ Needs work - Currently being updated and simplified Alignment with IOOS Metadata Profile (where applicable) IOOS Compliance Checker - https://compliance.ioos.us Responsibility of the DAC to run the compliance checker on submitted files and report the results Web interface - https://compliance.ioos.us API - https://github.com/ioos/compliance-checker-web/wiki/API glider.dac.support@noaa.gov
COMMON DATA SET PITFALLS Missing NetCDF files Non-conformance to the DAC specification Single (unlimited) record time dimension Each NetCDF file contains a single profile Incomplete or missing summary clearly describing the mission objectives, program goals and measured geophysical variables NC_GLOBAL:wmo_id -> NC_GLOBAL:wmo_platform_code (IOOS Metadata Profile v1.2) platform_meta:wmo_id -> platform_meta:wmo_platform_code (IOOS Metadata Profile v1.2) NC_GLOBAL:gts_ingest = True | False Incorrect or missing VARIABLE:standard_name attribute value Incorrect or missing VARIABLE:units attribute value Missing attribution: acknowledgement of the funding sources and agencies that made the deployment possible
METADATA ISSUES PREVENTING NCEI ARCHIVING Incomplete or inadequate summary Syntax errors Missing institution(s) Missing project(s) Controlled vocabularies: Institutions Projects Instrument make/models
METADATA: EXAMPLE #1: NC_GLOBAL:SUMMARY BAD: Glider deployed in the North Atlantic Ocean GOOD: Glider ru23 deployed in the Mid-Atlantic Bight as part of the 2022 hurricane observation network. This glider will perform transects from the coastal shelf region offshore to the shelf slope break, with new waypoints determined to optimize sampling objectives. This real-time data set contains low-resolution temperature, salinity, dissolved oxygen and chlorophyll a profiles. The full-resolution data set will be provided to the IOOS Glider DAC following recovery and post-processing.
METADATA: DATASET/DEPLOYMENT ATTRIBUTION Comma separated list of funding sources and/or agencies Acknowledgement of funding sources and agencies Displayed on the Glider DAC Map https://gliders.ioos.us/map/
GLOBAL & VARIABLE ATTRIBUTES Multiple entries separated by commas NC_GLOBAL:institution = United States Navy, NAVOCEANO, NOAA, AOML Moving to the use of controlled vocabularies to standardize metadata instrument_ctd:type = CTD ; instrument_ctd:type_vocabulary = https://vocab.nerc.ac.uk/collection/L05/current/130; instrument_ctd:maker = Sea-Bird Scientific ; instrument_ctd:model = Sea-Bird SBE 41CP CTD ; instrument_ctd:model_vocabulary = https://vocab.nerc.ac.uk/collection/L22/current/TOOL0669/;
WHATS NEXT? Geophysical variable profile and time series plots GTS status How many observations have been released to GTS? Timing of released observations Reasons for profiles not being released Improve feedback on reasons for missing/orphaned data sets
DISCUSSION What can we do to enable/simplify the process of submitting data to the DAC? Proper metadata is tedious What are the data provider responsibilities? What are the DAC responsibilities? The DAC can fix/add/augment data set metadata with the ok from data providers NC_GLOBAL:summary instrument_XXX:attribute Improve application and speed of QA/QC algorithms Geographic limits Seasonal limits
THANKS! glider.dac.support@noaa.gov
