Lidar Working Group Data Submission Status - Continuing Measurements Report

The Lidar Working Group conducted a data submission status report on continuing measurements, including information on various sites, principal investigators, last archive comments, and data update statuses. The report covers a range of locations and parameters, highlighting the status of measurements and any updates needed.

• Lidar Working Group
• Data Submission
• Continuing Measurements
• Principal Investigators
• Data Update

mal_ren Follow

Uploaded on Sep 23, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. NDACC Data Host Facility Data Submission Status Lidar WG Documentation Status DOI, Data License, Data Versions DHF Transfer NCEP/CPC to NASA LaRC Data Retrieval Report Jeannette Wild, Roger Lin (UMD/ESSIC & NOAA/NCEP/CPC) Ali Aknan (SSAI & NASA LaRC) Gao Chen (NASA LaRC) Lidar WG meeting June 2021

2. Lidar WG Data Submission Status Lidar WG meeting June 2021

3. Lidar WG Data Submission Status Continuing Measurements (5/27/21) Site / PI Ames HDF Last archive Comments Eureka / Strong (O3) 93 09 17 19 2019 Was Red in 2018 Data gap 2010 2016 HDF new Eureka / Strong (T) 93 09 2009 Ny Alesund / Ritter (Aer) 91 19 2020 Was Black in 2018 Thule / Di Sarra, Di Iorio (T) 93 19 2019 Was Green in 2018 Hohenpeissenberg / Steinbrecht (T, O3) 87 21 87 21 2021 Payerne / Haefele (H2O) No change OHP / Jumelet, Khaykin (Aer) 91 20 2020 Was Black in 2018 OHP / Kechut, Hauchecorne (T) 91 20 2020 OHP / Godin-Beekmann (O3) 85 21 03 20 2021 OHP / Ancellet (Trop O3) 90 19 2020 Was Green in 2018 London, Ontario / Sica (T, H2O) 93 10 2012 No change KEY: Last Data submitted: > 2 yrs ago > 1 yr ago < 1 yr ago, but not up to date < 1 year ago & up to date Lidar WG meeting June 2021

4. Lidar WG Data Submission Status Continuing Measurements (5/27/21) Site / PI Ames HDF Last archive Comments Rome / Liberti (H2O) 03 - 10 2018 Was Red in 2018 Huntsville / Newchurch, Kuang (Trop O3) 15 20 2020 Was Black in 2018 TMF / Leblanc (Aer) 89 13 04 21 2021 Was Black in 2018 HDF new TMF / Leblanc (O3) 89 18 18 21 2021 HDF new TMF / Leblanc (T) 89 13 18 21 2021 Was Black in 2018 HDF new TMF / Leblanc (Trop O3) 99 18 18 21 2021 HDF New TMF / Leblanc (H2O) 10 21 2021 Was Blue in 2018 Mauna Loa / Leblanc (Aer) 93 18 04 21 2021 HDF new Mauna Loa / Leblanc (T, O3) 93 18 99 21 2021 HDF new Mauna Loa / Barnes (Aer) 94 20 2020 HDF new Mauna Loa / Barnes (H2O) No change KEY: Last Data submitted: > 2 yrs ago > 1 yr ago < 1 yr ago, but not up to date < 1 year ago & up to date Lidar WG meeting June 2021

5. Lidar WG Data Submission Status Continuing Measurements (5/27/21) Site / PI Ames HDF Last Archive Comments Reunion Maido / Portafaix (O3) 13 17 13 2018 Was Green in 2018 Reunion Maido / Keckhut, Hauchecorne (T) 13 20 2020 Reunion Maido / Keckhut, Duflot (H2O) No Change Reunion Maido / Duflot (Trop O3) 13 19 2019 Was Green in 2018 Lauder / Querel (O3) 94 21 17 21 2021 HDF new Dumont D Urville / Jumelet, Snels (Aer) 89 20 2020 Was Black in 2018 Dome C / Snels (Aer) 14 20 2021 Improvements: 9 Red to Black Black to Blue Black to Green Blue to Green Decline: Green to Blue Green to Black 4 1 3 2 1 5 1 KEY: Last Data submitted: > 2 yrs ago > 1 yr ago < 1 yr ago, but not up to date < 1 year ago & up to date Lidar WG meeting June 2021

6. Lidar WG Data Submission Status Campaign Measurements (5/27/21) Site / PI Ames Comments Ny Alesund / McGee (STROZ) 1997, 1998 Sodankyla / McGee (STROZ) 2006, 2007 Hohenpeissenberg /McGee (STROZ) 2005 OHP / McGee (STROZ) 1992, 1997, 2017, 2018 New Data archived in Aug. 2018 Greenbelt / McGee (STROZ) 1993 2001 Should there be 2008, 2009? Greenbelt / McGee (AT) 1997, 2007, 2008, 2009 Table Mountain / McGee (STROZ) 1992, 1997, 2004, 2005, 2011 Should there be 2009? Table Mountain / McGee (AT) 2005 Should there also be 2006, 2007 & 2009? Mauna Loa / McGee (STROZ) 1995 - 1996, 2002, 2012 - 2013 Should there be 2014? Mauna Loa / McGee (AT) 2013 2015 Archived in Sept 2018 Reunion-Maido / McGee (STROZ) 2015 Archived in Aug 2018 Lauder / McGee (STROZ) 1992, 1994, 1995, 2001, 2002, 2011 Should there be also 2012? Polarstern / Schrems 1996, 2000, 2003, 2005 Prestwick / Schrems 2000 Lindenberg / Schrems 2003 Table Mountain / Schrems 1997 Paramaribo / Schrems 2004, 2006 Punta Arenas / Schrems 2000 London / Whiteman No data Beltsville / Whiteman (H2O) No data Lidar WG meeting June 2021

7. Lidar WG Documentation Status Lidar WG meeting June 2021

8. More About Documentation Visibility: With redesign of NDACC Web Pages the dataset level Metadata Files AND M&A Directory entries are now easily discoverable by the user. Each station has a page at www.ndacc.org List of Instruments Lidar WG meeting June 2021

9. More About Documentation Tabs on right access Documentation Tabs on left access Data One click to metadata file One click to data directories HDF Ames One click to M&A Directory entry Submit new metadata files and M&A Changes via email to DHF (Jeannette.Wild@noaa.gov). Lidar WG meeting June 2021

10. Metadata Status Continuing Measurements An amazing 18 new files since last Lidar WG meeting! Site / PI File Year at 2018 meeting File year at 2021 meeting Still 5 files substantially out of date. Eureka / Strong (T, O3) None 2020 Ny Alesund / Ritter (Aer) 1999 2019 Current files are in http://ftp.cpc.ncep.noaa.gov/ndacc /meta/lidar/ Thule / Di Iorio (T) 1996 2019 Hohenpeissenberg / Steinbrecht (T, O3) 2017 2019 Payerne / Haefale (H2O) None 2020 OHP / Khaykin (Aer) 2009 2009 OHP / Hauchecorne (T) 2009 2019 Email updates to Jeannette.Wild@noaa.gov OHP / Godin-Beekmann (O3) 2001 2001 OHP / Ancellet (Trop O3) 2017 2017 London, Ontario / Sica (T, H2O) None 2019 Site / PI File Year at 2018 meeting File Year at 2021 meeting Rome / Liberti (H2O) None 2019 GSFC/ AT / McGee / Aer, T 2000 2000 Reunion Maido / Portafaix (O3) None 2019 GSFC STROZ / McGee / O3, T 2000 2000 Reunion Maido / Keckhut (T) None 2019 Huntsville / Newchurch (Trop O3) None 2019 Table Mountain / Leblanc (T, O3, Aer) 2006 2019 Reunion Maido / Keckhut (H2O) None 2019 Table Mountain / Leblanc (Trop O3) 2006 2019 Reunion Maido / Duflot (Trop O3) None 2019 Table Mountain / Leblanc (H2O) None 2019 Lauder / Querel (O3) 2017 2017 Mauna Loa / Leblanc (T, O3, Aer) 2006 2019 Dumont D Urville / David, Jumelet (Aer) 2009 2009 Mauna Loa / Barnes (Aer 532 nm, H2O) 2010 2019 Dome C / Snels (Aer) 2017 2017 Lidar WG meeting June 2021

11. DOI, Data License, Data Versions, DHF Transfer Lidar WG meeting June 2021

12. DOIs for NDACC data providers are being published by EVDC Landing page is at EVDC Supply information in headers below including: Data Title, Format, License, Description, etc Will link to dataset at the NDACC DHF Contact Jeannette Wild for more info When created, DOI and license information should be added to data files and metadata files Lidar WG meeting June 2021

13. Data License and Data Versions Data License: Choose between 3 suggested licenses for NDACC (PIs must decide for their data): CC0 (most open credits not even (legally) required ) - in part. for US gov t data CC-By-SA (4.0) license (open but credits required) CC-BY-NC-SA license (not open for commercial use) When chosen, Data License information should be added to data files and metadata files Choose carefully. Know restrictions by a journal if you are creating DOI for a journal. ESSD for example requires CC-BY (4.0) or less restrictive. Contact your funding organization for guidance. Once published, there cannot be a change from less restrictive to more restrictive. Data Versions: Optional ability to augment DATA_SOURCE to include simple version number Optional ability to augment DATA_SOURCE with text descriptor/keyword (WEEKLY, CENTRAL) Document at www.ndaccdemo.org/sites/default/files/docs/NDACC_Data_Versioning_2_3_Main.pdf Each WG proposes Keywords which will be centrally managed at GEOMS/DHF level. Some approved Keyword examples: - HIRES (Hi-Res) - Monthly - GLASS (algorithm name) - Can include an algorithm or reprocessing version number. groundbased_lidar.aerosol_nasa.jpl002_glass.1.1_mauna.loa.hi_20040602t063747z_20040602t083820z_001.hdf Traditional Data Source Reprocessing version number (after . ) Algorithm name (After _ ) Lidar WG meeting June 2021

14. NDACC DHF Transition NOAA/CPC to NASA LaRC NDACC Entities at NOAA/NCEP/CPC NDACC DHF Database server (wwwdb) not accessible outside of CPC DHF managers - houses the relational database RDB Data ingest area ftp site for public data Tools area www.ndsc.ncep.noaa.gov/PI (interface to wwwdb RDB) NDACC Website www.ndacc.org (and development instances) - links to NDACC DHF, but does not contain data Secure (password protected) for NDACC Steering Committee NDACC DHF transition phases Phase 1 duplicate DHF functionality at LaRC Format check & Mirror of data Data ingest Data download External mirrors and downstream databases, metadata sharing Phase 2 Establish full relational database at LaRC Enable search engines Enable NDACC data tools Phase 3 Relocate NDACC website Lidar WG meeting June 2021

15. NDACC DHF Transition NOAA/CPC to NASA LaRC Phase 1 duplicate DHF functionality at LaRC Format check testing of ingest took sometime to fully test and develop Mirror of data & metadata - https://www-air.larc.nasa.gov/missions/ndacc/ - NDACC Station Data has been mirrored at LaRC since summer 2020. - Long-term mirror period tested that format checking codes accepted equivalent files - MUSICA, RD, NCEP, GMI, and metadata files are all now at LaRC Data ingest - https://www-air.larc.nasa.gov/cgi-bin/NDscan - CAMS 27 additional checks enabled for files submitted at LaRC - All data submission accounts transferred to LaRC as of 5/1/2021; no ingest at CPC. - NIWA, JPL, CNRS all successfully submitting Lidar files to LaRC - IP identification only used for data submission by script - Account / password web interface can be used from any location Single or multiple file upload Test format only or check and archive Lidar WG meeting June 2021

16. NDACC DHF Transition NOAA/CPC to NASA LaRC Phase 1 duplicate DHF functionality at LaRC (continued) User Manuals emailed to all data providers at account set up Data download - https://www-air.larc.nasa.gov/missions/ndacc/ - Web interface or script enabled for public and private External mirrors and downstream databases, metadata sharing - CEDA, NILU/EVDC mirrors, partnerships with BIRA, WOUDC, C3S, ACTRIS - Completion goal 6/30/2021 Lidar WG meeting June 2021

17. NDACC DHF Transition NOAA/CPC to NASA LaRC Phase 2 Establish full RDB at LaRC Enable search engines (www.ndsc.ncep.noaa.gov/PI) Goal by mid-2022 Enable NDACC data tools Phase 3 Relocate NDACC website including password protected SC Space www.ndacc.org All NDACC documents, news, M&A, Station tables, etc. Goal several years out Lidar WG meeting June 2021

18. Data Retrieval File Counts Lidar WG meeting June 2021

19. NDACC Data Retrieval NDACC Transfer Count Monthly Average 180000 160000 140000 # of File Transfer 120000 100000 80000 60000 40000 20000 0 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Year Lidar WG meeting June 2021

20. NDACC Data Retrieval # files in database Instr. 2019 # file transfers # files Web (Private) 2018 2019 Anon ftp 2019 2020 2020 2018 2020 537 9 2196 2365 426 426 2640 Bksnde Brewer Dobson 2516 7331 375 6124 17378 26930 43805 3560 48437 1136 158018 180041 126616 2610 8459 375 6844 20026 34866 45053 3692 56444 1246 2036 6933 720 6199 18884 41397 38870 3846 5235 1959 0 9 1 18403 50909 1822 43857 106076 135123 311013 17145 307776 6409 16597 54177 1628 59966 131482 139221 297519 21503 299341 8311 17823 45444 1342 54302 116472 135488 299949 17204 212064 6113 908566 45 0 36 Dustsonde FTIR Lidar 208 1327 523 4965 10068 Mwave 22 O3sonde 3143 15 1208 Spt UV 3 UV Vis 276 WVsonde 2 Total 7186 14674 1001173 1031941 Lidar WG meeting June 2021

21. Top 2020 Data Users: Rank 11 Rank Count Institution Count Institution 1 206037 University of Science and Technology, China University of Toronto Comcast (broadband, USA) 16307 USAFL, NASA 12 13 2 3 134922 134919 13432 13142 Elisa (broadband Estonia) Vodafone, (Broadband, Italy) Karlsruhe Inst. of Tech., Germany OHP, France Obs. Met. Aeronomy (OMA), Brussels DWD, Germany U Bern, Germany NASA Obs. St Michel, France 4 5 6 7 8 9 10 80238 75686 63488 44448 18343 17103 17102 LaRC, NASA EVDC, NILU Jussieu, France New Zealand BIRA, Belgium Gprs EE ECMWF 14 15 16 17 18 19 20 12244 10397 10266 6948 6855 6396 5426 Top 2019 Data Users: Chinese Acad of Science(249590), Rutherford Appleton (152526) Top 2018 Data Users: Bira, Belgium (263667), NASA GSFC (257647) Top 2017 Data Users: Telecom, Netherlands (403659); Obs Midi-Pyrenees (35447) Top 2016 Data Users: Univ. Leeds (120528); US EPA (111587) Lidar WG meeting June 2021