Sustainable Development of C3S Data Rescue Portal
Develop and maintain the C3S Data Rescue Portal with a global registry for data rescue activities, focusing on registering metadata, providing searchable inventories, and supporting DARE activities. The project aims to update inventories, facilitate data uploads, and offer user support and technical documentation. Tasks include determining metadata standards, developing the Global Registry Portal, and incorporating large databases. Collaboration with various institutions and adherence to metadata standards are key elements of this initiative.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
C3S Data Rescue Service C3S311a_Lot 1 WP2: Data Rescue Registry Services Main objectives: Develop and maintain in a sustainable way the part of the C3S Data Rescue Portal that contains the global registry for data rescue activities, by registering metadata information and supplying extensive searchable and traceable metadata inventories, location plots and other useful information about in-situ observations. Update periodically the inventories with new metadata from the large global databases. Facilitate the upload of metadata/data of DARE activities to the large global databases. Provide user support, technical documentation and guidance documents.
WP2 Overview Participants: FCI NCIAS.ID - Associa o para a Investiga o e Desenvolvimento de Ci ncias WP2 Leader (Maria Ant nia Valente, Research Fellow#1-SSE, Research Fellows#2 and #3) - (89 PM = 16 + 32 +32 + 9) MOHC -Meteorological Office Hadley Centre (Rob Allan, SSE) - (19.7PM = 3 + 16.7) KNMI - Koninklijk Nederlands Meteorologisch Instituut (Peter Siegmund) (3 PM) FURV - Universitat Rovira i Virgili (Manola Brunet, Senior Scientist) - (16 PM) Clive Wilkinson (7 PM)
WP2 Overview Tasks: 1.2.1: Determining the standards of metadata information in the registry and its format (September 2017) 1.2.2: Development of the Global Registry Portal (Prototype December 2017) 1.2.3: Incorporating inventories of large databases 1.2.4: Provide facilities for submitting digitised metadata/data sets to international climate data archives (Prototype Dec 2017) 1.2.5: Support activities (Docs December 2017, blog Feb 2018, how-to-video November 2017)
1.2.1: Determining the standards of metadata information in the registry and its format (M6 September 2017) The registry will cover land surface, upper air and marine metadata, at least, (plus Terrestrial). (contacted Dick Dee, who said it should include upper air and all types of relevant historical metadata) Metadata information based on the framework provided by GCOS, and WIGOS including the concept of Essential Climate Variables (ECVs). Dick Dee said we should include more than the ECVs in GCOS and that some GCOS ECVS have only started to be observed recently. We ll include in the beginning all ECVs we have in ERA-CLIM2, MEDARE and ICOADS and expand from there, as we upload inventories. Metadata information displayed on the registry tables will be selected to be as complete as possible, taking as guide the WIGOS Metadata standards. The standards regarding metadata have to be consistent through all WPs, including WP3 (naming of ECVs) The data policy surrounding conditions of onward use are a very important part of any metadata collection Ensuring consistency with the WP1 portal listing and pull through to both the global archives/Lot 2 data holdings DC3S311a_Lot 1.2.1 Guidelines for inventory metadata standards and formats (M6).
ERA-CLIM2 Registry eraclim-global-registry.fc.ul.pt Metadata standards and format to be settled by M3 ideally (all partners) enter with user/password anonymous/anonymous
Inventories format suggestion: use an extended version of MEDARE format (one ECV per row) for uploading which can be transformed/summarised in the ERA-CLIM2 format (all ECVs in one row) 2 formats
Land surface inventories should contain at least: - numbering of entries: unique_metadata_record_identity - include WMO number if it exists, national network number? (numbering of historical stations without numbers we should have a number for them) (pay attention to stations that close and slip under the radar and lose their numbers) - what location information will be in the inventory (station name, place, country, lon, lat, alt) - lon and lat will have no limit to decimals. Decide standards for lat, lon. (Rob Allan to look at standards for stations locations) - do we include opening/closing date of station? Yes - include starting and ending day/month/year of record. - data periodicity (sub-daily, daily, monthly, annual, etc) - indicate change of location - indicate gaps in data at monthly or annual level if daily/subdaily, annual if monthly or anual data -decide which ECVs will be indicated in the inventory and their names (according to WMO rules). Manola to look at standard ECVs for Atmospheric Land and Ice, Surface (Plus relevant historical Terrestrial ones ex: soil moisture, snow cover) (we also have surface ozone, is it Terrestrial?) -main metadata changes (instrument change, change in procedures, observation times - local to UTC other)
-Data in more than one unit of measure (add another row?) -Pressure data should include information on gravity corrections and conversion to 0 C - do we include information on the specific instrument used for each ECV? Yes!!! - indicate stage of DARE and QC (hardcopy, imaged, digitised raw format, digitised formatted, quality controled, homogenized, if included in Lot 2 database/other databases) - indicate if data is free access, restricted, etc. (all data in Lot 2 is free access, data registered in metadata registry may become free after being restricted appeal to persuasive powers of advisory board) - data source publication (with possible link to images or online library, etc.) - data owner (website owner and/or e-mail address) - data holder (link to digitised data, website, and/or e-mail) -link to C3S Data Rescue Portal where DARE project details are stored - button to plot station(s) on map - Snow metadata is to be included in land surface inventory (Antonia to look at FMI inventory)
Fixed Upper air stations (Definitely included in registry) Almost similar to Land surface stations inventory - Include type of platform (kite, pilot balloon, radiosonde) - Indicate if observations are in pressure or height levels - indicate if significant levels are included - Antonia to take care of Upper air inventories ECVs Moving platforms: Upper air moving platforms (see ERA-CLIM2 inventories) Could be upper air data sent from ships moving ships (cross between marine and upper air metadata) Marine platforms (see ERA-CLIM2 inventories and discuss) Clive Wilkinson s domain What marine inventories exist and how are they presented? Rob and Clive to investigate Is there an ICOADS inventory? Clive s domain WIGOS Metadata Standard publication is very useful for deciding the information to be included
1.2.2: Development of the Global Registry Portal (M48) (developed mostly by SSE MOHC and FCi ncias.ID according to C3S standards) 1.2.2.1 Portal access and inventories Basic software for the registry will be designed: - making use of the ERA-CLIM2 Registry and MEDARE inventories structure - Access to the registry (via C3SDARE link) for consulting (free access) and uploading inventories (registration) 1.2.2.2 Searching tools and inventories output In conjunction with WP1 1.2.2.3 Visualisation tools and plot output 1.2.2.4 Development of metadata submission tools Feedback with WP1 1.2.2.5 Development of QC tools for submitted metadata In conjunction with WP3 1.2.2.6 Maintenance and management First prototype to be delivered by M9 December 2017 (until last working day of month) Annual updates December 2018,2019,2020
Portal software and contracting SSEs: MO - SSEs will decide the software to be used FCi ncias SSE will follow in the MO decisions footsteps. Contracting FCiencias SSE dependent on opening cost centre, which is pending subcontract agreement negotiations and signature Call for fellowship ready to be launched, jury will include one element of FCUL Informatics Department FCUL-ID (Luiz Moniz). Candidates from FCUL-ID will be cajoled with C3S Data Rescue WP2 work plan summary after Easter. Only applicants with Bachelor or Master Degree can be admitted. Work could lead to a Master dissertation. Publication of work envisaged. Constant contact with MO-SSEs (mails and telecons) and eventually Copernicus SSEs.
Portal special software capabilities: -Has to be able to create tables from a database with hundreds of thousands of entries (we are using phpMyadmin, but can be another one as long as it is compatible with our Informatics Department systems here). - Needs to be able to add columns and rows to the inventory tables. This is crucial. - Inventories must have observations locations (lon,lat) in order to visualise them on a global map. - This visualisation software must be chosen by the MO SSE, maps have to have updated countries borders and allow for the plotting of thousands of locations (with a dot, for instance). - The software must allow for single/multiple/crossed searches, with the result being shown on a table and plottable on a global map (with zoom capabilities). Result of searches must be exported to Excel/Ascii/other formats tables and resulting maps to image files (jpeg, gif, etc). - Browsing of the registry can be done by all, but a counting mechanism of viewers must be developed. - Registry will upload inventories from registered users with editing permissions. - Uploaded inventories will ideally be in the agreed format being decided by WP2 group. - The medatada uploaded to the registry will be quality controlled by QC tools developed in WP3 and available in the main C3S Portal. - Layout to be developed by MO SSE, using the C3S logo and requirements (contact C3S personnel to obtain these and adjust to their requests). - Registration to be done in the main C3S Portal - Regular and automatic 2-way exchange of information with main C3S Portal about new projects and inventories (discuss with WP1 - Peter Siegmund). - Robust: 24/7 availability - Maintenance done by FCi ncias.ID SSE, as well as some of the software developments (to be agreed with MO SSE). - Compatibility between software chosen by MO SSE and FCi ncias.ID Informatics Department capabilities is crucial. We have to be able to exchange software that works both at MO and FCi ncias. We will have a mirror site. Discuss agreed registry capabilities with MO SSE.
1.2.3: Incorporating inventories of large databases (M48) Global Registry starts with updated ERA-CLIM2, MEDARE, ISPDv4 inventories Upload inventories: (a) I-DARE, ACRE and IEDRO, in collaboration with KNMI and MOHC (b) Updated ISPD inventories (talk to Gil and Yin about double units) to the latest version, in collaboration with MOHC (c) ISTI, GPCC and GHCN, in collaboration with MOHC and KNMI (d) CHUAN and IGRA upper air databases (agreed with Dick Dee) (e) ICA&D, in collaboration with KNMI (f) RECLAIM, ICOADS marine database, in collaboration with Clive Wilkinson (g) Other Coordinated with C3S311a_Lot2 (Global Databases) and with C3S311a_Lot4 in the case of the ICA&D database Periodicity of updates to be determined, coordinated with WP1 (strong link between WP2 and WP1 feedback process to be developed). Projects first included in Main Portal (WP1) and then inventories entered into Registry. Write a tool that is capable of transforming unformatted metadata into the registry upload format
1.2.4: Provide facilities for submitting digitised data/metadata sets to international climate data archives Fci ncias.ID and MOHC will supply online instructions and tools for users to submit the registered digitized datasets to the international climate data archives (included in the remaining C3S311a Lots), in accordance with the type of data (ECV, land surface, upper air or marine). Links to the several archives data submission guidelines will be provided, indicating the formats accepted by these archives and the relevant steps to be taken until submission is completed. The data assimilation by the global databases will be marked by the registry in the data status Forthcoming Teleconference with Peter Thorne, Stefan, Rob, etc to discuss this topic. These facilities will be available from M9 December 2017 Strong link with WP1, Decide level of feedback between WP1 and WP2 in this task, waiting for Data Model to be developed by Lot 2.
1.2.5: Support activities 1.2.5.1 Technical documentation, User Guide and User Support (M9, then annual) User Guide - Global Registry Manual including the steps taken by users to develop/build the inventories, including writing them according to the guidelines issued in C3S311a_Lot 1.2.1.1, registration (in WP1 Portal) as editing users under/without management supervision, and how to submit metadata. User support continuous, through ticketing system implemented in WP3 (talk to Phil Jones about putting it forward or using another system, we need it before 2020!). 1.2.5.2 Discussion Forum (M12 March 2018) To promote interaction between DARE community in terms of available data, possible future DARE projects (overlap with WP1?) and to exchange ideas about improving the Registry and its service. (Talk to ECMWF about all discussion forums) 1.2.5.3 Production of a How-to-use-the-registry video (M11 February 2018) To publicise the Global Registry and instruct users/potential users on the project s workshops (WP4), and in the DARE community at large. Needs to be ready by M8!! (Collaboration with Peter Siegmund agreed)
WP2 Deliverables Next 12 months
WP2 Management Process and Communication Plans Continuous contact between MOHC and Fci ncias.ID SSEs (weekly or more in the first 9 months), possible visit to MOHC by Fci ncias.ID SSE before 1st prototype delivered. WP2 Lead to send monthly e-mail to all WP2 partners to report on progress (MOHC is included) E-mail also sent to Advisory Board Schedule individual teleconferences at short notice when specific problems have to be solved Use GoTo or Webex instead of Skype! Quarterly teleconferences with all WP2 participants (more when deliverable dates approach...) Monthly during first 3 months (next one in May 10th 11:00-12:30, and another in June 2nd week) Internal review of reports (ideally with some time in advance...)
WP2 Staff Recruitment and IT System Plans Staff Fci ncias.ID Maria Ant nia Valente Lead - in kind contribution Fellow Researcher#1 SSE - to be recruited during April/May 2017 (collaboration with FCUL Informatics Department) working exclusively on WP2 (software, technical guides, user support) Fellow Researcher#2 to be recruited during April/May 2017 -Working exclusively on WP2 (inventories, user guides, user support, video) Fellow Researcher#3 to be recruited during April/May 2017 - Working on WP2 (metadata standards, user guides, user support, video) Staff MOHC SSE personnel applications end in May 7th 2017 Global Registry Main Software to be defined by SSE personnel, ideally by MOHC, fully compatible with C3S to state-of-the-art quality and FCUL IT Systems. Fci ncias.ID SSE personnel will participate in the choice process.
Interconnections ( C3S311a Lot 2 & other Lots) WP2 will interconnect with Lot 2 when: - uploading/updating large land surface metadata bases - Performing QC of large metadatabases and supplying error files to Lot 2 - providing guides/tools for uploading data to Lot 2 databases - providing links for inventories entries to Lot2 datasets - veryfying periodically if inventory links to Lot2 are working (WP1?) - asking if there are new updates of Lot2 databases (WP1?) WP2 will interconnect with other Lots when we have information on what databases they possess and whether their metadata needs to be included in the Global Registry (basically when we know what they do...) -> We want Lot2 s inventory deliverable to be completed in August 2017, will it be made in standard (ECVs and locations) format already?
