Update on LPV Stages and Metrics for Cal/Val
This update covers LPV stages and metrics discussed at a subgroup meeting, including an overview, paper status, validation stage table history, maturity matrix, and existing approaches. Topics include the LPV overview paper status, validation stage table evolution, and other similar approaches like the Maturity Matrix. Insights on the value and purpose of the LPV validation stage table are shared, along with discussions on actions and work distribution related to LPV.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Update on LPV Stages and Metrics for Cal/Val Gabriela Schaepman-Strub (U Zurich) LPV Subgroup meeting (ESA LPS side meeting) 11 May 2016
Overview 1. 2. 3. LPV overview paper status - GS LPV validation stage table status - GS Other similar approaches linked to CEOS (WG Climate) GS Maturity matrix (Bates and Privette, 2012) Core-Climax Climate data record assessment (EUMETSAT) Inconsistencies of current LPV table, with Maturity Matrices, and thoughts on WGCV usage Discussion - All Decision on actions and work distribution - All 4. 5. 6. 2
1. LPV Overview Paper Status Submission of abstract to RSE special issue Editors: Michael Buchwitz, Emilio Chuvieco, Samantha Lavendern (deadline 31 May) http://www.journals.elsevier.com/remote-sensing-of-environment/call- for-papers/special-issue-on-earth-observation-of-essential-climate-vari/ Part of paper is update of validation stage table and assignment of validation stage to each variable covered by LPV 3
2. LPV Validation Stage Table (History) Morisette 2006, 3 validation stages. Baret, EOS report, 2006, introduced 4thstage Nightingale, 2010, with updated validation stage table LPV website (current version) slight changes to Nightingale, 2010 ( rigorous instead of robust , publication already at stage 2) Nightingale, J. Schaepman-Strub, G. Nickeson, J. and Baret, F. 2010. Assessing Satellite-derived Land Product Quality for Earth System Science Applications: Results from the CEOS LPV Sub-group. Proceedings of Earth Observation for Land- Atmosphere Interaction Science, SP-688, European Space Agency, 2011. 4
Short Discussion What is the value/purpose of the LPV validation stage table for YOU? 6
3. Other Existing Approaches (Maturity Matrix) Bates, J. J., and J. L. Privette (2012), A maturity model for assessing the completeness of climate data records, Eos Trans. AGU, 93(44), 441, doi:10.1029/2012EO440006.
3. Other Existing Approaches (CORE-CLIMAX) EUMETSAT, Core-Climax CDR Assessment Report, CC/EUM/REP/14/004, v1A Draft, 23 December 2014 Based on Bates and Privette, 2012 Extending the model to be applicable to climate data records in general (satellite, in situ, combined (sat + in situ), reanalysis) Extending the validation column to include uncertainty assessment other than direct validation (e.g. downstream from instrument, propagated through algorithm) 9
CORE-CLIMAX 10
Maturi ty 1 UNCERTAINTY CHARACTERISATION None Standard uncertainty nomenclature is identified or defined; limited validation done; limited information on uncertainty available 2 Score 2 + standard nomenclature applied; validation extended to full product data coverage, comprehensive information on uncertainty available; methods for automated monitoring defined Score 3 + procedures to establish SI traceability are defined; (inter)comparison against corresponding CDRs (other methods, models, etc); quantitative estimates of uncertainty provided within the product characterising more or less uncertain data points; automated monitoring partially implemented Score 4 + SI traceability partly established; data provider participated in one inter-national data assessment; comprehensive validation of the quantitative uncertainty estimates; automated quality monitoring fully implemented (all production levels) Score 5 + SI traceability established; data provider participated in multiple inter- national data assessment and incorporating feedbacks into the product development cycle; temporal and spatial error covariance quantified; Automated monitoring in place with results fed back to other accessible information, e.g. meta data or documentation 3 4 5 6 11
Sub-Criteria Summary CLIMATE DATA RECORD (CDR) MATURITY EVALUATION GUIDELINES UNCERTAINTY CHARACTERIS ATION None Standard uncertainty nomenclature is idenitified or defined; limited validation done; limited information on uncertainty available Score 2 + standard nomenclature applied; validation extended to full product data coverage, comprehensive information on uncertainty available; methods for automated monitoring defined Automated Quality Monitoring None Matu rity Uncertainty quantification Standards1 Validation 1 None None None Validation using external reference data done for limited locations and times Limited information on uncertainty arising from systematic and random effects in the measurement Standard uncertainty nomenclature is identified or defined 2 None Validation using external reference data done for global and temporal representative locations and times Comprehensive information on uncertainty arising from systematic and random effects in the measurement Score 2 + Standard uncertainty nomenclature is applied Methods for automated quality monitoring defined 3 12
4. Inconsistencies of current LPV table, 1. Missing indication that stages build up on each other (stage 2 = stage 1 + additional critera, stage 3 = stage 2 + additional criteria, etc.) -> include Number of sites increasing with stage, but not clearly defined: stage 1 < 30, stage 2 significant set of locations over globally representative locations , stage 3 multiple locations . representing global conditions -> consolidate and refer to variable-specific protocols? Distribution of sites not clearly defined: stage 1 only number, stage 2 globally representative , etc. -> what does this mean? Stratified per ecosystem, variable histogram, or spatially (e.g. latitudinal zone) etc.? -> definition will differ with variable! -> consolidate and refer to variable-specific protocols? Same with requirements on temporal component -> consolidate and refer to variable-specific protocols? Statistically rigorous versus robust (include criteria on required metrices, statistical approaches?) -> define in more detail, consistent with GUM/VIM? 2. 3. 4. 5. 13
6. Quality criteria of reference data missing (e.g. fiducial reference data with included uncertainties, unertainties of intercomparison data set) -> include additional stage (e.g. 5 where quality of reference data is emphasized or at much earlier stage?) Keep LPV application in mind (we do not want to be limited to Climate Data Records, criteria need to be open enough to be useful for different user communities (e.g. time range needed, uncertainty metrices, etc.)). 14
4. inconsistencies with and between different approaches 6 stages, starting with none (Bates and CoreClimax); 4 stages for LPV LPV stage 4 new product version and time series expansion; covered by CoreClimax as new product (with full assessment) and by automated validation Bates and LPV validation only; CoreClimax including broader uncertainty assessment (uncertainty characterization based on algorithm) Bates and LPV are missing uncertainty characterization of reference data (fiducial reference data); included in CoreClimax as SI traceability CoreClimax uncertainty characterization column as summary of Sub-Criteria columns 15
4. and thoughts on WGCV usage. Miguel what is the request from WGCV? [Response from WGCV Plenary]: Sorting out how we proceed with the terminology / Who does what and when ? What is our understanding for a validation metrics from the different subgroups (and finally WGCV chair) perspective ? How we collect the eventually different views and how do we proceed to achieve from the treasure of existing approaches in the different communities a way forward which leads to a common definition/path forward (which may also be extended by each individual community if needed in certain limits) ? 16
5. Discussion Other existing or developing tables of relevance that were missed in this presentation (e.g. QA4ECV)? Should LPV adopt the Maturity Matrix (Bates or CoreClimax) approach? CEOS definition of validation CEOS defines validation as the process of assessing the quality of the data products derived from system outputs by independent means (Morisette, TGARS 06, citing Justice, IJRS, 2000) -> Bates in line with this definition, CoreClimax goes beyond validation (uncertainty characterization) -> adopting the CoreClimax definition might go beyond the LPV mission? GS sees 2 potential solutions 1. soft update of current LPV table (green suggestions on previous slide), include it in paper, work on WGCV and maturity matrix consolidation in a second step 2. directly consolidate with maturity matrix (J rg Schulz welcomed the discussion on this topic!) 17
6. Actions and Work Distribution 1. Next steps Paper abstract submission (31 May) Consolidation within LPV -> who is interested to contribute? Application to all variables Paper finalization Outreach to WGCV for general application (WGCV chair, IVOS) -> Miguel? Outreach to WG Climate (Bates, Schulz) -> who is interested to contribute? Actions and work distribution 2. 18
