Spatial Extreme Verification in Weather Prediction

Spatial Extreme Verification in Weather Prediction
Slide Note
Embed
Share

Scheme for verifying spatial structure of extremes in numerical weather prediction, focusing on precipitation. Overview of NWP verification, challenges, ideal solutions, and methods. Includes motivation, computational methodology, and characteristics of the new scheme.

  • Weather Prediction
  • Spatial Verification
  • Extreme Events
  • NWP Model
  • Precipitation

Uploaded on Feb 21, 2025 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript


  1. A scheme for verifying the spatial structure of extremes in numerical weather prediction: exemplified for precipitation. Bent Hansen Sass, DMI, December 2020 OUTLINE Verifying Numerical Weather prediction (NWP), why and how ? Motivation for verifying local extremes in a NWP model domain Computational methodology of new scheme: A forecast example Characteristics of new scheme relative to FSS and SAL spatial schemes Additional information Contact and References

  2. Verfying NWP : Why ? 3 simple reasons: We need to show how the NWP model compares to the real world ! Does the model improve over time ? How does our model compare with other models ?

  3. Verfying NWP : how ? Many ways of verifying NWP models Tutorial: WWRP/WGNE Joint Working Group on Forecast Verification Research http://www.cawcr.gov.au/projects/verification/ ( see also references )

  4. Verfying NWP : how ? Challenges when making verification schemes for NWP: (1) Observations : Is there a balance between model resolution and the density of observations used for verification ? Traditionally: Verification has a large focus on surface observations in a supported network of observations, with emphasis on selected parameters ( e.g. MSLP, wind, temperature, humidity, clouds, acc. precipitation) . These are to be compared with NWP prediction in some way, e.g. by interpolations to observation points New opportunities: Remote sensing , e.g. based on satellites and radars, transferred to a form that is suitable for NWP verification

  5. Verfying NWP : how ? Present day model grid sizes in Limited Area models are typically 1 3 km which is much lower than the distance between traditional synoptic observations : However in the future high density crowd-source observations may change the picture. 50 km IMPLICATION: FIG.1 : No guarantee that available surface observations pick up the extremes occurring in the real world - that present day models start to be able to simulate.

  6. Verfying NWP : how ? Ideal solution: Work out an analysis on the full model grid , with analysis independent from forecast FIG2 : The new verification schemes operating on entire fields are socalled ` SPATIAL verifications The precipitation verification mentioned in this presentation makes use of the analyzed precipiation at DMI based on DMI radar system and in situ ground observations in Denmark . In this way the analysis becomes independent from the NWP model.

  7. Verfying NWP : how ? An additional challenge occurs when making verification schemes for NWP: (2) Does the verification take into account the scales that the model can predict (resolve) ? , e.g. verifying models on the level of single grid points is problematic and does not provide enough information to the users. Problem seen : the socalled `double penalty issue

  8. `Double penalty needs to be accounted for to give a fair picture of capability of a fine scale model: That is : a neighborhood treatment is needed Forecast F(x) OBS FC1 FC2 FC1 and FC2 have equal error over this part FC2 has larger error over this part, double penalty caused by phase error, but Amlipude of extreme is correct Independent variable X FIG.3 ( Using an ensemble of forecasts may reduce this issue )

  9. Verfying NWP : how ? 3) Will the scheme sometimes provide a high score even though the forecast from other considerations are considered poor ? ` hedging : Example from forecasting: Forecasters may choose to avoid predicting extremes in order to avoid making big failures which is understandable. This issue is called `hedging . (e.g. Murphy 1978, 1993 ) A verification scheme however should try to avoid `hedging in order to give a realistic picture of model quality. A scheme rewarding the prediction of extremes seems desirable.

  10. Verfying NWP : how ? Conclusion based on 1), 2) and 3) above: It is desirable to make a verification scheme which compares an entire forecast field with a corresponding independent analysis field. The verification should take into account the lack of predictability on the scale of individual grid points. This may be done by considering a neighborhood of a certain size around individual points. It is desirable to make a scheme which avoids `hedging , the scheme should reward a realistic prediction of `significant weather.

  11. MOTIVATION for verifying extremes : There is a large focus on extremes today, both in climate prediction and in daily weather forecasts, but traditional verification schemes do not verify extremes ! In daily weather forecasting it would be of great value on a given day, e.g. in the context of many types of outdoor activities, if both the wet extreme and a dry pattern could be forecasted with high accuracy , regarding both absolute amount and spatial accuracy.

  12. MOTIVATION for verifying extremes : Figure 4a-b illustrate a challenging forecasting situation when NWP users experience significant spatial variability of precipitation (red area with observed maximum (ob-max), and blue area with observed minimum (ob-min) which could potentially be with zero precipitation ). The grey `back-ground value represents intermediate precipitation amounts. The observed local extremes occur within close distance geographically. The lighter red and blue areas represent forecast highest and lowest values, respectively . Fig.4a: non-optimal forecast with relatively long distance ( represented by length of arrows) between forecasted and observed precipitation extremes (fc-max, ob-max, fc-min, ob-min ) Fig.4b: Improved overlapping forecast with relatively short distance ( represented by length of arrows) between forecasted and observed precipitation extremes (fc-max, ob-max, fc-min, ob-min )

  13. A spatial verification scheme verifying precipitation extremes A boundary zone of width B is included to allow computations neighborhood size close to the lateral boundaries. using full Alternatively, full neighborhoods are not feasible close to the lateral boundary points Example from Figure: obmax(K1): Observed maximum , - K1identifies extreme point, i.e. multiple number of extremes may be accounted for up to M1 considered if highest values occur with almost identical values (small tolerance) Maximum forecasted value in a neighborhood of dimension L: max(L, K1)=Max{ (i, j ) }, i [iK1 -L, , iK1 +L] , j [jK1 -L, , jK1 +L] iK1, jK1 are coordinates of the related forecasted values in central point.

  14. Transformation of MOTIVATION to a spatial verification scheme `SLX ( Structure of Local EXtremes ) obmax(K1) is maximum of the analysis field max(L,K1) is forecasted maximum in neighborhood of dimension L around obmax (K1) obmin(K2) is minimum of the analysis field min (L,K2) is forecasted minimum in neighborhood of dimension L around obmin (K2) fcmax(K3) is maximum of the forecast field 1 max(L,K3) is analyzed maximum in neighborhood of dimension L around fcmax (K3) Fig.5 : Illustration of identified local extremes with neighborhoods of dimension L. In case of multiple extreme points with ~ the same value 1 K1 M1 , 1 K2 M2 1 K3 M3 , 1 K4 M4 , M1 , M2, M3, M4 are the number of extremes of each type fcmin(K4) is minimum of the forecast field min (L,K4) is analyzed minimum in neighborhood of dimension L around fcmin (K4)

  15. SLX - a novel scheme developed 2019-2020 current application: precipitation fields input needed : decision on neighborhood size - prerequisite : A score function is defined - Structure of Local EXtremes measures - - i) how does the forecast match identified local maxima of the analysis ii) how does the identified local maxima of the forecast agree with the analysis iii) how does forecast match identified local minima of the analysis, and finally iv) how does the identified local minima of the forecast agree with the analysis. i) , ii) , iii) , iv) represent separate comparisons leading to scores defined in interval [0, 1] : SLX (ob_max), SLX (fc_max), SLX(ob_min), SLX(fc_min) Score function: 1 defines perfect match, 0 poor match between forecast and analysis in the neighborhood chosen. Average computation for multiple extreme points Also a weighted mean of the 4 score computations are carried out . This gives in total 5 outputs of a verification.

  16. `SLX verification steps (1): The steps of the verification process for the score component SLXob_maxmay be summarized as follows: I) Choose the current dimension of neighbourhood size L to be used for a given computation II) Determine the value (and positions in grid ) of the observed maxima obmax(K1.) III) Compute the forecasted maxima max(L,K1)in the neighbourhood(s) of the extreme point(s).

  17. `SLX verification steps (2) : IV) Insert the values obmax(K1) as value = OB and max(L,K1) as a forecast to the score function S of Figure 3. The values and distance between the two determine the value between 0 and 1 of the score Sob_max(K1) V) The procedure is repeated when multiple extreme points are diagnosed , 1 K1 M1and the average of all computations is computed as the final value of the score (SLXob_max) Score S 1 0.9 0.8 0.7 S - value 0.6 0.5 0.4 0.3 0.2 0.1 0 OB-k OB Forecast 5*OB Fig.6 Example of score function S which is asymmetric (developed in collaboration with professional NWP users )

  18. `SLX verification scheme : - SLX ( Structure of Local EXtremes ) measures: Using these steps for each type of extreme leads to four individual scores defined in interval [0, 1] : SLX (ob_max), SLX (fc_max), SLX(ob_min), SLX(fc_min) A weighted score SLX is computed as a combined score.

  19. Understanding SLX with multiple extreme points Average computation: All extreme points are considered with equal weight in a neighborhood computation. The Figure illustrates situation for maxima. For minima an important option exists to look for a value closet to a specified value Vmin which may be larger than zero. This tends to prevent the score related with minima to become close to 1 in situations with large dry areas. Fig. 7 Relation between neighborhood and score components : If obmax=fcmaxin grey colored areas and zero values apply outside, then Sob_max=Sfc_maxis zero for L < Ldist, which is the separation distance between obmaxand fcmaxprecipitation areas. Lob(max) is a neighborhood size above which Sob_max becomes 1. Lfc(max) is a neighborhood size above which Sfc_maxbecomes 1.

  20. Fig.8 For large domains multiple sub-areas may be included that can be treated with separate or combined statistics

  21. SLX for Ensembles The SLX concept involving neighborhoods may be used for ensembles , e.g. examining the performance of ensemble median and percentiles.

  22. Forecast Example : Convection over parts of Denmark F Fig. 9 Forecasted accumulated precipitation (kg/m2) valid from 9 UTC -12 UTC 27 August 2019 over the light squared areas of dimensions 325 km*325 km. Figure A, B, C and D apply to forecasts starting at 00 UTC , 03 UTC , 06 UTC and 09 UTC respectively. Figure E shows the corresponding analyzed field of precipitation 9-12 UTC 27 August 2019. The black crosses indicate the maxima. Fig. 9 F shows the resulting combined scores of SLX , from four forecasts starting at different origin times.

  23. Basic comparisons with Fractions Skill Score (FSS) and SAL (Structure , Amplitude and Location) CHARACTERISTICS FSS SAL SLX Main characteristics of scheme Predict fractions correct , - Identify which scales can be resolved Identify large- scale features, e.g. bias and variability of fields Identify match of forecast and analysis around extreme values Number of score components 1 3 5 Type of spatial scheme N F N + F Dimensions (0-D, 1-D, 2-D) of input parameters in normal tests. 2-D 0-D 1-D N- size and define Threshold/ percentile Uniquely defined once objects are fixed N-size + define Score- function between 0 and 1 N=Neighborhood , F= Features based

  24. Additional information An early version of the idea of verifying local extremes has been operational in DMI for several years (Sass and Yang 2012 ) . Some other relevant references are provided in the last slide. A publication on SLX is under review in an international journal. The scheme has been tested in many idealized cases and in a simulation of pre- operational conditions. The scheme has been prepared for operational use. An operational precipitation analysis in DMI makes this feasible from 2021. Acknowledgements to Kasper Hintz who contributed to the realization of the 3D test case Xiaohua Yang for contributing to a first version of the idea for verifying extremes spatially. Henrik Feddersen for collaboration on operational readiness of the scheme in DMI The DMI radar group, in particular Rashpal Gill and Thomas B vith, who developed the present precipitation analysis product used to analyze precipitation over Denmark.

  25. Contact information and References Contact: Bent Hansen Sass, Danish Meteorological Institute , Lyngbyvej 100, 2100 , Copenhagen , E-mail: bhs@dmi.dk , Tel. +45 50 93 38 23 References : Gilleland, E., Ahijevych, D.A., Brown, B.G. and Ebert, E.E. 2010: Verifying Forecasts Spatially. Bull. Amer. Meteor. Soc., October, 1365 1373. Gilleland,E., Skok, G., Brown,G.B., Casati,B., Dorninger, M., Mittermaier, M.P., Roberts, N. and Wilson, L.J. 2020: A Novel Set of Geometric Verification Test Fields with Application to Distance Measures. Mon. Wea. Rev. 148, 1653 - 1673 Murphy, A.H., 1978: Hedging and the mode of expression of weather forecasts. Bull. Amer.Meteor. Soc.,59: 371-373. Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293. Roberts, N.M. and Lean, H.W. 2008: Scale-selective verification of rainfall accumulations from high- resolution forecasts of convective events. Mon. Wea. Rev., 136, 78-97.

  26. References Sass, B.H. and Yang, X. 2012: A verification score for high resolution NWP: Idealized and preoperational tests. HIRLAM Tech,Rep. No 69, 28 pp , Dec. 2012 , [Available from www.hirlam.org] Verification Methods Workshop, International Verification Methods Workshop Online (2020-IVMW-O), Nov.2020, https://jwgfvr.univie.ac.at Wernli, H., Paulat, M. 2008: SAL A Novel Quality Measure for the Verification of Quantitative Precipitation forecasts. Mon. Wea. Rev., 136, 4470-4486 WWRP/WGNE Joint Working Group on Forecast Verification Research http://www.cawcr.gov.au/projects/verification/ Theme: Forecast Verification methods Across Time and Space Scales , 7thInternational

More Related Content