JEDI Data Assimilation: Integration and Algorithms
D
a
t
a
A
s
s
i
m
i
l
a
t
i
o
n
A
l
g
o
r
i
t
h
m
s
Joint Center for Satellite Data Assimilation (JCSDA)
JEDI Academy - 21-25 June 2021
The Joint Effort for Data
assimilation Integration (JEDI)
JEDI
Data Assimilation Reminders
NWP is a cycling process, repeated for every new forecast, typically:
-
Every 6 hours for global forecasts
-
More frequently for local forecasts
T
h
e
p
e
r
i
o
d
o
v
e
r
w
h
i
c
h
o
b
s
e
r
v
a
t
i
o
n
s
a
r
e
u
s
e
d
i
s
t
h
e
a
s
s
i
m
i
l
a
t
i
o
n
w
i
n
d
o
w
,
t
h
e
r
e
p
e
t
i
t
i
v
e
p
r
o
c
e
s
s
i
s
t
h
e
d
a
t
a
a
s
s
i
m
i
l
a
t
i
o
n
c
y
c
l
i
n
g
.
The cycling algorithm is coded in scripts and workflows (100s or 1000s of tasks per cycle).
The data assimilation itself is happening in one (a few) executable(s).
Cycling for JEDI is in development, the initial release (and this academy) only cover the data
assimilation executables.
Cycling Data Assimilation
time
Previous
forecast
Observations
Analysis
Assimilation window
4D Assimilation Window
time
Previous
forecast
Assimilation window
Observations
Analysis
Not used any more in operational systems
3D Assimilation Window
time
Previous
forecast
Assimilation window
Observations
Analysis
3D-FGAT
(First Guess at Appropriate Time)
OOPS Analysis and Design
JEDI
Data assimilation aims at estimating the state of a system given all
available information:
-
Previous forecast (background)
-
Observations
-
Possibly other constraints (theoretical information)
Example: variational data assimilation
JEDI
Data Assimilation in JEDI
Cost Function Design
JEDI
•
Naive approach:
-
One object for each term of the cost function.
-
Compute each term (or gradient) and add them together.
-
Problem: The model is run several times (Jo, Jc, Jq)
•
Another naive approach:
-
Run the model once and store the full 4D state.
-
Compute each term (or gradient) and add them together.
-
Problem: The full 4D state is too big (for operational use).
•
A feasible approach:
-
Run the model once.
-
Compute each term (gradient) as the model is running.
-
Add all the terms together.
Cost Function Implementation
JEDI
•
One class for each term (more flexible).
•
Call a method on each object on the fly while the model is running:
-
Uses the PostProcessor structure.
-
Finalize each term and add the terms together at the end.
-
Saving the model linearization trajectory is also handled by a PostProcessor.
•
Each formulation derives from an abstract CostFunction base class:
-
Code duplication: it was decided to keep 3D-Var and 4D-Var for readability.
•
The terms can be re-used (or not), 4D-En-Var was added in a few hours:
-
OO is not magic and will not solve scientific questions by itself.
-
Scientific questions (localization) remain but scientific work can start.
-
Months of work would have been necessary in traditional systems.
Post Processors
JEDI
PostProcessors are called regularly during model integration
-
The presence or not of post processing does not affect the forecast
-
PostProcessors isolate the model code from many other unrelated
aspects (separation of concerns)
Examples:
-
Output of forecast fields
-
Generation of products for users
-
Computation of filtered state (for DFI)
-
Generation of trajectory for linearized models
-
Generation of simulated observations
-
Diagnostics
-
…
4D-Var yaml
JEDI
cost function
:
cost type
:
4D-Var
window begin
:
'{{window_begin}}’window length
:
PT6H
geometry
:
$(GEOMETRY)
model
:
$(MODEL)
analysis variables
:
$(an_variables)
background
:
$(BACKGROUND)
background error
:
$(BACKGROUND_ERROR)
observations
:
$(OBSERVATIONS)
variational
:
minimizer
:
algorithm
:
DRPCG
iterations
:
-
ninner
:
30
gradient norm reduction
:
0.01
geometry
:
$(GEOMETRY)
linear model
:
$(LINEAR_MODEL)
output
:
$(AN_OUTPUT)
cost function
:
cost type
:
3D-Var
window begin
:
'{{window_begin}}’window length
:
PT6H
geometry
:
$(GEOMETRY)
analysis variables
:
$(an_variables)
background
:
$(BACKGROUND)
background error
:
$(BACKGROUND_ERROR)
observations
:
$(OBSERVATIONS)
variational
:
minimizer
:
algorithm
:
DRPCG
iterations
:
-
ninner
:
30
gradient norm reduction
:
0.01
geometry
:
$(GEOMETRY)
output
:
$(AN_OUTPUT)
3D-Var
3D-FGAT yaml
JEDI
cost function
:
cost type
:
4D-Var
window begin
:
'{{window_begin}}’window length
:
PT6H
geometry
:
$(GEOMETRY)
model
:
$(MODEL)
analysis variables
:
$(an_variables)
background
:
$(BACKGROUND)
background error
:
$(BACKGROUND_ERROR)
observations
:
$(OBSERVATIONS)
variational
:
minimizer
:
algorithm
:
DRPCG
iterations
:
-
ninner
:
30
gradient norm reduction
:
0.01
geometry
:
$(GEOMETRY)
linear model
:
name: identity
variable change: Identity
tstep: PT6H
cost function
:
cost type
:
4D-Var
window begin
:
'{{window_begin}}’window length
:
PT6H
geometry
:
$(GEOMETRY)
model
:
$(MODEL)
analysis variables
:
$(an_variables)
background
:
$(BACKGROUND)
background error
:
$(BACKGROUND_ERROR)
observations
:
$(OBSERVATIONS)
variational
:
minimizer
:
algorithm
:
DRPCG
iterations
:
-
ninner
:
30
gradient norm reduction
:
0.01
geometry
:
$(GEOMETRY)
linear model
:
$(LINEAR_MODEL)
output
:
$(AN_OUTPUT)
Editing yaml
JEDI
Some editor let you fold/unfold yaml sections (here atom)
JEDI Applications
JEDI
JEDI: Applications
JEDI
OOPS-JEDI Applications can be found in
src/oops/runs
, for example:
-
Variational.h
-
Forecast.h
-
HofX.h
Applications are the normal mode of execution
Unit tests are specific applications that run tests for a given class
The main program manages tests and adds test reporting
This is hidden for most users and developers
JEDI: Variational Application
JEDI
Several cost functions are
available:
-
4D-Var
-
3D-Var
-
4D-En-Var
-
Weak constraint 4D-Var
The last two are parallel in time.
Minimizers are available:
-
In primal (model) space
-
In dual (observation) space
-
I
n
d
o
u
b
l
e
-
p
r
i
m
a
l
s
p
a
c
e
(
G
S
I
,
D
e
r
b
e
r
-
R
o
s
a
t
i
,
n
o
B
-
1
)
-
As saddle point (weak constraint)
In each space, several algorithms:
-
PCG
-
Lanczos
-
GMRES
-
…
JEDI: EnsembleApplication
JEDI
src/oops/runs
also contains
EnsembleApplication.h
for… ensemble applications!
The EnsembleApplication class is templated on an Application and runs an ensemble
of that application. It provides a generic way of running an ensemble of Applications in
the same executable.
It’s yaml file is very simple, for an ensemble of variational DA (EDA):
EDA is coded with 77 lines of C++ (including comments) and some yaml.
It also work for ensemble forecasts, ensembles of H(x)… The only constraint is that all
members run the same application.
files
:
-
"testinput/eda.3dvar.1.yaml”
-
"testinput/eda.3dvar.2.yaml”
-
"testinput/eda.3dvar.3.yaml”
-
"testinput/eda.3dvar.4.yaml"
JEDI: Applications
JEDI
Many more Applications can be found in
src/oops/runs
:
-
AddIncrement.h
-
ConvertState.h
-
DiffStates.h
-
Dirac.h
-
EnsRecenter.h
-
EnsVariance.h
-
ExternalDFI.h
Many are still in development and not fully documented or
supported, feel free to explore.
-
Forecast.h
-
GenEnsPertB.h
-
HofX.h
-
HofXNoModel.h
-
LocalEnsembleDA.h
-
RTPP.h
-
StaticBInit.h
-
Variational.h
Working with JEDI Applications
JEDI
All JEDI Applications are generic:
-
Given the appropriate yaml file and input data they work with any model
Debug and validate with simple toy models, apply to operational models
-
No recoding
-
Even in an operational center it will save time
The Lorenz 95 and QG models are provided with OOPS
(also a good place to understand interfaces)
Running the code in an interactive debugger can be a good way to learn
The Joint Effort for Data Assimilation Integration (JEDI) explores data assimilation algorithms through the Joint Center for Satellite Data Assimilation (JCSDA). Learn about cycling data assimilation, assimilation windows, cost function design, and more in this comprehensive overview.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
The Joint Effort for Data assimilation Integration (JEDI) Data Assimilation Algorithms Joint Center for Satellite Data Assimilation (JCSDA) JEDI Academy - 21-25 June 2021
JEDI Data Assimilation Reminders
Cycling Data Assimilation NWP is a cycling process, repeated for every new forecast, typically: Every 6 hours for global forecasts - More frequently for local forecasts - The period over which observations are used is the assimilation window, the repetitive process is the data assimilation cycling. The cycling algorithm is coded in scripts and workflows (100s or 1000s of tasks per cycle). The data assimilation itself is happening in one (a few) executable(s). Cycling for JEDI is in development, the initial release (and this academy) only cover the data assimilation executables.
4D Assimilation Window Observations Analysis Previous forecast Assimilation window time
3D Assimilation Window Observations Analysis Previous forecast Assimilation window time Not used any more in operational systems
3D-FGAT (First Guess at Appropriate Time) Observations Analysis Previous forecast Assimilation window time
OOPS Analysis and Design Data assimilation aims at estimating the state of a system given all available information: - Previous forecast (background) - Observations - Possibly other constraints (theoretical information) Example: variational data assimilation JEDI
JEDI Data Assimilation in JEDI
Cost Function Design Naive approach: - One object for each term of the cost function. - Compute each term (or gradient) and add them together. - Problem: The model is run several times (Jo, Jc, Jq) Another naive approach: - Run the model once and store the full 4D state. - Compute each term (or gradient) and add them together. - Problem: The full 4D state is too big (for operational use). A feasible approach: - Run the model once. - Compute each term (gradient) as the model is running. - Add all the terms together. JEDI
Cost Function Implementation One class for each term (more flexible). Call a method on each object on the fly while the model is running: - Uses the PostProcessor structure. - Finalize each term and add the terms together at the end. - Saving the model linearization trajectory is also handled by a PostProcessor. Each formulation derives from an abstract CostFunction base class: - Code duplication: it was decided to keep 3D-Var and 4D-Var for readability. The terms can be re-used (or not), 4D-En-Var was added in a few hours: - OO is not magic and will not solve scientific questions by itself. - Scientific questions (localization) remain but scientific work can start. - Months of work would have been necessary in traditional systems. JEDI
Post Processors PostProcessors are called regularly during model integration - The presence or not of post processing does not affect the forecast - PostProcessors isolate the model code from many other unrelated aspects (separation of concerns) Examples: - Output of forecast fields - Generation of products for users - Computation of filtered state (for DFI) - Generation of trajectory for linearized models - Generation of simulated observations - Diagnostics - JEDI
4D-Var yaml cost function: cost type: 4D-Var window begin: '{{window_begin}} window length: PT6H geometry: $(GEOMETRY) model: $(MODEL) analysis variables: $(an_variables) background: $(BACKGROUND) background error: $(BACKGROUND_ERROR) observations: $(OBSERVATIONS) variational: minimizer: algorithm: DRPCG iterations: - ninner: 30 gradient norm reduction: 0.01 geometry: $(GEOMETRY) linear model: $(LINEAR_MODEL) output: $(AN_OUTPUT) cost function: cost type: 3D-Var window begin: '{{window_begin}} window length: PT6H geometry: $(GEOMETRY) analysis variables: $(an_variables) background: $(BACKGROUND) background error: $(BACKGROUND_ERROR) observations: $(OBSERVATIONS) variational: minimizer: algorithm: DRPCG iterations: - ninner: 30 gradient norm reduction: 0.01 geometry: $(GEOMETRY) output: $(AN_OUTPUT) 3D-Var JEDI
3D-FGAT yaml cost function: cost type: 4D-Var window begin: '{{window_begin}} window length: PT6H geometry: $(GEOMETRY) model: $(MODEL) analysis variables: $(an_variables) background: $(BACKGROUND) background error: $(BACKGROUND_ERROR) observations: $(OBSERVATIONS) variational: minimizer: algorithm: DRPCG iterations: - ninner: 30 gradient norm reduction: 0.01 geometry: $(GEOMETRY) linear model: $(LINEAR_MODEL) output: $(AN_OUTPUT) cost function: cost type: 4D-Var window begin: '{{window_begin}} window length: PT6H geometry: $(GEOMETRY) model: $(MODEL) analysis variables: $(an_variables) background: $(BACKGROUND) background error: $(BACKGROUND_ERROR) observations: $(OBSERVATIONS) variational: minimizer: algorithm: DRPCG iterations: - ninner: 30 gradient norm reduction: 0.01 geometry: $(GEOMETRY) linear model: name: identity variable change: Identity tstep: PT6H JEDI
Editing yaml Some editor let you fold/unfold yaml sections (here atom) JEDI
JEDI Applications JEDI
JEDI: Applications OOPS-JEDI Applications can be found in src/oops/runs, for example: - Variational.h - Forecast.h - HofX.h Applications are the normal mode of execution Unit tests are specific applications that run tests for a given class The main program manages tests and adds test reporting This is hidden for most users and developers JEDI
JEDI: Variational Application Several cost functions are available: - 4D-Var - 3D-Var - 4D-En-Var - Weak constraint 4D-Var Minimizers are available: - In primal (model) space - In dual (observation) space - In double-primal space (GSI, Derber-Rosati, no B-1) - As saddle point (weak constraint) The last two are parallel in time. In each space, several algorithms: - PCG - Lanczos - GMRES - JEDI
JEDI: EnsembleApplication src/oops/runs also contains EnsembleApplication.h for ensemble applications! The EnsembleApplication class is templated on an Application and runs an ensemble of that application. It provides a generic way of running an ensemble of Applications in the same executable. It s yaml file is very simple, for an ensemble of variational DA (EDA): files: - "testinput/eda.3dvar.1.yaml - "testinput/eda.3dvar.2.yaml - "testinput/eda.3dvar.3.yaml - "testinput/eda.3dvar.4.yaml" EDA is coded with 77 lines of C++ (including comments) and some yaml. It also work for ensemble forecasts, ensembles of H(x) The only constraint is that all members run the same application. JEDI
JEDI: Applications Many more Applications can be found in src/oops/runs: - Forecast.h - GenEnsPertB.h - HofX.h - HofXNoModel.h - LocalEnsembleDA.h - RTPP.h - StaticBInit.h - Variational.h - AddIncrement.h - ConvertState.h - DiffStates.h - Dirac.h - EnsRecenter.h - EnsVariance.h - ExternalDFI.h Many are still in development and not fully documented or supported, feel free to explore. JEDI
Working with JEDI Applications All JEDI Applications are generic: - Given the appropriate yaml file and input data they work with any model Debug and validate with simple toy models, apply to operational models - No recoding - Even in an operational center it will save time The Lorenz 95 and QG models are provided with OOPS (also a good place to understand interfaces) Running the code in an interactive debugger can be a good way to learn JEDI
