Revolutionizing Data Delivery with Python ADDE Project

 
Python ADDE Project
 
David Santek, Becky Schaffer, Jon Beavers,
Tommy Jasmin, Jerry Robaidek, Dave Hoese
 
2019 McIDAS Users’ Group Meeting
17 September 2019
 
Overview
 
A cornerstone of McIDAS is the modular and
A cornerstone of McIDAS is the modular and
extensible Abstract Data Distribution Environment
extensible Abstract Data Distribution Environment
(ADDE):
(ADDE):
Servers for both local and remote data access
Servers for both local and remote data access
All satellite data in SSEC’s Satellite Data Services are
All satellite data in SSEC’s Satellite Data Services are
accessible via ADDE
accessible via ADDE
Used throughout the McIDAS community by both
Used throughout the McIDAS community by both
McIDAS-X and McIDAS-V
McIDAS-X and McIDAS-V
We expect the ADDE data delivery protocol to be
We expect the ADDE data delivery protocol to be
important for many years to come.
important for many years to come.
 
2
 
Overview
 
ADDE servers primarily written at SSEC, with
Unidata, EUMETSAT, and AEMET also
contributing.
More than 20 servers, including:
Satellite: GOES ABI, Himawari AHI, MODIS,
AVHRR, VIIRS, Meteosat, COMS
Grid: GRIB, McIDAS Grid
Other: netCDF, HDF, GeoTIFF, Text
 
3
 
Objectives
 
ADDE infrastructure refactoring:
ADDE infrastructure refactoring:
Move ADDE server development into a 
Move ADDE server development into a 
modern programming
modern programming
environment.
environment.
Ensure a future for ADDE for new satellites and other image data:
Ensure a future for ADDE for new satellites and other image data:
Motivated by the fact that 
Motivated by the fact that 
current SSEC ADDE developers are semi-retired
current SSEC ADDE developers are semi-retired
ADDE continues to be an important data delivery mechanism 
ADDE continues to be an important data delivery mechanism 
for SSEC,
for SSEC,
Unidata, and many of the current McIDAS-X and McIDAS-V users.
Unidata, and many of the current McIDAS-X and McIDAS-V users.
Outcome will be Python library and functional ADDE server that will:
Outcome will be Python library and functional ADDE server that will:
Interface and be packaged with McIDAS-X and –V
Interface and be packaged with McIDAS-X and –V
Work on Linux, Windows, and Mac
Work on Linux, Windows, and Mac
Enable a programmer or scientist with 1 year of Python experience to
Enable a programmer or scientist with 1 year of Python experience to
create their own ADDE server without knowledge of underlying McIDAS
create their own ADDE server without knowledge of underlying McIDAS
Fortran and C code.
Fortran and C code.
 
4
 
pyADDE Status
pyADDE Status
Phase 1: Study - 
Phase 1: Study - 
Complete
Complete
 
Investigate the current ADDE protocol and implementation:
Investigate the current ADDE protocol and implementation:
Evaluate the feasibility, and
Evaluate the feasibility, and
Estimate the effort to enable the use of Python to write ADDE servers.
Estimate the effort to enable the use of Python to write ADDE servers.
 
Yes, it is programmatically possible to develop servers
Yes, it is programmatically possible to develop servers
independent of McIDAS-X
independent of McIDAS-X
 
5
 
Python ADDE Servers
 
Write a new server without McIDAS-X knowledge
Only 
Input Module 
will need to be written:
Read native file
Compute lat/lon for each 
n
th
 point
Reshape into Python Dataset Object
pyADDE does the rest
 
6
 
pyADDE Status
pyADDE Status
Phase 2: Prototype - 
Phase 2: Prototype - 
Complete
Complete
 
A Python directory server was written
A Python directory server was written
A Python data server was written
A Python data server was written
A basic server to read a PNG file and return the data along with
A basic server to read a PNG file and return the data along with
McIDAS-necessary metadata (directory and navigation block)
McIDAS-necessary metadata (directory and navigation block)
The 
The 
ADDE protocol tests were successful 
ADDE protocol tests were successful 
for both local and remote
for both local and remote
instances of the Python directory and data servers on Linux
instances of the Python directory and data servers on Linux
Note: This uncovered an unexpected issue in the transmission protocol
Note: This uncovered an unexpected issue in the transmission protocol
not seen with typical McIDAS-X ADDE servers
not seen with typical McIDAS-X ADDE servers
. Python seemingly
. Python seemingly
does not block by default on stdout going to a socket (remote serving),
does not block by default on stdout going to a socket (remote serving),
whereas going to a pipe works fine (local serving). Issue resolved by
whereas going to a pipe works fine (local serving). Issue resolved by
calling os.set_blocking(1, True).
calling os.set_blocking(1, True).
 
7
 
pyADDE Status
pyADDE Status
Phase 3: Refactoring – 
Phase 3: Refactoring – 
Beginning Oct. 1
Beginning Oct. 1
 
Rewrite low-level C and Fortran routines into Python:
Python ADDE servers and library (pyADDE) is a stand-alone package
ADDE servers can be written without in-depth knowledge of McIDAS-X
Investigate interfacing Python ADDE servers to the SatPy library:
Access to a variety of geostationary and polar satellite data, through a
common API
New Python ADDE servers could be written without a deep
understanding of data formats used by the different satellites
Write a Python version of the McIDAS-X ABI server
Document and test with Windows and Linux
 
8
 
Summary
 
Advantages:
Advantages:
Evolution
Evolution
 rather than revolution
 rather than revolution
May make use of McIDAS-X Fortran and C library functions in the
May make use of McIDAS-X Fortran and C library functions in the
short-term
short-term
Extensions
Extensions
 more easily implemented
 more easily implemented
Return other formats (e.g., netCDF), not only McIDAS-X Area
Return other formats (e.g., netCDF), not only McIDAS-X Area
Perhaps additional geographic projections (using 
Perhaps additional geographic projections (using 
proj.4
proj.4
)
)
Status:
Status:
Prototype server 
Prototype server 
being developed
being developed
Plan is to have first version of 
Plan is to have first version of 
pyADDE
pyADDE
 available by 
 available by 
mid-2020
mid-2020
 
9
Slide Note
Embed
Share

Python ADDE Project aims to modernize data delivery infrastructure by developing Python libraries for ADDE server development, enabling easy creation of servers without the need for McIDAS-X knowledge. The initiative seeks to ensure a sustainable future for ADDE in serving new satellite and image data, with a focus on user-friendly, cross-platform functionality. The project has completed initial phases including protocol investigation, feasibility assessment, and prototype development of Python-based ADDE servers.


Uploaded on Aug 03, 2024 | 4 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Python ADDE Project David Santek, Becky Schaffer, Jon Beavers, Tommy Jasmin, Jerry Robaidek, Dave Hoese 2019 McIDAS Users Group Meeting 17 September 2019

  2. Overview A cornerstone of McIDAS is the modular and extensible Abstract Data Distribution Environment (ADDE): Servers for both local and remote data access All satellite data in SSEC s Satellite Data Services are accessible via ADDE Used throughout the McIDAS community by both McIDAS-X and McIDAS-V We expect the ADDE data delivery protocol to be important for many years to come. 2

  3. Overview ADDE servers primarily written at SSEC, with Unidata, EUMETSAT, and AEMET also contributing. More than 20 servers, including: Satellite: GOES ABI, Himawari AHI, MODIS, AVHRR, VIIRS, Meteosat, COMS Grid: GRIB, McIDAS Grid Other: netCDF, HDF, GeoTIFF, Text 3

  4. Objectives ADDE infrastructure refactoring: Move ADDE server development into a modern programming environment. Ensure a future for ADDE for new satellites and other image data: Motivated by the fact that current SSEC ADDE developers are semi-retired ADDE continues to be an important data delivery mechanism for SSEC, Unidata, and many of the current McIDAS-X and McIDAS-V users. Outcome will be Python library and functional ADDE server that will: Interface and be packaged with McIDAS-X and V Work on Linux, Windows, and Mac Enable a programmer or scientist with 1 year of Python experience to create their own ADDE server without knowledge of underlying McIDAS Fortran and C code. 4

  5. pyADDE Status Phase 1: Study - Complete Investigate the current ADDE protocol and implementation: Evaluate the feasibility, and Estimate the effort to enable the use of Python to write ADDE servers. Yes, it is programmatically possible to develop servers independent of McIDAS-X 5

  6. Python ADDE Servers Write a new server without McIDAS-X knowledge Only Input Module will need to be written: Read native file Compute lat/lon for each nth point Reshape into Python Dataset Object pyADDE does the rest 6

  7. pyADDE Status Phase 2: Prototype - Complete A Python directory server was written A Python data server was written A basic server to read a PNG file and return the data along with McIDAS-necessary metadata (directory and navigation block) The ADDE protocol tests were successful for both local and remote instances of the Python directory and data servers on Linux Note: This uncovered an unexpected issue in the transmission protocol not seen with typical McIDAS-X ADDE servers. Python seemingly does not block by default on stdout going to a socket (remote serving), whereas going to a pipe works fine (local serving). Issue resolved by calling os.set_blocking(1, True). 7

  8. pyADDE Status Phase 3: Refactoring Beginning Oct. 1 Rewrite low-level C and Fortran routines into Python: Python ADDE servers and library (pyADDE) is a stand-alone package ADDE servers can be written without in-depth knowledge of McIDAS-X Investigate interfacing Python ADDE servers to the SatPy library: Access to a variety of geostationary and polar satellite data, through a common API New Python ADDE servers could be written without a deep understanding of data formats used by the different satellites Write a Python version of the McIDAS-X ABI server Document and test with Windows and Linux 8

  9. Summary Advantages: Evolution rather than revolution May make use of McIDAS-X Fortran and C library functions in the short-term Extensions more easily implemented Return other formats (e.g., netCDF), not only McIDAS-X Area Perhaps additional geographic projections (using proj.4) Status: Prototype server being developed Plan is to have first version of pyADDE available by mid-2020 9

More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#