SMOS Pilot Mission Exploitation Platform (Pi-MEP) Overview

SMOS
Pilot-Mission Exploitation Platform
(Pi-MEP)
Presentation of the Pi-MEP tools
Sébastien Guimbard
1
, Nicolas Reul
2
, Jean-Francois Piollé
2
, Frédéric Paul
2
,
Sylvain Herlédan
1
,
 
Ziad El Khoury Hanna
1
, 
Roberto Sabia
3
1. Ocean Data Lab, Plouzane, France
2. Ifremer, Laboratoire d’océanographie physique et spatiale, Plouzane, France
3. 
Telespazio-Vega UK Ltd for ESA, ESRIN, Frascati, Italy
SAG meeting, ESTEC, 3 May 2017
How to provide 
instantaneous answers 
to “simple” questions like:
 
“Which data are available on this area ?”
 
“Which data are co-located at this specific period ?”
 
“Can I see these data and their co-located ones ?
3 ICT infrastructures : Nephelae / Pebsco / Datarmor
FELYX/NAIAD
Datavore/Datalaps/Merginator
Syntool web
Jupyter notebooks
h
t
t
p
:
/
/
f
e
l
y
x
.
o
r
g
Felyx provides a 
free and open source solution
 to the
community for common applications such :
Comparison of various processing algorithms (L1, L2)
Inter-comparison of various datasets performances
Detection and analysis of processing issues
Time series analysis
Allows to create match-up databases with in situ data, time series at
specific locations or data extraction over hurricanes
W
r
i
t
t
e
n
 
i
n
 
p
y
t
h
o
n
,
 
u
s
i
n
g
 
I
f
r
e
m
e
r
 
c
e
r
b
e
r
e
 
l
i
b
r
a
r
y
 
f
o
r
 
d
a
t
a
r
e
a
d
i
n
g
 
i
n
t
e
r
f
a
c
e
 
w
i
t
h
 
a
n
y
 
f
o
r
m
a
t
A
d
a
p
t
a
t
i
o
n
 
e
f
f
o
r
t
 
f
o
r
 
P
I
-
M
E
P
 
m
o
s
t
l
y
 
c
o
n
s
i
s
t
s
 
i
n
 
w
r
i
t
i
n
g
m
i
s
s
i
n
g
 
c
e
r
b
e
r
e
 
d
a
t
a
 
r
e
a
d
e
r
s
 
a
n
d
 
i
n
g
e
s
t
i
o
n
 
o
f
 
t
h
e
 
i
n
 
s
i
t
u
d
a
t
a
 
i
n
t
o
 
f
e
l
y
x
U
s
e
r
 
c
a
s
e
 
1
 
:
 
S
M
O
S
 
S
t
o
r
m
U
s
e
r
 
c
a
s
e
 
1
 
:
 
S
M
O
S
 
S
t
o
r
m
 
 
 
Felyx use case 2 : Sentinel-3 MDB
C
o
l
o
c
a
t
i
o
n
 
w
i
n
d
o
w
 
:
 
2
h
 
(
1
2
h
 
f
o
r
 
A
r
g
o
)
,
 
5
k
m
2
1
 
x
 
2
1
 
p
i
x
e
l
 
b
o
x
e
s
±
 
6
h
 
o
f
 
i
n
 
s
i
t
u
 
d
a
t
a
 
h
i
s
t
o
r
y
In situ data :
Copernicus/CMEMS (Coriolis)
ISAR radiometer on opportunity ships
(delayed-mode)
Sentinel-3 data :
L1 infra-red channels
L2 (SST) – all fields, incl. meteo and
ancillary fields
N
e
w
 
:
 
c
a
l
c
u
l
a
t
e
d
 
S
S
T
 
f
r
o
m
 
s
c
i
e
n
t
i
f
i
c
p
r
o
t
o
t
y
p
e
Other sensor data
Metop-B/AVHRR, MODIS, VIIRS,
MSG/SEVIRI
Resampling of all data to SLSTR grid
Daily aggregated  match-up files on FTP : stack
all matchups into a single file.
 
C
u
r
r
e
n
t
l
y
 
t
w
o
 
v
e
r
s
i
o
n
s
 
:
 
N
R
T
a
n
d
 
R
E
P
 
 
In situ sources for reprocessed MDB
 
 
Naiad
Naiad is the historical Cersat tool for data search (similar to datahub) and cross-over detection
Now completely rewritten as an open source python using same interfaces and dependencies as
felyx => writing the cerbere data readers for felyx make automatically the corresponding
datasets usable in Naiad
Now more oriented as an easy to install/use command line tool
Python API for direct usage in scripts, Jupyter notebooks, etc.
Used for multi-sensor cross detection at Eumetsat (Sentinel-3 cal/val) => cross-overs between
SLSTR, OLCI, Metop AVHRR and IASI, SEVIRI,…
http://naiad.readthedocs.io/en/latest/
Naiad: Principle
Swath (any type of data) are virtually tiled, each tile stored with some metadata (for
search) and reference to the source product (ex : here SMAP)
The tile metadata are stored in a powerfull indexing engine (ElasticSearch, similar to
Solr used in DataHub)
Dynamic search is performed through different dataset indexes for cross-over
detection, with flexible criteria defined by users
Datavore
Quick discovery and preview of Pi-MEP datasets, interactive access to colocated
data, precomputed statistics, raw data
Datavore engine easily ingested most of Pi-MEP data 
  satellite scenes, orbits, L3/L4 grids 
  model, in-situ profiles, buoys 
  images (eg. quicklooks) 
  misc. (eg. storm/icebergs tracks) 
Indexing (by tile, at several zoom levels) 
  time and spatial coverage meta-data 
  pre-computed statistics 
  data (resampled or full) 
Providing API & interactive ways to make request 
Datavore
Use datavore framework to explore time series
Datalaps
Merginator
Explore spatial patterns of numerous datasets at a glance
Merginator
Syntool web
Quick visualization tool
, enabling environmental understanding through easy
and convenient display of images.
  
 => It achieves a responsive display by having pre-rendered "tiles" of images
that are directed to the users display, rather than generating images on the fly.
The images are handled using a robust interface based on OpenLayer software.
A key component of the display is the user's control over the order and
transparency of images, allowing different sensors to be displayed without the
base images being totally obscured.
http://pimep.oceandatalab.com
 
Syntool web
Jupyter Notebooks :
Interactive analysis of data on a distant server via a web browser
Any suggestions/questions?
SAG meeting, ESTEC, 3 May 2017
Technical solution
Slide Note
Embed
Share

The presentation highlights the tools and capabilities of the SMOS Pilot-Mission Exploitation Platform (Pi-MEP) for providing instantaneous answers to data-related questions, showcasing three ICT infrastructures, and outlining FELYX/NAIAD, Datavore, Datalaps, Merginator, Syntool, and Jupyter notebooks. It also introduces Felyx, an open-source solution for data processing and analysis, along with user cases involving SMOS storms, Sentinel-3 MDB colocation, and S3VT event data processing. The presentation showcases the collaboration between different organizations and the adaptation of tools for the Pi-MEP project.


Uploaded on Sep 12, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. SMOS Pilot-Mission Exploitation Platform (Pi-MEP) Presentation of the Pi-MEP tools S bastien Guimbard1, Nicolas Reul2, Jean-Francois Pioll 2, Fr d ric Paul2, Sylvain Herl dan1,Ziad El Khoury Hanna1, Roberto Sabia3 1. Ocean Data Lab, Plouzane, France 2. Ifremer, Laboratoire d oc anographie physique et spatiale, Plouzane, France 3. Telespazio-Vega UK Ltd for ESA, ESRIN, Frascati, Italy SAG meeting, ESTEC, 3 May 2017

  2. How to provide instantaneous answers to simple questions like: Which data are available on this area ? Which data are co-located at this specific period ? Can I see these data and their co-located ones ? 3 ICT infrastructures : Nephelae / Pebsco / Datarmor

  3. Outline FELYX/NAIAD Datavore/Datalaps/Merginator Syntool web Jupyter notebooks

  4. http://felyx.org Felyx provides a free and open source solution to the community for common applications such : Comparison of various processing algorithms (L1, L2) Inter-comparison of various datasets performances Detection and analysis of processing issues Time series analysis Allows to create match-up databases with in situ data, time series at specific locations or data extraction over hurricanes Written in python, using Ifremer cerbere library for data reading interface with any format Adaptation effort for PI-MEP mostly consists in writing missing cerbere data readers and ingestion of the in situ data into felyx

  5. User case 1 : SMOS Storm

  6. User case 1 : SMOS Storm

  7. S3VT 15-17 Feb 2017 Felyx use case 2 : Sentinel-3 MDB Colocation window : 2h (12h for Argo), 5km 21 x 21 pixel boxes 6h of in situ data history In situ data : Copernicus/CMEMS (Coriolis) ISAR radiometer on opportunity ships (delayed-mode) Sentinel-3 data : L1 infra-red channels L2 (SST) all fields, incl. meteo and ancillary fields New : calculated SST from scientific prototype Other sensor data Metop-B/AVHRR, MODIS, VIIRS, MSG/SEVIRI Resampling of all data to SLSTR grid Daily aggregated match-up files on FTP : stack all matchups into a single file. Currently two versions : NRT and REP

  8. S3VT 15-17 Feb 2017 In situ sources for reprocessed MDB

  9. S3VT 15-17 Feb 2017

  10. Naiad Naiad is the historical Cersat tool for data search (similar to datahub) and cross-over detection Now completely rewritten as an open source python using same interfaces and dependencies as felyx => writing the cerbere data readers for felyx make automatically the corresponding datasets usable in Naiad Now more oriented as an easy to install/use command line tool Python API for direct usage in scripts, Jupyter notebooks, etc. Used for multi-sensor cross detection at Eumetsat (Sentinel-3 cal/val) => cross-overs between SLSTR, OLCI, Metop AVHRR and IASI, SEVIRI, http://naiad.readthedocs.io/en/latest/

  11. Naiad: Principle Swath (any type of data) are virtually tiled, each tile stored with some metadata (for search) and reference to the source product (ex : here SMAP) The tile metadata are stored in a powerfull indexing engine (ElasticSearch, similar to Solr used in DataHub) Dynamic search is performed through different dataset indexes for cross-over detection, with flexible criteria defined by users

  12. Datavore Quick discovery and preview of Pi-MEP datasets, interactive access to colocated data, precomputed statistics, raw data Datavore engine easily ingested most of Pi-MEP data satellite scenes, orbits, L3/L4 grids model, in-situ profiles, buoys images (eg. quicklooks) misc. (eg. storm/icebergs tracks) Indexing (by tile, at several zoom levels) time and spatial coverage meta-data pre-computed statistics data (resampled or full) Providing API & interactive ways to make request

  13. Datavore Datavore

  14. Datalaps Use datavore framework to explore time series Datalaps

  15. Merginator Explore spatial patterns of numerous datasets at a glance Merginator

  16. Syntool web Quick visualization tool, enabling environmental understanding through easy and convenient display of images. => It achieves a responsive display by having pre-rendered "tiles" of images that are directed to the users display, rather than generating images on the fly. The images are handled using a robust interface based on OpenLayer software. A key component of the display is the user's control over the order and transparency of images, allowing different sensors to be displayed without the base images being totally obscured.

  17. Syntool web http://pimep.oceandatalab.com

  18. Jupyter Notebooks : Interactive analysis of data on a distant server via a web browser

  19. Any suggestions/questions? SAG meeting, ESTEC, 3 May 2017

  20. Technical solution Nephelae-architecture

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#