Enhancing Research Collaboration with EGI Applications Database
Marios Chatziangelou, et al. <mhaggel@iasa.gr>
Institute of Accelerating Systems and Applications (IASA)
EGI Applications Database
Capabilities
A community driven, central service that stores and provides:
software solutions
(in the form of native software and/or virtual
appliances), originated from almost every scientific area/discipline
reference of scientific
datasets
(
pilot - under development
)
the
programmers
and
scientists
responsible for them
the
publications
derived from the registered items (SW, VA & datasets)
General features (1/2)
dissemination of information
custom RSS/Atom news feeds
news e-mail subscription lists
user focused communication (messaging, requests,
etc)
special dissemination tool for sending ad-hoc
messages to scientists
'follow' button for receiving all the activity related to
a registered item
dissemination features customizable through user
preferences
sharing content with social networks
i
n
f
o
r
m
a
t
i
o
n
r
e
t
r
i
e
v
a
l
advanced searching mechanism (rated search
results)
'faceted search' mechanism for
refinements
quality of information
content tagging, ratting, commenting
per registered item contact expertise
information
problem and comment abuse report
centrally managed quality control
taxonomy
technical classification
scientific classification
tagging
General features (2/2)
AuthN/AuthZ and security
advanced AuthN/AuthZ mechanisms (
simpleSAML
)
integrated with
EGISSO, EGI AAI, eduGAIN
support for
multiple accounts
for accessing user’s
personal profile
internally managed AuthZ
, based on allowed actions,
roles and permissions
everything is under https channel
relations…
… between all the entities listed below, are possible:
–
software
–
virtual appliances
–
datasets
–
persons
–
virtual organizations
–
sites / resource providers
–
organizations
–
projects
integration with AppDB
RESTfull API, supports operations following a CRUD
convention.
flexible API stateless authentication mechanism using
Personal Access Tokens
(no need for X509)
API documentation in good shape
or even, by adapting the AppDB Gadget
(easy – copy & paste, one line of code – no technical
skills required, you may get it
here
)
AppDB already integrated with EGI services and
beyond…
EGI GOCDB
for the list of sites and their end-points
Top-BDII
fetching sites dynamic information
EGI AAI
for auth and high level authorization attributes
Perun
and
EGI Operations Portal
for VO related details + Authorization attributes
OpenAIRE
: retrieving FP7 and H2020 projects and
linked Organizations, persons and in the near
future… publications
Indicative Statistics
Software Marketplace
R
e
g
i
s
t
r
y
f
o
r
S
o
f
t
w
a
r
e
i
t
e
m
s
:
Applications, tools, Workflow frameworks and instances, Science Gateways,
Middleware products, …)
O
f
f
e
r
s
r
e
l
e
a
s
e
m
a
n
a
g
e
m
e
n
t
c
a
p
a
b
i
l
i
t
i
e
s
- unlimited series of releases
- light-weight & collaborative, release
management process
A
c
t
s
a
s
a
r
e
p
o
s
i
t
o
r
y
f
o
r
b
i
n
a
r
y
a
r
t
i
f
a
c
t
s
unlimited number of repositories per register software
generic tarballs, RPM & DEB (32bit/64bit) binaries
multiple flavor / operating system combinations
simplified, web-based, process for uploading the binary artifacts
YUM & APT repositories for automatic distribution
artifacts populated through the UMD Community Repository
Cloud Marketplace (1/3)
R
e
g
i
s
t
r
y
f
o
r
v
i
r
t
u
a
l
a
p
p
l
i
a
n
c
e
s
(
V
A
)
a logical container of versioned image file &
metadata bundles
R
e
g
i
s
t
r
y
f
o
r
s
o
f
t
w
a
r
e
a
p
p
l
i
a
n
c
e
s
(
S
A
)
a logical container of VA versions &
contextualization scripts bundles
V
A
d
i
s
t
r
i
b
u
t
i
o
n
m
e
d
i
u
m
distributing endorsed VAs to the resource
providers/sites
R
e
s
o
u
r
c
e
p
r
o
v
i
d
e
r
s
c
a
t
a
l
o
g
u
e
list of the VAs which are available by each site/resource provider
V
i
r
t
u
a
l
O
r
g
a
n
i
z
a
t
i
o
n
s
(
V
O
)
c
a
t
a
l
o
g
u
e
list of the VAs which are available for each VO member
Cloud Marketplace (2/3)
Users may….
V
i
s
i
t
o
r
s
:
S
e
a
r
c
h
,
d
o
w
n
l
o
a
d
f
o
r
p
e
r
s
o
n
a
l
/
l
o
c
a
l
u
s
e
,
g
e
t
a
l
l
t
h
e
t
e
c
h
n
i
c
a
l
i
n
f
o
r
m
a
t
i
o
n
o
n
h
o
w
a
V
A
c
a
n
b
e
u
s
e
d
u
n
d
e
r
t
h
e
E
G
I
F
e
d
C
l
o
u
d
I
n
f
r
a
s
t
r
u
c
t
u
r
e
,
w
h
e
r
e
(
s
i
t
e
)
i
s
a
v
a
i
l
a
b
l
e
,
t
o
w
h
i
c
h
V
O
t
h
e
u
s
e
r
s
h
o
u
l
d
b
e
m
e
m
b
e
r
f
o
r
u
s
i
n
g
t
h
e
V
A
,
…
.
V
i
r
t
u
a
l
A
p
p
l
i
a
n
c
e
P
r
o
v
i
d
e
r
s
/
S
u
b
m
i
t
t
e
r
s
–
Register Virtual Appliance
–
Create/Update Virtual Appliance versions and link a VM image to them
–
Provide a contextualization script for each VM image
P
r
o
j
e
c
t
&
V
O
M
a
n
a
g
e
r
s
–
Select which of the registered Virtual Appliances, should be pushed to the cloud
infrastructure (sites).
–
The AppDB system takes the responsibility for distributing the “chosen” ones to
the respective Resource providers/sites.
–
AppDB gets feedback from the sites that the Virtual Appliances are indeed
available ready to be used by the users
E
v
e
r
y
u
s
e
r
–
Create a Software Appliance from an existing VA version and a contextualization
script he/she provides
Cloud Marketplace (3/3)
Remote Sites
vmcatcher
local info
system
cloud m/w
AppDB
Cloud Marketplace
•
Insert/update/remove image(s)
•
Publish new vAppliance version
•
Register a vAppliance
•
Create a new vAppliance version
AppDB Image List Store
•
(Re)publish VO or Project’s accepted list of Virtual Appliances
•
Select/update/exclude vAppliances
•
Search for vAppliances
•
G
e
t
a
v
a
i
l
a
b
i
l
i
t
y
&
u
s
a
g
e
d
e
t
a
i
l
s
•
Start/Stop/…. an image
user::visitor
VO or Project
Manager
user::submitter
subscribe and/or get any update
perform VM operations
using rocci command line tool
push info
about the VA
a
v
a
i
l
a
b
i
l
i
t
y
a
n
d
u
s
a
g
e
generate
image_list
Information System
(Top-BDII for
FedCloud sites)
N
e
x
t
l
o
g
i
c
a
l
s
t
e
p
:
p
e
r
f
o
r
m
V
M
o
p
e
r
a
t
i
o
n
s
t
h
r
o
u
g
h
t
h
e
A
p
p
D
B
AppDB VM Operations
T
h
e
o
b
j
e
c
t
i
v
e
(
E
G
I
-
E
n
g
a
g
e
)
:
“
T
h
e
E
G
I
A
p
p
l
i
c
a
t
i
o
n
s
D
a
t
a
b
a
s
e
(
A
p
p
D
B
)
w
i
l
l
e
v
o
l
v
e
f
r
o
m
i
t
s
c
u
r
r
e
n
t
r
o
l
e
a
s
c
a
t
a
l
o
g
u
e
o
f
a
p
p
l
i
c
a
t
i
o
n
s
a
n
d
v
i
r
t
u
a
l
m
a
c
h
i
n
e
s
(
V
M
)
t
o
i
n
c
l
u
d
e
a
g
r
a
p
h
i
c
a
l
u
s
e
r
i
n
t
e
r
f
a
c
e
a
l
l
o
w
i
n
g
a
u
t
h
o
r
i
z
e
d
u
s
e
r
s
t
o
p
e
r
f
o
r
m
b
a
s
i
c
V
M
m
a
n
a
g
e
m
e
n
t
o
p
e
r
a
t
i
o
n
s
”
AppDB
Portal
VM operations dashboard
P
o
r
t
a
l
–
Select VMI (=VA version )
–
or, select VMI (VA version), & VO & site/endpoint & template
–
…. and be redirected to the VM ops dashboard for further adjustments
and execution
The AppDB portal
The VMops dashboard (1/3)
H
i
g
h
l
i
g
h
t
e
d
f
e
a
t
u
r
e
s
f
o
r
t
h
e
e
n
d
-
u
s
e
r
User identification
Based of the aforementioned identification, provide a personalized list of the
VMs that are available for him (based on his VO membership and his role within
the VO, ex. vm_operator@fedcloud.egi.eu)
Create a new topology with one or more instances of a single VM – VM addition
is on going
Attach additional storage to the VM instances
Deploy/Un-deploy a topology
Start/Stop a topology (=start/stop all the VM instances of a topology)
Start/Stop a single VM instance
Provide personalized
views of Topologies and/or
VM instances
The VMops dashboard (2/3)
H
i
g
h
l
i
g
h
t
e
d
t
e
c
h
n
i
c
a
l
f
e
a
t
u
r
e
s
TOSCA compatible
Infrastructure Manager [1] as the OCCI communication layer
Rest API for integration with other services
Single-
S
ign-On with the portal and the rest of the AppDB sub-services
Scalable architecture (one front-end many back-ends for load balancing)
[
1
]
.
h
t
t
p
s
:
/
/
a
p
p
d
b
.
e
g
i
.
e
u
/
s
t
o
r
e
/
s
o
f
t
w
a
r
e
/
i
n
f
r
a
s
t
r
u
c
t
u
r
e
.
m
a
n
a
g
e
r
.
i
m
The VMops dashboard (3/3)
…lets see in action:
https://dashboard.appdb.egi.eu
Status & Open Items
In a pre-release state – VMOps its being tested by a group of experts
First release is expected in the next month
Contextualization script builder
VM Monitoring component
Quite a few issues have been identified and should be internally discussed
and preferably addressed before the first production-grade release:
Policy related:
Common policy for public IPs assignment/availability
Common policy for block storage i.e. storage size, need for device id or not, etc.
Security based policies i.e. who should be able to revoke suspicious instances,
user should provide his own key or let the AppDB to create (and keep!) one for him
Infrastructure related:
Information mismatch between Information system and OCCI
Image update issue on sites and/or issues on Information system
Incompatible image files
Summary
Users may ….
AppDB
vAppliance
Software
Datasets
VM image
EGI fedCloud
infrastructure
Cont. scripts
Block storage(s)
Publish or pick one
Publish or
pick one
Publish or pick one
The EGI Applications Database is a central service facilitating the storage and sharing of software solutions, scientific datasets, and publications across various disciplines. Users can benefit from advanced search mechanisms, information dissemination tools, and quality control features. The system integrates with RESTful APIs, offers stateless authentication, and supports multiple accounts, ensuring secure access. Furthermore, it seamlessly interacts with EGI services, enabling efficient collaboration and resource management within the research community.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
EGI Applications Database Marios Chatziangelou, et al. <mhaggel@iasa.gr> Institute of Accelerating Systems and Applications (IASA) www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
Capabilities A community driven, central service that stores and provides: software solutions (in the form of native software and/or virtual appliances), originated from almost every scientific area/discipline reference of scientific datasets (pilot - under development) the programmers and scientists responsible for them the publications derived from the registered items (SW, VA & datasets) 2
General features (1/2) informationretrieval advanced searching mechanism (rated search results) 'faceted search' mechanism for refinements dissemination of information custom RSS/Atom news feeds news e-mail subscription lists user focused communication (messaging, requests, etc) special dissemination tool for sending ad-hoc messages to scientists 'follow' button for receiving all the activity related to a registered item dissemination features customizable through user preferences sharing content with social networks quality of information content tagging, ratting, commenting per registered item contact expertise information problem and comment abuse report centrally managed quality control taxonomy technical classification scientific classification tagging 3
General features (2/2) integration with AppDB RESTfull API, supports operations following a CRUD convention. flexible API stateless authentication mechanism using Personal Access Tokens (no need for X509) API documentation in good shape or even, by adapting the AppDB Gadget (easy copy & paste, one line of code no technical skills required, you may get it here) AuthN/AuthZ and security advanced AuthN/AuthZ mechanisms (simpleSAML) integrated with EGISSO, EGI AAI, eduGAIN support for multiple accounts for accessing user s personal profile internally managed AuthZ, based on allowed actions, roles and permissions everything is under https channel relations between all the entities listed below, are possible: software virtual appliances datasets persons virtual organizations sites / resource providers organizations projects AppDB already integrated with EGI services and beyond EGI GOCDB for the list of sites and their end-points Top-BDII fetching sites dynamic information EGI AAI for auth and high level authorization attributes Perunand EGI Operations Portal for VO related details + Authorization attributes OpenAIRE: retrieving FP7 and H2020 projects and linked Organizations, persons and in the near future publications 4
Software Marketplace Registry for Software items: Applications, tools, Workflow frameworks and instances, Science Gateways, Middleware products, ) Offers release management capabilities - unlimited series of releases - light-weight & collaborative, release management process Acts as a repository for binary artifacts unlimited number of repositories per register software generic tarballs, RPM & DEB (32bit/64bit) binaries multiple flavor / operating system combinations simplified, web-based, process for uploading the binary artifacts YUM & APT repositories for automatic distribution artifacts populated through the UMD Community Repository 6
Cloud Marketplace (1/3) Registry for virtual appliances (VA) a logical container of versioned image file & metadata bundles Registry for software appliances (SA) a logical container of VA versions & contextualization scripts bundles VA distribution medium distributing endorsed VAs to the resource providers/sites Resource providers catalogue list of the VAs which are available by each site/resource provider Virtual Organizations (VO) catalogue list of the VAs which are available for each VO member 7
Cloud Marketplace (2/3) Users may . Visitors: Search, download for personal/local use, get all the technical information on how a VA can be used under the EGI FedCloud Infrastructure, where (site) is available, to which VO the user should be member for using the VA, . Virtual Appliance Providers/Submitters Register Virtual Appliance Create/Update Virtual Appliance versions and link a VM image to them Provide a contextualization script for each VM image Project & VO Managers Select which of the registered Virtual Appliances, should be pushed to the cloud infrastructure (sites). The AppDB system takes the responsibility for distributing the chosen ones to the respective Resource providers/sites. AppDB gets feedback from the sites that the Virtual Appliances are indeed available ready to be used by the users Every user Create a Software Appliance from an existing VA version and a contextualization script he/she provides 8
Cloud Marketplace (3/3) Remote Sites Search for vAppliances Information System (Top-BDII for FedCloud sites) Get availability & usage details cloud m/w Start/Stop/ . an image local info system push info about the VA availability and usage perform VM operations using rocci command line tool perform VM operations through the AppDB Next logical step: vmcatcher user::visitor subscribe and/or get any update AppDB Image List Store AppDB generate image_list Cloud Marketplace AppDB portal Publish new vAppliance version (Re)publish VO or Project s accepted list of Virtual Appliances Insert/update/remove image(s) Select/update/exclude vAppliances Create a new vAppliance version Register a vAppliance VO or Project Manager AppDB portal user::submitter AppDB portal 9
AppDB VM Operations The objective (EGI-Engage): The EGI Applications Database (AppDB) will evolve from its current role as catalogue of applications and virtual machines (VM) to include a graphical user interface allowing authorized users to perform basic VM management operations AppDB Portal VM operations dashboard 10
The AppDB portal Portal Select VMI (=VA version ) or, select VMI (VA version), & VO & site/endpoint & template . and be redirected to the VM ops dashboard for further adjustments and execution 11
The VMops dashboard (1/3) Highlighted features for the end-user User identification Based of the aforementioned identification, provide a personalized list of the VMs that are available for him (based on his VO membership and his role within the VO, ex. vm_operator@fedcloud.egi.eu) Create a new topology with one or more instances of a single VM VM addition is on going Attach additional storage to the VM instances Deploy/Un-deploy a topology Start/Stop a topology (=start/stop all the VM instances of a topology) Start/Stop a single VM instance Provide personalized views of Topologies and/or VM instances 12
The VMops dashboard (2/3) Highlighted technical features TOSCA compatible Infrastructure Manager [1] as the OCCI communication layer Rest API for integration with other services Single-Sign-On with the portal and the rest of the AppDB sub-services Scalable architecture (one front-end many back-ends for load balancing) [1]. https://appdb.egi.eu/store/software/infrastructure.manager.im 13
The VMops dashboard (3/3) lets see in action: https://dashboard.appdb.egi.eu 14
Status & Open Items In a pre-release state VMOps its being tested by a group of experts First release is expected in the next month Contextualization script builder VM Monitoring component Quite a few issues have been identified and should be internally discussed and preferably addressed before the first production-grade release: Policy related: Common policy for public IPs assignment/availability Common policy for block storage i.e. storage size, need for device id or not, etc. Security based policies i.e. who should be able to revoke suspicious instances, user should provide his own key or let the AppDB to create (and keep!) one for him Infrastructure related: Information mismatch between Information system and OCCI Image update issue on sites and/or issues on Information system Incompatible image files 15
Summary Users may . Publish or pick one vAppliance Publish or pick one EGI fedCloud infrastructure Software VM image Cont. scripts AppDB Publish or pick one Datasets Block storage(s) 16
Thank you for your attention. Questions? www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
