Advancements in iMarine Infrastructure for Biology Research
Expanding the iMarine infrastructure is a significant initiative focusing on maritime and freshwater biology research. Led by Pasquale Pagano, the project aims to establish a data infrastructure supporting the Ecosystem Approach to Fisheries Management. By harmonizing data and utilizing niche modeling algorithms, the project leverages various open-source software and data sources to enhance biodiversity and fisheries studies. The D4Science infrastructure plays a key role in managing the data life cycle and providing a standard access interface for users. This initiative enables secure and scalable environments for data hosting and delivery, ultimately reducing the costs of data maintenance.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Expanding the iMarine infrastructure for maritime and freshwater biology Pasquale Pagano D4Science Technical Director Senior Researcher at CNR ISTI (Italy) www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
iMarine Objective Sept 2014 Nov 2011 Apr 2016 Launch an Initiative aimed at establishing and operating a data Infrastructure supporting the principles of the Ecosystem Approach to Fisheries Management and Conservation of Marine Living Resources Expanding the iMarine infrastructure 10/5/2024 2
Address data harmonization Niche modelling algorithms Open SDMX - CLM Open Source software GBIF MyOceanSeaDatane Other sources Ocean environment Taxonomy EoL Emodnet Biology WORMS VLIZ t T2 OBIS IOC Aquamap s FishBase NEAFC FIN DG-MARE National DOF CRIA ESTAT IRD ICES FAO RDB Ecoscope FIGIS Courtesy by Marc Taconet (FAO) Biodiversity Fisheries Expanding the iMarine infrastructure 10/5/2024 3
Concepts The initiative the visionary leadership The e-infrastructure the operational platform The system the enabling sw system Expanding the iMarine infrastructure 10/5/2024 4
The D4Science infrastructure iMarine is exploiting D4Science, Hybrid Data Infrastructure combining over 500 software components into a coherent and centrally managed system of hardware, software, and data resources. Expanding the iMarine infrastructure 10/5/2024 5
Born from the user needs to host applications in a secure and scalable environment to maintain and preserve data to securely delivery data to known users Capacities to manage the full data life-cycle from import to harmonization and publication to validate and analyze datasets to provide a standard access interface Applications to access authoritative datasets to mash-up data to reduce the costs of data maintenance Data Expanding the iMarine infrastructure 10/5/2024 6
D4Science enables e-infrastructure by Overcoming administrative boundaries Integrating geographically distributed computing infrastructure Operation Built on SLAs Exploiting private and commercial providers Support monitoring, auditing, reporting, and notification Providing service allocations, deployment, monitoring, and operation Trust Privacy, governance, and attribution Ensuring uniform resource and data access Security, trusted network Expanding the iMarine infrastructure 10/5/2024 7
gCube turns Entity to Resource Entity As a resource As a service Publication Lifecycle mgmt. Failure mgmt. Authorization Accounting Access Orchestrate Reference CPU, Storage Container Software Data Software as Resource: transforms servlets-based applications/services in e-Infrastructure resource Container as Resource: transforms standard servlets-based container in e-Infrastructure resource Federated Sources as Resource: transforms external DBs and Repositories in e-Infrastructure resource Algorithm as Resource: for any new model, procedure, workflow, it is possible to manage policies and assign dedicated Hardware and Storage resources Dataset and single product as Resource: for any dataset, map, timeseries, code list, . It is possible to manage policies and monitor their exploitation Expanding the iMarine infrastructure 10/5/2024 8
Logical vs. Physical Entity View Logical View Applications Data Hardware Configuration Registry Physical View Software, Services Data Expanding the iMarine infrastructure 10/5/2024 9
Virtual Research Environment to access, share and collaborate Share Communicate Post Favourite Connection Collaborate Secure Policy Controlled Database Tables Workflow Files Expanding the iMarine infrastructure 10/5/2024 10
Virtual Research Environment a distributed and dynamically created environment where subset of resources (data, services, computational, and storage resources) regulated by tailored policies are assigned to a subset of users via interfaces for a limited timeframe at little or no cost for the providers of the participatory data e-infrastructures L. Candela, D. Castelli, P. Pagano (2013) Virtual Research Environments: An Overview and a Research Agenda. Data Science Journal, Vol. 12 Expanding the iMarine infrastructure 10/5/2024 11
VRE Exploitation Exploited for Public VREs (used to offer an application environment to a subset of users of a community) and Private VREs (used for experiments, data access and preparation, and data analytics) Fully operational VRE available in one hour Software deployment and hardware setup completely hidden Evolving needs of its users completely supported Expanding the iMarine infrastructure 10/5/2024 12
VRE Definition Metadata Applications Simple and effective process to define a new environment Data Configuration Expanding the iMarine infrastructure 10/5/2024 13
gCube Computational Engine Not another computational engine but a platform where executions can be repeated, compared, discussed, logged Not another computational engine but a platform where interdisciplinary tools and services can be easily contributed by the communities Expanding the iMarine infrastructure 10/5/2024 14
gCE: exploitation models Tools (R, Java, ) must be uploaded to the storage Dispatcher Tools are then deployed and configured on the worker nodes assigned to the VRE Data are made accessible to the worker nodes according to the specification provided Monitoring, accounting, failures management, partial re-execution, sharing, and repeatability are granted Predefined data splitting models are provided Platform A large array of models and algorithms can be exploited to define custom workflows Large array of algorithms to compare results are provided Expanding the iMarine infrastructure 10/5/2024 15
gCE Architecture Soap via https Client Infrastructure Registry Soap via https REST via https Soap via https WS Resource (status) gCE 1 gCE 2 Executor Executor Executor JMS Active Message Queue SM Queue Storage Client Hibernate Distributed Storage (NoSQL) Relational Storage Expanding the iMarine infrastructure 10/5/2024 16
gCE vs. Executor duties gCube Computational Engine gCube Executor Runs in Tomcat empowered by SmartGears partitions a job in tasks puts messages in the queue for each tasks Each message contains instructions to download the software to execute, the way to configure it, and the data to fetch publishes a notification on the Registry to alert the workers reads the feedback and resubmit the task if needed collect and merge the outputs More than 100 models are available (data analysis, harmonization, partitioning, clustering, ) Runs in Tomcat empowered by SmartGears at startup SmartGears publishes the profile in the Registry making it available for one or more VREs wakes up when alerted by the Registry starts fetching tasks from the queue and puts feedback messages back to another queue The instructions found in the messages are used to execute the software Expanding the iMarine infrastructure 10/5/2024 17
Executor: some features A Worker accepts a message only if this is compliant with its execution environment The messages on the queue contain compliancy specifications for the Workers: R version, Operating System, Java version, Workers are not dedicated to a specific job but they interleave tasks from different jobs Workers can be added and removed at any time Upon activation a worker will participate to the running computations after few seconds Expanding the iMarine infrastructure 10/5/2024 18
Actions Create a Virtual Appliance for the Executor Exploit jOCCI for Start, Suspend, Stop, etc A new gCube service is planned to monitor the queues and spoon workers into the computational environment according to the QoS and Quota assigned to the VRE Exploit the Contextualization to configure the VM at start-up by specifying the VREs to join The VO (which the VRE belongs to) key will be needed into the worker Create a Virtual Appliance for the SmartExecutor (Standard Executor yet equipped with models supporting direct OGC WPS calls) Expanding the iMarine infrastructure 10/5/2024 19
Thank you for your attention. Questions? Read more at www.d4science.org www.gcube-system.org more with pasquale.pagano@isti.cnr.it Discuss www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142
