Virtual Research Environments as-a-Service: Advancing Collaborative Research Efforts

Slide Note

Explore how Virtual Research Environments as-a-Service, exemplified by D4Science, enhance research collaboration by providing operational ecosystems, genealogy testbeds, and e-Infrastructures. These environments cater to diverse user needs, from data management to cost reduction, offering capabilities like data access, analysis, and validation to a global community of scientists.

dunk_a Follow

Uploaded on Sep 30, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Virtual Research Environments as-a-Service Pasquale Pagano, CNR pasquale.pagano@cnr.it EGI Community Forum 10-13 November 2015 Bari, Italy www.d4science.org 1 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Outline E-Infrastructure History Context as a Service Capabilities Virtual Research Environment D4Science Features Numbers gCube www.d4science.org 2 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

e-Infrastructure An operational combination of digital technologies (hardware and software), resources (data and services), communications (protocols, access rights and networks), and people and organizational structures needed to support research efforts and collaboration in the large www.d4science.org 3 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Genealogy Testbed: Virtual Research Environment DILIGENT 2004-2007 D4Scienc e 2008-2010 Operational: several use cases (fisheries), gCube became an open source project Operational Ecosystem: use cases (marine biodiversity use cases), D4Science born to go beyond project lifetime D4Science-II 2010-2012 Operational HDI: exploit D4Science, iMarine CoP, >1500 active users iMarine 2012-2014 www.d4science.org 4 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

D4Science operates VREs for +2000 scientists in 44 countries, integrating +50 heterogeneous data providers, executing +20,000 processes/month; providing access to over a billion quality records in repositories worldwide, with 99,7% service availability. D4Science hosts +40 VREs www.d4science.org 5 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Born to serve user needs I need to host my applications in a secure and scalable environment I need to maintain my database I need to backup my data I need to securely delivery my data to a set of known people I want to offer a flexible sharing, storage, reporting, search and retrieval tool I need to manage and analyze data I need to manage the full data life-cycle from import to validation, curation, harmonization and publication I need to offer to my team a powerful tool to manage code-lists I need to reduce the costs of data maintenance of my dept. Applications Capacities I need to access authoritative data I need to simplify the access to my data I need to mash-up statistical and geospatial data I need to analyse my big datasets I need to validate my datasets and provide a standard access to them Data www.d4science.org 6 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Distinguishing capabilities of the e-infrastructure D4SCIENCE www.d4science.org 7 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

The D4Science infrastructure Hybrid Data Infrastructure combining over 500 software components into a coherent and centrally managed system of hardware, software, and data resources www.d4science.org 8 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

D4Science enables e-infrastructure by ... Overcoming administrative boundaries Exploiting private and commercial providers Integrating geographically distributed computing infrastructure Operation Built on SLAs Support monitoring, auditing, reporting, and notification Providing service allocations, deployment, monitoring, and operation Trust Privacy, governance, and attribution Ensuring uniform resource and data access Security, trusted network www.d4science.org 9 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Storage as Service to host and maintain data Database Cloud Storage Geographical DB High-availability Standard Ready-to-use Scalable Reliable Secure Policies Standard Privacy and Attribution www.d4science.org 10 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Applications as a Service to curate and manage data Metadata Generation Geospatial Data Biodiversity Data Statistical Data Textual Data Harmonization Disambiguate Validate Integrate and Consistency Check Data Exchange OGC protocols DarwinCore SDMX DublinCore www.d4science.org 11 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Computing as Service to process and extract knowledge Scalable Easy to Manage Across Boundaries Tailored Elastic Heterogeneous High Throughput Map-Reduce Parallel R Assignment of Computing Assignment of Processors Virtual Research Environment www.d4science.org 12 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Computational Engine Not another cloud computer platform but a platform where executions can be repeated, compared, discussed, logged Not another computational engine but a platform where interdisciplinary tools and services can be easily contributed by the communities www.d4science.org 13 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Two exploitation models Dispatcher Tools (R, Java, ) must be uploaded to the storage Executable is deployed on the worker nodes assigned to the VRE Data are made accessible to the worker nodes according to the specification provided Monitoring, accounting, failures management, partial re-execution, sharing, and repeatability are granted Application Framework Predefined data splitting models are provided A large array of models and algorithms can be exploited to define custom workflows Large array of algorithms to compare results are provided www.d4science.org 14 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Virtual Research Environment to access, share and collaborate Share Communicate Post Favourite Connection Organize Dynamic Secure Policy Driven Database Tables Workflow Files www.d4science.org 15 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Virtual Research Environment a distributed and dynamically created environment where subset of resources (data, services, computational, and storage resources) regulatedby tailored policies (e.g. data encryption with VRE specific key, quota on service calls and storage usage, ) are assigned to a subset of users via interfaces for a limited timeframe at little or no cost for the providers of the participatory data e- infrastructures L. Candela, D. Castelli, P. Pagano (2013) Virtual Research Environments: An Overview and a Research Agenda. Data Science Journal, Vol. 12 www.d4science.org 16 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

VRE Definition Metadata Applications Simple and effective process to define a new environment Data Configuration www.d4science.org 17 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Applications vs Services Logical View Applications Data Hardware Configuration Registry Physical View Data Software, Tools, Services www.d4science.org 18 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Application Bundles https://www.gcube-system.org/catalogue-of-applications AppsCube ConnectCube BiolCube To aid modelling and analysing of distribuition data, comparing checklists, and producing maps To facilitate data publication with appropriate tools including semantic technologies To develop applications interfacing gCube facilities GeosCube StatsCube IceCube To assist tabular data validation, data enrichment ad efficient analytical tools To support deployment, operation & mgmt of a gCube-based infrastructure To properly access, consume and produce geospatial information www.d4science.org 19 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

VRE Exploitation Exploited for Public VREs (used to offer an application environment to a subset of users of a community) and Private VREs (used for experiments, data access and preparation, and data analytics) Fully operational VRE available in one hour Software deployment and hardware setup completely hidden Evolving needs of its users completely supported www.d4science.org 20 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Entity as Resource Entity As a resource As a service Server, Storage Container Software Data Publication/Discovery Lifecycle management Failure management Authorization-accounting Access Orchestrate Reference Software as Resource: transforms servlets-based applications/services in e-Infrastructure resource Container as Resource: transforms standard servlets-based container in e-Infrastructure resource Federated Sources as Resource: transforms external DBs and Repositories in e-Infrastructure resource Algorithm as Resource: for any new algorithm, model, procedure, workflow, it is possible to manage policies and assign dedicated Hardware and Storage resources Dataset and single product as Resource: for any dataset, map, timeseries, code list, . It is possible to manage policies and monitor their exploitation www.d4science.org 21 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

SmartGears a set of Java libraries that turn Servlet-compliant containers and applications into infrastructure resources, transparently. gCube Wiki turn software and containers into resources what does it mean ? www.d4science.org 22 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

SmartGears [cont.] Software-as-Resource Container-as-Resource software and nodes we can discover use without hardcoded knowledge Actual Solution Zero constraints human solutions not practical, often impossible automated solutions local enabling software, remotely controlled monitor and control take actions when not operational management tasks compile and publish descriptions track and change status enforce policies dedicate to user groups change policies, assign roles www.d4science.org 23 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

gCube: One stable open-source platform gCube enables the D4Science HDI Statistics form openhub.net/p/gCube www.d4science.org 24 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

Multi-tenant Delivery Model Dynamic deployment Hosting Resource Lifecycle Monitoring Accounting Security Infrastructure as a Service VRE BiolCube ConnectCube GeosCube StatsCube Software as a Service FeatherWeightStack SmartGears ApplicationSupportLayer SOA3 Platform as a Service www.d4science.org 25 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano

References / Links D4Science: http://www.d4science.org Policies https://wiki.d4science.org/D4Science_Deployment_and_Operation:_Policies Procedures https://wiki.d4science.org/D4Science_Deployment_and_Operation gCube: http://www.gcube-system.org Catalogue of Applications https://www.gcube-system.org/catalogue-of-applications Software Key Features https://wiki.gcube-system.org/GCube_Features Developer Guide https://wiki.gcube-system.org/Developer%27s_Guide FeatherWeightStack https://wiki.gcube-system.org/Featherweight_Stack SmartGears https://wiki.gcube-system.org/SmartGears gCube APIs https://wiki.gcube-system.org/GCube_Application_Programming_Interface Administration Guide https://wiki.gcube-system.org/Administrator%27s_Guide www.d4science.org 26 EGI CF 2015 - Virtual Research Environments as-a-Service P. Pagano