Overview of EGI Online Storage Service Architecture and Interfaces
EGI Online Storage service provided by EOSC-hub under Horizon 2020 program offers File Storage, Block Storage, and Object Storage options. Users can access highly scalable storage with different interfaces. The service allows data access through standard protocols, fault tolerance replication, global identifiers, and flexible data organization. Access policies include acceptable usage and business models. Various technologies like DPM, OpenStack Cinder, and OpenStack Swift are utilized with different interfaces like SRM, HTTP/WebDAV, S3, and more.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
EGI Online Storage Dissemination level: Public eosc-hub.eu @EOSC_eu EOSC-hub receives funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No. 777536.
Content Motivation and driving consideration about the service Service architecture and interfaces: overview - How the user can access the service E.g.: REST, GUI, CLIs, etc. - Service options and attributes Acceptable Usage Policy (AUP) Access policy and business model Use cases Documentation/tutorial/information 7/17/2018 2
Motivation Different service options to allow : - Access data through different standard protocols and replicate data across different providers to increase fault-tolerance. - Assign global identifiers to files - Access highly-scalable storage from anywhere - Control the data sharing - Organise data using a flexible hierarchical structure - Extends storage resources for Compute instances 7/17/2018 3
Architecture and interfaces 7/17/2018 4
Service architecture and interfaces The service comprices 3 main options: File Storage, Block Storage and Object Storage. File Storage: - Highly scalable storage system accessible from anywhere allowing to easily share data through different standard interfaces. - Technology: DPM, dCache, StoRM - Interfaces: SRM, HTTP/WebDAV, XRootD, GSIFTP, CDMI 7/17/2018 5
Service architecture and interfaces Block Storage: - block-level storage solution that allows to expand the storage capacity of instances in the EGI Federated Cloud. - increase storage without increasing the size or capacity of the instance or by provisioning new ones; delete servers, keeping data intact and ready for the next time is need it. - Technology: Openstack Cinder - Interfaces: POSIX for access, OCCI and Openstack CLI for management 7/17/2018 6
Service architecture and interfaces Object Storage: - Manages data as objects. Each object includes the data itself, a variable amount of metadata, and a globally unique identifier. - Cloud object storage allows relatively inexpensive, scalable and self-healing retention of massive amounts of unstructured data - Technology : OpenStack Swift, CEPH - Interfaces: S3, Swift, CDMI 7/17/2018 7
Service access File Storage - The service exposes standard interface for data access that can be contacted via CLIs and libraries - For grid based protocols ( SRM, gsiftp, Xrootd) the gfal2 library and CLI is the recommended choice - For HTTP/Webdav many known clients are available, like curl Block Storage - Storage is accessible via POSIX interface. 7/17/2018 8
Service access Object Storage - for Openstack Swift implementation, the OpenStack CLI can be used to access objects and containers - for Object storage instances exposing the CDMI and S3 protocols any CDMI and S3 command line client can be used - Many libraries are also available for programmatic access via S3 , CDMI and Swift protocols 7/17/2018 9
Service options and attributes File Storage - Storage capacity [TB]: [1, 5, 10, other] - Technology: [DPM, dCache, StoRM] - Special requirements: e.g. performance, close to the computational resources Object Storage - Storage capacity [TB]: [1, 5, 10, other] - Interfaces: [CDMI, S3, Swift] - Special requirements: e.g. performance, close to the computational resources 7/17/2018 10
Service options and attributes Block Storage - Storage capacity [TB]: [1, 5, 10, other] - Special requirements: e.g. performance, close to the computational resources 7/17/2018 11
Acceptable Usage Policy No dedicated AUP and Condition of Use exist - Existing EGI-wide documentation will apply 7/17/2018 12
Access policies and Funding models Access policies: - Policy-based, - Wide-access, - Market-driven Payment model: - Sponsored (free for a certain quota) - Wide-access - Market-driven 7/17/2018 13
Featured use cases - File Storage File storage can be used for storing and accessing files on the infrastructure as input to EGI HTC computations The EGI Workflow Management systems ( DIRAC) is able to access files stored in File Storage instances via different protocols, and schedule the computation in order to be executed close to where the input files are stored By accessing input files close to computing resources the overall efficiency of data intensive computations increase significantly. 7/17/2018 14
Featured use cases - Block storage A block-level storage solution allows to expand the storage capacity of VM instances in the EGI Federated Cloud and offers the lowest possible latency for applications Data are kept intact when deleting VMs if stored in block storage therefore is highly recommended 7/17/2018 15
Featured use cases - Object Storage Object Storage can be used for storing massive amount of unstructured data within the EGI Federated Cloud Object storage is ideal for applications that need durability, availability and scalability for data storage The main uses cases are: - Backups and Data Archive, AI and ML , Cloud Native applications 7/17/2018 16
Documentations/Links https://docs.egi.eu/users/online-storage/ https://www.egi.eu/services/online-storage/ https://marketplace.egi.eu/34-online-storage 7/17/2018 17
@EOSC_eu eosc-hub.eu
