Introduction to CernVM File System (CVMFS)

Slide Note

CernVM File System (CVMFS) is a scalable, reliable, and low-maintenance software distribution service used by various computing communities. It was developed to support High Energy Physics (HEP) collaborations and has since been adopted by other fields like Medical, Space, and Earth Sciences. Using standard technologies, CVMFS allows easy access to software on distributed computing infrastructure, making it versatile for different environments. The service architecture provides various options such as REST, GUI, and CLIs, with an acceptable usage policy and clear documentation available. CVMFS consists of elements like Stratum-0, responsible for maintaining the repository and ensuring data availability through a hierarchy of caches. Overall, CVMFS simplifies software deployment and access across different computing setups.

jjac Follow

Uploaded on Sep 30, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

CernVM File System - CVMFS Scalable, reliable and low-maintenance software distribution service eosc-hub.eu @EOSC_eu EOSC-hub receives funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No. 777536.

Content Motivation and driving consideration about the service Service architecture and interfaces: overview - How the user can access the service E.g.: REST, GUI, CLIs, etc. - Service options and attributes Acceptable Usage Policy (AUP) Access policy and business model Use cases Documentation/tutorial/information 9/30/2024 2

Motivation CVMFS was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide distributed computing infrastructure used to run data processing applications Technology has been also adopted by other computing communities outside HEP Medical, Space and Earth, Biological sciences 9/30/2024 3

What is CVMFS? Built using standard technologies (fuse, sqlite, http, squid and caches) Files and directories are hosted on standard web servers and get distributed through a hierarchy of caches to individual nodes Software needs one single installation, then it is available at any site with CernVM-FS client installed and configured Mounted in the universal /cvmfs namespace at client level 9/30/2024 4

What is CVMFS? Because of standard technologies, it can be used everywhere - Not only Grid, but also Cloud, local clusters - Add CernVM-FS client to a VM image => /cvmfs space automatically available Due to increasing interest, technology has been optimized also for access to conditions data and other auxiliary data - File chunking for large files - Garbage collection on revision level - File system history 9/30/2024 5

CVMFS Elements Stratum-0 - The single source of the CernVM File System (the repository) - Protected read/write instance maintained by a dedicated release manager machine or installation box - The CVMFS tool kit is used to publish the current state of the repository on the installation box - Data stored in CVMFS is converted into a CVMFS repository during the process of publishing - Basically an Apache HTTP server 9/30/2024 6

CVMFS Elements Stratum-1 - Or Replica Server - Standard web server that uses the CVMFS tool kit to create and maintain a mirror of the CVMFS repository served by a Stratum-0 server Site-local proxy servers - Between the clusters of nodes with CernVM-FS clients and the Stratum-1 servers - Used to reduce load on the Stratum-1 servers by local caching Stratum-0 server, Stratum-1 servers and the site-local proxy servers can be seen as content distribution network 9/30/2024 7

EGI CVMFS Service and Infrastructure Maintained and operated by EGI Resource Centres Stratum-0 service - Maintains and publishes the current state of the repositories (part of the egi.eu CVMFS namespace) Stratum-1 service - Worldwide network of servers replicating the egi.eu repositories Uploader service - Provides upload area for egi.eu repositories - GSI-OpenSSH interface 9/30/2024 8

EGI CVMFS Service and Infrastructure Topology July 2019 9/30/2024 9

Service options and attributes Service attributes: - Size of the repository - Expected grow of the repository (if possible) - DNs (Distinguished Names) of VO users responsible for maintaining the repository in the future - Emails of the VO users responsible for the maintaining the repository in the future 9/30/2024 10

Additional info EGI PROC22 in place to describe process of creating a repository within the EGI CVMFS infrastructure The involved computing community is expected to be supported by more that one EGI production Resource Centre Ideally the maintainers of the repository should have access to a X.509 certificate 9/30/2024 11

Acceptable Usage Policy EGI AuP To use the service users have to apply the Virtual Organisation AUP 9/30/2024 12

Use Case - CVMFS within WLCG All LHC experiments are making use of CVMFS for: - Production software distribution - Experiment conditions database access - Continuous integration and testing software All involved repositories are part of the cern.ch CVMFS namespace 9/30/2024 13

Use Case - CVMFS within WLCG All LHC experiments are making use of CVMFS for: Production software distribution Experiment conditions database access Continuous integration and testing software All involved repositories are part of the cern.ch CVMFS namespace CernVM-FS content distribution network for the cern.ch domain cvmfs.readthedocs.io/en/latest/cpt-replica.html#fig-stratum1 Stratum1 replica servers are located in Europe, the U.S. and Asia One protected read/write instance (Stratum-0) located at CERN is feeding up the public, distributed mirror servers A distributed hierarchy of proxy servers fetches content from the closest public mirror server 9/30/2024 14