Xcache Possibilities and Effective Use in HPC Environments

Xcache, a versatile caching system, offers various possibilities for data transfer, remote access, data streaming, and HPC optimization. The system enables efficient data reuse and provides enhanced data integrity and accessibility features. Explore the potential applications and considerations for integrating Xcache into high-performance computing environments.

• Caching System
• Data Transfer
• Remote Access
• HPC Optimization
• Data Streaming

mckernan_d Follow

Uploaded on Oct 05, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Xcache Possibilities 4th US ATLAS HPC Meeting LBNL Berkeley CA September 26, 2019 Andrew Hanushevsky, SLAC http://xrootd.org

2. Xcache and File Transfer Writable Xcache caches not supported This is a strictly pull model We do support FRM caching for full files Push or pull mode possible Not clear we need this at all Well, beyond what we have, unless We want a mix of Rucio and on-demand styles Possible to do but would require FTE s September 26, 2019 2

3. Xcache and Remote Access Perhaps the most used model Essentially provides a CDN for remote data Definitely, the most successful application However Maintaining data integrity expectations is difficult Bad data is sticky in a cache and hard to find Work on the way to improve this situation Use TLS to weed out transmission errors Overkill but the fastest solution for now Enhance file system integrity for data at rest September 26, 2019 3

4. Xcache and Data Streaming Block caching simulates true streaming Prefetching practically eliminates data jitter Anything missing? Perhaps deletion upon close Technically, single use streams so data not needed However, purge takes care of this eventually Anything else to consider Server-less Xcache may be very relevant Certainly applicable for single use streams September 26, 2019 4

5. Xcache and HPC Optimum access uses HPC FS as cache Xcacheruns on DTN s Except for random outages best location Allows RDMA access to fully cached files E.g. Lustre Xcache + direct cache access @ NERSC Workable but not the best solution Requires file to be fully cached (low probability) Has security implications in terms of access Best to add RDMA support to Xcache Requires additional FTE effort September 26, 2019 5

6. Xcache Effective Use The following is true of any cache Effective use is proportional to data reuse Only two known proposals on this Virtual Placement from Ilija This is simply a Rucio placement optimization Non-simulated (i.e. real) effectiveness unknown Cache affinity scheduling from Andy Requires Panda to add cache as a scheduling resource Concept is effective for LSST query scheduling Will it work with Panda? September 26, 2019 6

7. Xcache Optimum Workflow Since we need high reuse . Caches most suitable to analysis Will not help production, unless Used as a streaming appliance for event delivery To reduce disk usage and steady the stream Refer to the previous slide on streaming September 26, 2019 7