DDPC Overview: November 2023 - Global Data Processing Center Insights
"Explore the insights into the operations of the Data Processing Center in November 2023, including global data exchange, specific software details, pipeline management, storage technologies, and more. Uncover the intricate workings of the center's communication systems and processing capabilities."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
LISA DDPC OVERVIEW 21/11/2023 APC Bastien Barthet
DDPC OVERWIEW NOVEMBER 2023 1.DDPC Global Overview 2.Inner DDPC communications 3.Main DCC Specific Software 4.Pipeline Runner 5.Storage 6.Prototyping Lab 7.SDK 8.Planning 2
DDPC OVERWIEW NOVEMBER 2023 DDPC Global Overview 1 interaction with the SOC 1 entry point : Data Exchange facility In charge of the security of both exchanges and accesses Holds : o Storage o Processing o Monitoring o Labs 3
DDPC OVERWIEW NOVEMBER 2023 Inner DDPC communications Main DCC content : o Pipeline Manager o Own DCC (optional) o Global Index + Global Storage Main DCC interfaces : o Send processing requests to a DCC Each DCC content : o Own Pipeline Runner o Own Local Storage Each DCC interfaces : o Register its processing capabilities to the Main DCC o Search/index data from/to the Global Index o Push/Pull Data to/from the Global Storage 4
DDPC OVERWIEW NOVEMBER 2023 MAIN DCC SPECIFIC SOFTWARE Knows DCCs processing capabilities Sends Processing Requests Monitors Pipeline Processing 5
DDPC OVERWIEW NOVEMBER 2023 Pipeline Runner Reactive Design o Each Element gets a processing event o Each Element reacts to a processing event o Each Element sends a processing event Business Rules Engine o Reacts to processing events (processing modules completion) o Contains the pipeline orchestration logic Processing Module o Reacts to processing events (processing task to perform) o Contains the scientific algorithm 6
DDPC OVERWIEW NOVEMBER 2023 STORAGE Global Storage o Data exchange and persistence feature o Object Storage Technology (in 2023) o Only accessed by specific Modules (Ingestion and Dispatch) ? Local Storage o Posix storage o Accessed by each Processing module (algorithm) Global Index o Both global and local data indexation CO2 footprint o Global Storage use for pertinent results persistence only o Prefer Local Storage for intermediate/temporary Data 7
DDPC OVERWIEW NOVEMBER 2023 o PROTOTYPING LAB Local Processing capabilities o Able to execute a Pipeline on a laptop Prototyping instance Prototyping Tools o Pipelines development, test and execution o Data visualization, analysis, management, and simulation Extended Processing capabilities o User can send a Pipeline processing request to a Prototyping Lab instance (then executed on an HPC like cluster) Large scale Processing capabilities ? o User can send a Pipeline processing request to a DDPC like infrastructure (executed on the most capable cluster) 8
DDPC OVERWIEW NOVEMBER 2023 o SDK Framework Access Layer o ProcessingEvent management (event bus communication) o Dataset payload management (Ingestion & Dispatch Modules) Algo Control Interface o Algorithm execution (start, stop) Data Management o Libraries based on the Data Model Development Best Practices o SDK available for Python and C/C++ (??) languages o Containers usage more development opportunities o Build operations, Quality and Security checks supported by the DevSecOps activities 9
DDPC OVERWIEW NOVEMBER 2023 o PLANNING 2024 Q2 Q3 Q4 2025 Q2 Q3 Q4 2026 Q2 Q3 Q4 2027 Q2 Q3 Q4 2028 Q2 Q3 Q4 2029 Q2 Q3 Q4 2030 System coordination, Design and Operations DDPC Architecture V0 V1 V0 V1 design Data Model for System & Scientist coordination DevSecOPS & Deployment Define & code CI/CD pipeline steps V0 V1 V0 V1 V0 SDK V0 V1 DCC Pipeline Runner V0 V1 Prototyping Lab V0 V1 Data Storage V0 V1 default data storage solution metadata storage solution (index) common access library Data Exchange Facility V1 Main DCC Specific Software Components DCCs installation and configuration V1 V1 Expertise Lab V1 10
THANK YOU FOR YOUR ATTENTION ANY QUESTIONS ? 11
