EGI-InSPIRE SA2 Software Provisioning Overview
Discover the achievements and objectives of EGI-InSPIRE SA2 in software provisioning, including liaising with product teams, introducing new technological solutions, and coordinating the UMD release team. Learn about the URT achievements in re-assessing support calendars for UMD products. Explore the role of EGI in providing verified software for the EGI infrastructure.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EGI-InSPIRE SA2 Software provisioning Peter Solagna EGI.eu peter.solagna@egi.eu www.egi.eu www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE RI-261323
Outline SA2 Overview Achievements Technology Management Software Provisioning Resource usage Plans for the future SA2 Software provisioning 2 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
SA2 Overview WP Task Beneficiary Total PMs The Netherlands Spain Portugal Czech Republic Greece Germany Italy Sweden WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-E WP5-N WP5-N WP5-N WP5-N WP5-N WP5-N WP5-N WP5-N WP5-N TSA2.1 TSA2.2 TSA2.2 TSA2.3 TSA2.3 TSA2.4 TSA2.4 TSA2.5 TSA2.5 TSA2.5 TSA2.5 TSA2.5 TSA2.6 TSA2.6 TSA2.6 TSA2.6 TSA2.6 TSA2.6 TSA2.6 TSA2.6 TSA2.6 EGI.eu CSIC 36 35 35 35 35 60 95 26 12 29 11 6 3 22 6 10 22 12 6 6 12 18 Countries 19 Beneficiaries 514 PMs 43 FTEs LIP CSIC LIP CESNET GRNET CESNET KIT-G INFN VR-SNIC NORDUNET CESNET SA2 Effort SA3 8% SA2 6% JRA1 4% NA1 4% NA2 16% KIT-G CSIC CNRS INFN UI SAV NA3 4% SA1 58% STFC (Oxford) TSA2.1: Peter Solagna, EGI.eu TSA2.4: Kostas Koumantaros, GRNET FOM UU TSA2.2: Enol Fernandez, CSIC TSA2.3: Esteban Freire, FCTSG TITLE OF PRESENTATION - REPLACE 3 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
SA2 Objectives Liaise with the Product Teams providing the software relevant for the EGI infrastructure. Introduce new technological solutions relevant for the evolution of EGI Release verified software in the Unified Middleware Distribution repositories for the EGI communities SA2 Software provisioning 4 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
UMD Release Team Purposes Coordination of the product teams focusing on interoperations between products Communication forum between UMD team and the product teams and between the product teams Lightweight activity Mailing list, wiki pages Periodic phone meetings (every 2 weeks) Volunteer participation Good coverage of the products distributed in the UMD: gLite middleware, Gloubs, ARC, dCache, QCG INFN, CERN, Desy, PSC, Nordugrid SA2 Software provisioning 5 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
URT achievements Re-assessment of the support calendar for the UMD products Most of the UMD products had supporting calendars ending in April 2014 Assessed through the URT representatives the new calendars, and the plans for new major releases Negotiated extension of dCache and StoRM security support SA2 Software provisioning 6 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Technology Coordination Board Five meetings during PY4 Agendas focused on the post-EMI, post-EGI organisation of the technology providers activities For PY5 TCB will evolve as an umbrella for technology related working groups Activities focused on user communities technology needs SA2 Software provisioning 7 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Software provisioning process Acceptance criteria definition Acceptanc e criteria verification Staged Rollout UMD Release New release from the Technology Provider SA2 Software provisioning 8 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Software releases in PY4 Total of 17 UMD updates, 170 product updates UMD-1: only security updates Fully decommissioned UMD-2: minor and revision updates Reached end of life May 31st UMD-3: Major release and minor updates Support will continue UMD-1 1 security update 4 minor updates 3 revision updates (1 security) 1 major release 6 minor updates 2 revision updates 4 releases 1 release UMD-2 Unified Middleware Distribution UMD-3 EGI PKIX trust anchors SAM 9 SA2 Software provisioning 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Acceptance criteria definition New releases of the Acceptance Criteria Document Revision 6: October 2013 Revision 7: April 2014 New quality criterion Support for IPv6 Consolidated criteria Common and generic criteria applicable to all the products Focus on Security and interoperability with the EGI infrastructure Simplified documentation for verifiers Increased level of automation in the test process Simplified templates Extensive wiki documentation, tests examples for many products available on public repository New test developed using Behavior Driven Development Involved external contributors in the verification process Successfully done for ARC and QCG products SA2 Software provisioning 10 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Acceptance criteria verification Product releases are tested by deploying the software on a private cloud Virtual machines configured by default to use IPv6 Important to test in pre-production a feature that cannot be easily tested in production At least one product (VOMS) failed support for IPv6 in some components Federated Cloud tools and technical solutions are used to manage also the private cloud Automatic distribution of the images of the tested components in the EGI Images catalogue The testbed is in fact- a subset of a resource centre in the EGI Federated Cloud Configuration templates made available in a public repository SA2 Software provisioning 11 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Effort for verification Total of ~350 PPAs processed Average verification time per component: 3.8 hours (was 6h) 350 100 Total verificatiomn effort (h) 90 Number of components 300 80 250 70 63 58 released 55 60 200 50 50 150 40 32 30 100 20 18 15 20 13 9 8 50 6 10 2 2 2 2 0 0 SA2 Software provisioning 12 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Staged rollout Early adoption is a volunteer activity carried out by some EGI production sites Central coordination provided by SA1.3 Staged rollout numbers: Number of early adopter sites 81(PY3 + 10%) NGIs contributing to SR 28 Average number of early adopters per product Number of SR test performed during PY3 2 160 SA2 Software provisioning 13 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Release in UMD Releases are extensively tested for dependencies issues Automated process using a tool integrated with the verification testbed Support infrastructure Adapted to the new technology ecosystem Integrated with the Fedora Community repositories Extended to support generic sources for the software Continue the provisioning of the collaboration tools SA2 Software provisioning 14 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Improvements in the UMD repositories EMI yum/apt UMD Untested UMD Tested UMD Production IGE yum/apt SA2 Software provisioning 15 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Improvements in the UMD repositories EMI IGE EPEL UMD Untested UMD Tested UMD Production HTTP PT specific SA2 Software provisioning 16 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Added value of UMD Provisioning Process UMD repositories Single location to download middleware and find information about release notes and support calendars Protection from non tested in production updates in the community repositories Software verification and Staged Rollout Prevent disruptive bugs to affect production infrastructure Document minor bugs and workaround Provide quick feedback to developers Integrate the installation notes provided by developers 23 products were rejected during verification or staged rollout (6% of the total) 40 GGUS tickets submitted during software provisioning 20 15 10 5 0 Mar-13 Apr-13 May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Less urgent Urgent 17 SA2 Software provisioning 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
UMD upgrade activities Coordinated upgrade campaigns are needed when: Widely deployed software versions are ending support life cycle. Critical requirements demand for deployment of new software within a short timeline Three main decommissioning campaigns during PY4: Decommissioning of UMD-1 At the end of PY3 Decommissioning of services non supporting SHA-2 July November 2013 Decommissioning of UMD-2 March June 2014 Coordination with product teams to extend support where the new release is not ready for production SA2 Software provisioning 18 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Upgrade campaigns of deployed software 450 Number of service instances to upgrade 400 350 300 SHA-2 250 200 150 UMD-2 100 50 0 Operational effort for all the levels of operation Technology effort to make available in time newer versions of software that is fit for production SA2 Software provisioning 19 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Customer satisfaction survey Distributed among sites one month ago 25% of sites answered Results 60% of the sites are using UMD repositories to install middleware Of which, more than 90% think that UMD is a real added value in they day-by-day work Benefits of using UMD 1. Protection from non verified updates released in EPEL or similar community repositories 2. Single repository that contains most of the services deployed by site manager 3. Additional quality assurance Where UMD can improve? Release more rapidly in production new releases from product teams Test more the new releases in staged rollout SA2 Software provisioning 20 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Working groups Evaluate new technologies Foster the adoption of new services GPGPU task force Analysis of the technical requirements to expose GPGPU resources through the current services Gather experiences from the resource centres and the users CVMFS task force Foster the adoption of this system for the distribution of application software among user communities Two workshops organised during the Community Forum 2014 SA2 Software provisioning 21 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Use of resources in SA2 Task PM Declared Committed PM Achieved PM TSA2.1 7.9 9.0 87.7% TSA2.2 19.6 17.5 111.8% TSA2.3 21.1 17.5 120.3% TSA2.4 34.4 38.8 88.9% TSA2.5 0.0 82.8 Total: 83.0 100.2% Task PM Declared Committed PM Achieved PM TSA2.6 58.6 58.6 51.0 51.0 115.0% 115.0% Total: Small deviations in SA2 tasks: Over spending TSA2.3 spent in improving the verification tools, on top of the verification effort Under spending in TSA2.4 mainly in the collaboration tools sub-task. The effort has been used for a mini-project, this did not affected the service provisioning Over spending TSA2.6 mainly due to the effort used to roll in production Cloud services 22 SA2 Software provisioning 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
UMD after EGI-InSPIRE UMD activities have been identified as core for the EGI infrastructure Will continue to be provided funded by the council members fees and by the partners currently engaged in such activities (Ibergrid, CESNET and GRNET) OLAs in place for the service provisioning Acceptance criteria One new release of the document per year Verification of 250 PPAs/year Staged rollout coordination Continue with effort comparable to the past EGI-InSPIRE Software provisioning infrastructure Maintenance and extensions where needed SA2 Software provisioning 23 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Summary The acceptance criteria verification process has been consolidated to be more efficient and easy to perform Verification effort has been reduced Implemented the extensions needed to adapt the process to the new technology ecosystem Support for UMD will continue beyond EGI-InSPIRE as an EGI.eu core service SA2 Software provisioning 24 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
Questions Members of the EGI-InSPIRE collaboration thank the EC for supporting EGI TITLE OF PRESENTATION - REPLACE 25 2-3 July 2014 www.egi.eu EGI-InSPIRE RI-261323
