Efficient Bag-of-Task Application Execution on Federated Clouds
Arrebol offers an easy and efficient solution for executing Bag-of-Task applications on federated clouds, speeding up research cycles and utilizing cloud infrastructures for optimal performance. The service automates tasks, creating VMs, managing execution, fetching outputs, and destroying VMs seamlessly. Demonstrated at the OpenStack Summit in Barcelona, this innovation presents a streamlined approach to managing eScience applications efficiently.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Arrebol: Easy and Efficient Execution of Bag-of-Task Applications on Federated Clouds Federal University of Campina Grande Systems and Computing Department Distributed Systems Laboratory Prof. Francisco Brasileiro, fubica@computacao.ufcg.edu.br Prof. Andrey Brito, andrey@computacao.ufcg.edu.br OpenStack Summit, Barcelona, October 2016
Outline Motivation Managing Bag-of-Tasks applications The Arrebol service Connecting multiple clouds using Fogbow Short demonstrations Q&A OpenStack Summit, Barcelona, October 2016
Motivation Many eScience applications are structured as a Bag-of-Tasks (BoT) Usually: they have sporadic demand for very large computing capacities tasks are independent, thus, their execution can be parallelized in a very easy way OpenStack Summit, Barcelona, October 2016
Motivation The research cycle could be speeded up, if applications could complete faster Cloud infrastructures are the perfect match for BoT applications Capacity Costs Capacity Time Time OpenStack Summit, Barcelona, October 2016
Managing the execution of BoT applications in the cloud VM 3. Start remote execution 1. Create VM 2. Stage input files 4. Monitor execution 5. Retrieve output data 6. Destroy VM ?USER CLOUD SERVICE OpenStack Summit, Barcelona, October 2016
Automating the execution of BoT applications in clouds with Arrebol job: label: simulation_experiment_50 requirements: Glue2RAM >= 1024 AND Glue2CPU >= 1 init: sudo apt-get update sudo apt-get -y install simulator ARREBOL SERVICE final: task: put /home/fubica/experiments/50/input/file01 /tmp/file01.in run_simulator i /tmp/file01.in o /tmp/file01.out get /tmp/file01.out /home/fubica/experiments/50/output/file01 task: put /home/fubica/experiments/50/input/file02 /tmp/file02.in run_simulator i /tmp/file02.in o /tmp/file02.out get /tmp/file02.out /home/fubica/experiments/50/output/file02 VM rm /tmp/*.out VM ??????? VM VM VM 1. User submits a job (CLI) 2. Arrebol service parses the job 3. Arrebol service creates VMs 4. Arrebol service manages execution 5. Arrebol service fetches outpus 6. Arrebol service destroys VMs VM VM USER CLOUD SERVICE OpenStack Summit, Barcelona, October 2016
Arrebol in action Demo #1: the Arrebol service OpenStack Summit, Barcelona, October 2016
Increasing the capacity available Cloud providers usually impose restrictive resource quotas Promotes fair share of resources Facilitates long term capacity planning Capacity can be extended in several ways: If using a public cloud Multi-clouds: accessing other public providers (directly or via a broker service) If using a private cloud Cloudbursting on public clouds/multi-clouds Federating with other private clouds Cloudbursting + federating OpenStack Summit, Barcelona, October 2016
Using Fogbow to extend the capacity of the available infrastructure Fogbow is a suite of open source software projects developed by the Federal University of Campina Grande Supports access to: Multiple public clouds Federation of private clouds Any combination of the above Supports the deployment of opportunistic green clouds using conventional desktops OpenStack Summit, Barcelona, October 2016
Fogbow & Arrebol in action Demo #2: using a federation of clouds + OpenStack Summit, Barcelona, October 2016
Fogbows architecture OpenStack Summit, Barcelona, October 2016
Multi-level A&A A&A is performed at multiple levels A local request carries federation credentials that are A&A by a Federation Identity Provider A&A federation credentials are associated to a token that is used to map the federation user to the local cloud credentials that will be used to access the local cloud The allocation manager performs A&A of a remote request considering both this token and the identity of the requesting allocation manager Mapping is autonomously defined by each cloud administrator OpenStack Summit, Barcelona, October 2016
Plugin-based approach Interoperability plugins make it simple to support different cloud orchestrators Behavioral plugins allow cloud administrators to customize the federation functionalities that they deploy in their local cloud E.g. prioritization policies OpenStack Summit, Barcelona, October 2016
Internet-friendly communication A messaging service allows membership and allocation managers to communicate with each other locally and remotely, without requiring them to have their endpoints publicly accessible A reverse tunneling service provides public IP access to all virtual machines created in the private clouds, even if the local cloud offers only private IPs to these virtual machines OpenStack Summit, Barcelona, October 2016
Success stories so far Fogbow has been used to federate private clouds in a research project that involved Brazilian and European institutions (http://eubrazilcloudconnect.eu/) VOMS was used to implement the federation level A&A This federation had OpenStack and OpenNebula clouds The Brazilian NREN (RNP) is currently experimenting the use of Fogbow to federate its private clouds, as well as those of some of its clients (http://portal.naf.lsd.ufcg.edu.br/) CAFe federation of identity providers (Shibboleth-based) is used to implement authentication at the federation level This federation has OpenStack and CloudStack private IaaS clouds OpenStack Summit, Barcelona, October 2016
Q&A Thanks for your attention! Useful links: Arrebol (http://arrebol.lsd.ufcg.edu.br/) Fogbow (http://fogbowcloud.org/) Distributed Systems Lab at UFCG (http://lsd.ufcg.edu.br/) OpenStack Summit, Barcelona, October 2016
