Actor-Based Services: Geo-Distribution Challenges and Solutions
Explore the complexities of geo-distributing actor-based services, handling multiple users, devices, data centers, and concurrent code execution. Delve into client app architecture, challenges of correctness under failures, and the quest for the right abstractions in composed services. Discover strategies for recoverable, scalable, and localized service architectures in the face of failures and increasing latencies due to geodistribution.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
GEO-DISTRIBUTION OF ACTOR-BASED SERVICES OOPSLA 2017, Vancouver OOPSLA 2017, Vancouver PHILIP A. BERNSTEIN, Microsoft Research, USA SEBASTIAN BURCKHARDT, Microsoft Research, USA SERGEY BYKOV, Microsoft, USA NATACHA CROOKS, University of Texas, Austin, USA JOSE M. FALEIRO, Yale University, USA GABRIEL KLIOT, Google, USA ALOK KUMBHARE, Microsoft Research, USA MUNTASIR RAIHAN RAHMAN, Microsoft, USA VIVEK SHAH, University of Copenhagen, Denmark ADRIANA SZEKERES, University of Washington, USA JORGEN THELIN, Microsoft Research, USA
Multiple users, devices, data centers. Code runs concurrently, reactively, interactively.
Client App GUI State Client App Local State GUI State Binding Client App Local State Binding GUI State Local State Binding Storage save/restore Storage save/restore Storage Messages save/restore Messages Compute Layer Layer Compute Layer Layer Compute Compute Layer Layer Compute Compute Layer Layer Compute Compute Layer Layer Compute Compute Layer Layer Compute Compute Layer Compute Compute Layer Data Center Data Center Messages Messages Messages Storage Backend Storage Backend Replica Storage Backend Replica Storage Backend sync sync Challenge: correctness under failures and topology changes. Ad-hoc plumbing problematic. Need robust application-level abstractions.
RIGHT ABSTRACTION? COMPOSED SERVICES SERVICES Stateful Service Stateless Service CLIENT Stateful Service Stateless Service Model View Stateless Service Stateful Service
COMPOSED SERVICES recoverable retryable restartable Stateful Service Stateless Service Localized failure model. Stateful Service Stateless Service Stateless Service Stateful Service
COMPOSED SERVICES M M M M M M Localized scalability. M M M M M M M M M M M M
GEO-DISTRIBUTED SERVICES same problem: failures, scalability new problem: even higher latencies (ultimately limited by speed of light)
COMPOSED SERVICES localized geo-configuration.
GEO-SERVICES VS. GEO-STORAGE Not our problem: how to durably persist information so it does not get lost. Our problem: how to build geo-distributed services on top of storage that is potentially slow (because it is geo-remote or geo-replicated). geo-distributed caching, not geo-replicated storage.
SOLUTIONS How to write elastic services? Orleans Virtual Actor Framework. How to hide geo-latency for services with high locality? Single-Instance Caching. How to hide geo-latency for services without locality? Multi-Instance Caching w/ consistency choices.
Orleans is an open-source actor framework for C# Makes it easy to build fault-tolerant, elastic, composed services https://dotnet.github.io/orleans/ Many internal and external users, e.g. all of Halo s web services Introduces the grain (a.k.a. virtual actor) abstraction
Object Actor Grain a.k.a. virtual actor Service A grain is a micro-service w/ built-in recovery
DISTRIBUTED VIRTUAL ACTORS, CALLED GRAINS Runtime Implementation Programmer view manages elastic cluster with grain directory (distributed hash table) grains represent application entities grains identified by key only (not location) activates grains when called, deactivate when idle grains exist perpetually (no creation or deletion) automatically recovers from failures Game 00121 Game 00120 Game 0012 1 User Tom User Ben User Lev User Ben User Tom User Lev ... Server 1 Server 2 Server 100
NOT JUST OBJECTS, NOR ACTORS objects, actors Grains new + GC always there x = new User(); x.DoSomething(); x = User[ Frank673 ]; x.DoSomething();
TYPES OF GRAINS VOLATILE PERSISTENT state cleared after failure/restart state kept durably in external storage Usage Examples: Usage Examples: user profile monitoring (e.g. list of connected users) account balance
DURABILITY Orleans runtime does not implement durability on its own. Persistent grains use external storage service for durability. Orleans Cluster Developers can use provided interfaces for common storage services, or hand-craft their own Separate Storage Service Storage Storage
ORLEANS MINI-EXAMPLE : CHATROOM APP Clients post messages to chatrooms Administrators can ban users Implementation: two types of grains, both persistent A ChatRoom grain per chatroom, stores list of posts A User grain per user, stores user info
ORLEANS MINI EXAMPLE (C#) GRAIN INTERFACES users are identified by a string public interface IUser : IGrainWithStringKey { Task Ban(); // marks user as banned Task<bool> MayPost(); // returns true iff user is not banned } grain calls are asynchronous chatrooms are identified by a Guid public interface IChatRoom : IGrainWithGuidKey { Task<bool> Post(string user, string msg); // adds a user post Task<List<string>> Read(); // returns all posts }
public class ChatRoom : Orleans.Grain<List<string>>, IChatRoom { public async Task<List<string>> Read() { return State; } public async Task<bool> Post(string user, string msg) { // call usergrain to check if banned var usergrain = GrainFactory.GetGrain<IUser>(user); var hasPermission = await usergrain.MayPost(); type of persisted state construct grain ref. asynchronous grain call if (!hasPermission) { return false; } State.Add(msg); await WriteStateAsync(); // persist in storage return true; } } for durability: wait for writeback to complete
ORLEANS GEO-DEPLOYMENTS: MULTI-CLUSTERS
We treat Storage Service as separate layer Storage - programmer has full control over storage system/layout/placement/replication
Note: Perfect solutions (fast read + fast write + strong consistency) are impossible. [Lipton,Sandberg 88] [Attiya,Welch 94] or be replicated. Storage Storage
GEO-SOLUTION 1: THE BASIC API Like an exclusive cache.
MINIMAL CHANGE same API + semantics just add attribute to class [SingleGlobalInstance] public class ChatRoom : Orleans.Grain<List<string>>, IChatRoom { public async Task<List<string>> Read() { return State; } public async Task<bool> Post(string user, string msg) { // call usergrain to check if banned var usergrain = GrainFactory.GetGrain<IUser>(user); var hasPermission = await usergrain.MayPost(); if (hasPermission) { State.Add(msg); await WriteStateAsync(); // persist in storage
GLOBAL SINGLE-INSTANCE Route all grain calls to a single grain activation in the cluster where it already exists, or in the caller s cluster (if no activation exists) Cross-cluster mutual exclusion protocol ensures single activation eventually 29
PERFORMANCE? DEPENDS. r/w great for volatile grains with local load. - all accesses local. 30
PERFORMANCE? DEPENDS. r/w r/w ok for volatile grains 150ms RTT with global load. half of all accesses (whether reads or writes) suffer 150 ms RTT. this may still be o.k. for scenarios where strong consistency is required. (e.g. lobby grain for Halo) 31
PERFORMANCE? DEPENDS. r/w ok for persistent grains with local load and local storage 10ms RTT reads: local writes: 10ms RTT 32
PERFORMANCE? DEPENDS. r/w ok for persistent grains with local load and remote storage and mostly reads reads: local 33
PERFORMANCE? DEPENDS. r/w not great for persistent grains with local load and remote storage and many writes write latency: 150ms RTT throughput limited to 6.66 req/s 34
PERFORMANCE? DEPENDS. r/w r/w terrible for worst-case 150ms RTT write latency can be 300ms! 35
GEO-SOLUTION 2: THE VERSIONED API New API & consistency model for incremental storage. Supports multi-instance (replication). Supports flexible consistency tradeoff.
GOAL: SATISFY ALL ACCESSES LOCALLY. r/w r/w ? requires - consistency model - protocol implementation. 37
GOAL: SATISFY ALL ACCESSES LOCALLY. r/w r/w ? requires - consistency model ? - protocol implementation. 38
NEW STATE API : EVENTS Separate event (what happened) from effect of event (how it modifies state) Achieve consistency via background consensus on a global event sequence. public class ChatState { public List<string> Messages = new List<string>(); public void Apply(MessagePostedEvent e) { Messages.Add(e.Content); } } public class MessagePostedEvent { public string Content; }
NEW API public class ChatRoom : Orleans.EventSourcing.JournaledGrain<ChatState>, IChatRoom { public async Task<List<string>> Read() { return TentativeState.Messages; } reads local approximation of current state, always available new base class public async Task Post(string user, string msg) { EnqueueEvent(new MessagePostedEvent() {Content = msg}); } } public class ChatState { public List<string> Messages = new List<string>(); schedules event for global sequencing, never blocks public void Apply(MessagePostedEvent e) { Messages.Add(e.Content); } } public class MessagePostedEvent { public string Content; }
GSP CONSISTENCY MODEL Global Sequence Protocol [ECOOP 15] synchronizes state in the background, off the critical path. Similar to CRDTs, but for a user-defined data type Guarantees eventual consistency, and per-grain causal consistency Supports strong consistency (linearizability) on demand, via fence-like synchronization primitives similar but not identical to TSO (for precise comparison see [DISC 17])
Application Code GSP: LOCAL STATE Local Updates Event Confirmed State Event raised events are stored in queue Local Event Local Reads Tentative State Confirmed state = Latest known global state Tentative state = events applied to copy of confirmed state
Event GSP: AUTOMATIC PROPAGATION LOCAL GLOBAL Confirmed State Event Local Event Tentative State Background process applies queued events to global state Network Remote Global state interleaved sequence of events from all participants (in order, no duplication) Actual State Global State
Event GSP: AUTOMATIC PROPAGATION GLOBAL LOCAL Confirmed State Event Local Event All changes to global state are pushed to confirmed state Tentative State Network Events are removed from queue when echoed, i.e. included in confirmed state Remote Global State Actual State
WHAT KIND OF INCONSISTENCY? Tentative event sequence is a subsequence of final sequence. No reordering, but missed events. Chat Example: A> who wants to volunteer? B> Sure, I have time. C> I ll do it. A> Thanks, C.
ON-DEMAND LINEARIZABILITY public async Task<List<string>> LinearizableRead() { await RefreshNow(); brings local state up to date with current global state } return State.Messages; public async Task LinearizablePost(string user, string msg) { RaiseEvent(new MessagePostedEvent() { Content = msg }); } await ConfirmEvents(); waits until all local events are confirmed to be part of the global state
LINEARIZABLE CONDITIONAL UPDATE public async Task LinearizablePost(string user, string msg) { try { await RaiseEvent(new MessagePostedEvent() { Content = msg }); } catch(InconsistentStateException) { ... } } - applies event only if global state unchanged (not in paper or release yet... recent addition)
WHERE/WHAT IS THE GLOBAL STATE ? global state simply represents global consensus on a event sequence actual representation depends on chosen configuration and protocol. Many possibilities. in currently implemented protocols for persistent grains: latest global state is stored in external storage for volatile grains: latest global state is stored in leader instance
SELECTABLE CONFIGURATIONS & DISTRIBUTED PROTOCOLS USED Basic API Single-Instance New API Single-Instance New API Multi Instance Volatile leader-based GSP single act. single act. single act. single act. notifications Persistent simple writeback batching CAS batching CAS
EXPERIMENTS Latency experiments: results as discussed earlier Throughput experiments: simulate order processing (mini TPC-W) measure throughput for different configurations of inventory grains
