Evolution of LinkedIn's Service Architecture: From Monolith to Microservices

Slide Note
Embed
Share

LinkedIn transitioned from a monolithic architecture to a microservices-based approach, introducing REST along the way. Starting with their original codebase in Java, Servlets, JSP, and JDBC, they evolved to a service-oriented architecture with fine-grained services. Challenges such as test failures and complex dependencies between services were faced, leading to a shift towards microservices for better scalability and flexibility. Solutions included continuous delivery, codebase decomposition, and better-defined boundaries between tiers.


Uploaded on Sep 22, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. From a monolith to microservices + REST The evolution of LinkedIn s service architecture by Steven Ihde and Karan Parikh (LinkedIn)

  2. Leo Our original codebase Java, Servlets, JSP, JDBC Leo Oracle 4

  3. Remote Graph Graph: member-to-member connection graph Complex graph traversal problems not suited to SQL queries RPC was used to keep it separate from Leo Our first service 5

  4. Remote Graph RPC Leo Graph JDBC Databus Oracle 6

  5. Mid/Back Tier Services Back tier services encapsulate data domains Mid tier services provide business logic We applied the service pattern to many domains, e.g. member profiles, job postings, group postings 7

  6. Front Tier Services Front tier services aggregate data from many domains Transform the data through templates to present to the client Should be stateless for scaling purposes 8

  7. Service Explosion Over 100 services by 2010 Most new development occurring in services, not Leo Site release every two weeks 9

  8. Architectural Challenges Test failures Incompatibilities Complex orchestration Rollback difficult or impossible Complex dependencies between services 10

  9. Microservices? Services were fine grained But monolithic build and release process did not allow us to realize the benefits of microservice architecture 11

  10. Solutions Continuous delivery Break apart the code base Devolution of control Strict backwards compatibility Better defined boundaries between tiers 12

  11. Continuous Delivery Shared trunk Pre- and post-commit automated testing Easy promotion of builds to production environment 13

  12. Decentralize Codebase Separate, independently buildable repositories Shared trunk within each repository Versioned binary dependencies between repositories 14

  13. Devolution of Control Service owners control release schedule, release criteria Service owners are responsible for backwards compatibility Services must release independently 15

  14. Backwards compatibility Insulates teams from each other at runtime Allows service owners to deploy on their own schedule without impacting clients 16

  15. Boundaries Between Tiers Limit aggregation to the front tier Limit crosstalk in the back tier: superblocks 17

  16. Boundaries Between Tiers 18

  17. Java RPC Difficult to maintain backwards compatibility Verb-centric APIs Use case specific APIs Difficult to navigate the proliferation of APIs 19

  18. Rest.li plus Deco equals Microservices at LinkedIn

  19. What is Rest.li? Rest.li is an open source REST framework for building robust, scalable RESTful architectures using type-safe bindings and asynchronous, non-blocking I/O. Primarily JSON over HTTP. 21

  20. Why Rest.li? Polyglot (frontend) ecosystem - Java, Scala, Python, Node.js, Objective-C Uniform service interfaces (REST) 22

  21. The Rest.li stack Rest.li Data layer and RESTful operations D2 Dynamic discovery and load balancing R2 Network Communication 23

  22. Request Response (R2) REST abstraction that can send messages over any application layer protocol (HTTP, PRPC (old custom LinkedIn protocol)) Client - fully asynchronous Netty Server - Jetty, Netty (experimental) Rest.li D2 R2 24

  23. Dynamic Discovery (D2) Apache ZooKeeper Dynamic server discovery Client side software load balancing D2 service Rest.li D2 R2 25

  24. Rest.li Data using PDSCs (Pegasus Data Schemas) RESTful API that developers use to build services CRUD + finders + actions API and data backwards compatibility checking Rest.li D2 R2 26

  25. 830 Rest.li resources. 90 billion Rest.li calls/day across multiple datacenters. 65% service-to-service calls.

  26. What is deco? 28

  27. Aside: Normalized Domain Models Links over inclusion (denormalization) URNs are fully qualified foreign keys InfluencerPost Member Long id Long id String title String content URN author String firstName String lastName String summary URN company 29

  28. What is Deco? URN resolution library What data you want, not how you want it 1 Time 2 3 30

  29. Deco Example: Influencer Post Dummy Post by Karan Parikh at LinkedIn /influencerPosts /profiles /companies Hi QCon! /influencerPosts 31

  30. Deco Example: Influencer Post deco://influencerPosts/123?projection=(title, content, author~(firstName, lastName, company~(companyName))) 32

  31. Three services. One client call. Deco.

  32. Rest.li plus Deco equals Microservices at LinkedIn

  33. How Rest.li enables Microservices Rest.li + D2 facilitate domain specific services Services can easily configure clients via D2 D2 helps us scale the architecture 35

  34. How Deco enables Microservices Deals with service explosion Abstracts away services from clients 36

  35. Challenges Coordinating a massive engineering effort. (LiX to the rescue!) Ensuring uniform RESTful interfaces Performance 37 Rest.li API Hub

  36. Wins All languages talk to the same service Developer productivity Reduction of hardware load balancers Ability to expose APIs directly to third parties 38

  37. LinkedIn Microservices

  38. Questions?

  39. References and links Rest.li: http://rest.li/ Rest.li API Hub: https://github.com/linkedin/rest.li-api-hub Rest.li user guide: https://github.com/linkedin/rest.li/wiki/Rest.li-User-Guide Modeling resources with Rest.li: https://github.com/linkedin/rest.li/wiki/Modeling-Resources-with-Rest.li LinkedIn engineering blog posts about Rest.li: http://engineering.linkedin.com/architecture/restli-restful-service- architecture-scale http://engineering.linkedin.com/restli/linkedins-restli- moment LinkedIn s GitHub projects http://linkedin.github.io/ 43

Related


More Related Content