Application Log Processing Methodologies in Service-Oriented Architecture
Understanding the methodologies for processing application logs in a service-oriented architecture, including data collection, purpose, programming language toolkits, centralized log collection, non-discardable logs collection, event/enterprise bus integration, cloud platform data collection, and Kubernetes logging practices.
- Log Processing
- Service-Oriented Architecture
- Data Collection
- Programming Language
- Centralized Collection
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Application log processing methodologies in a service oriented architecture Bogdans Ozerkins LU, 2020
Application data collection Human generated Machine generated Low volumes High volumes Persistent storage Volatile storage References data management systems References to technical components
Application logs purpose CENTRALIZED LOG COLLECTION SECURITY AUDIT LOGS DATABASE REPLICATION LOGS
Programming language toolkit Focused on interface Error handling / reporting Freedom of format (default: raw) Freedom of storage (default: local file) Freedom of delivery mechanism (default: direct connection)
Centralized log collection Write logs to file Ship files to destination Write directly Use special agent
Non-discardable logs collection Save to persistent storage (database) Ship to centralized service Replication logs Hight-Availability storage
Event / Enterprise bus Database replication Data centralization methods Service abstraction (HTTP / TCP API) Message queues
Cloud platform data collection Stdout / Stderr CloudWatch Logs HTTP CloudWatch agent (File System + HTTP) CloudWatch Metrics HTTP API File System Azure Monitor TCP
Kubernetes logs - Node-level logging Stdout / Stderr logs Handled by Kubernetes host
Kubernetes logs - Cluster-level logging Stdout / Stderr logs Pod-level agent Logging Backend
Research Problem There are different use cases for logs Write first, analyze later approach Differences in delivery mechanisms Differences in logs usage Size and velocity of service architecture components defines log collection techniques
Research Direction Construct a service architecture prototype variations Test out different log collection & aggregation approaches Define key aspects of log analysis in service architecture Research how the approach changes depending on volume changes
