Optimizing Log Rotation and Management in Bro Network Security Tool
Discover best practices for rotating and managing logs efficiently in Bro network security tool to prevent CPU spikes, power loss issues, and system reboots. Learn about atomic operations, file compression, and serialization techniques to enhance log handling and ensure data integrity.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Labrotory 2018
Lazeektory 2018
How does bro rotate/archive logs? mv conn.log conn-yaddayadda.log gzip < conn-yaddayadda.log > .../2018/10/10/conn.09:00:00-10:00:00.gz What happens if? Power loss Reboot OOM Something is trying to read .gz files as they are created
Atomic operations Atomic mv (on same filesystem) Not Atomic cp mv (across filesystems) curl -o file.csv http://example.com/file.csv gzip < in > out
Solution? mv (across filesystem) cp src dst.tmp && mv dst.tmp dst && rm src (mv will do cp src dst && rm src) Compress gzip < src > dst.tmp && mv dst.tmp dst && rm src
How does bro rotate/archive logs? gzip < conn-yaddayadda.log > .../conn.09:00:00-10:00:00.gz & gzip < dns-yaddayadda.log > .../dns.09:00:00-10:00:00.gz & gzip < http-yaddayadda.log > .../http.09:00:00-10:00:00.gz & gzip < ssl-yaddayadda.log > .../ssl.09:00:00-10:00:00.gz &
Serialize! Bro side mv conn-yaddayadda.log /bro/logs/log_queue/conn & mv http-yaddayadda.log /bro/logs/log_queue/http & External tool for each file in /bro/logs/log_queue Atomically rotate $file
bro-atomic-rotate https://github.com/ncsa/bro-atomic-rotate
Before: worker2manager events module Mod; global my_event: event(source: addr); redef Cluster::worker2manager_events += /Mod::my_event/; event connection_established(c: connection) { local src = c$id$orig_h; event Mod::my_event(src); #magically sent to manager }
Before: worker2manager events globals conns: table[addr] of count &default=0; @if ( Cluster::local_node_type() == Cluster::MANAGER ) event my_event(src) { if(++conns[src] > 100) NOTICE([...]) } @endif
Before: worker2manager events Worker-1 Worker-2 Manager Worker-3 Worker-n
Before: worker2manager events Worker-1 -> Manager event my_event(10.10.10.1); Worker-2 -> Manager event my_event(10.10.10.2); Worker-3 -> Manager event my_event(10.10.10.1); Worker-4 -> Manager event my_event(10.10.10.2); Worker-1 -> Manager event my_event(10.10.10.3); Worker-2 -> Manager event my_event(10.10.10.4);
Before: worker2manager events Manager sees 6 events and stores: Manager sees 6 events and stores: 10.10.10.1 = 2 10.10.10.2 = 2 10.10.10.3 = 1 10.10.10.4 = 1
After: Cluster::publish_hrw module Mod; global my_event: event(source: addr); event connection_established(c: connection) { local src = c$id$orig_h; Cluster::publish_hrw(Cluster::proxy_pool, src, #hash key Mod::my_event, src); }
After: Cluster::publish_hrw globals conns: table[addr] of count &default=0; event my_event(src) { if(++conns[src] > 100) NOTICE([...]) }
After: Cluster::publish_hrw Worker-1 Proxy-1 Worker-2 Proxy-2 Worker-3 Worker-n
After: Cluster::publish_hrw Worker-1 -> Proxy-1 event my_event(10.10.10.1); Worker-2 -> Proxy-2 event my_event(10.10.10.2); Worker-3 -> Proxy-1 event my_event(10.10.10.1); Worker-4 -> Proxy-2 event my_event(10.10.10.2); Worker-1 -> Proxy-2 event my_event(10.10.10.3); Worker-2 -> Proxy-1 event my_event(10.10.10.4);
After: Cluster::publish_hrw Proxy Proxy- -1 sees 3 events and stores: 1 sees 3 events and stores: 10.10.10.1 = 2 10.10.10.4 = 1 Proxy Proxy- -2 sees 3 events and stores: 2 sees 3 events and stores: 10.10.10.2 = 2 10.10.10.3 = 1
Bro Side @load NCSA/ZeroMQWriter redef LogZeroMQ::endpoint = "tcp://127.0.0.1:9999";
ZMQ Side context = zmq.Context() socket = context.socket(zmq.SUB) socket.bind( tcp://*:9999 ) socket.setsockopt_string(zmq.SUBSCRIBE, * ) while True: stream = socket.recv_string() entry = socket.recv_string() rec = json.loads(entry) print(stream, rec)
