Skip to main content

Open Source System Logging Solutions

For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.


Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.

logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.

Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.

Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.



Comments

  1. I just discovered logstash myself and i'm very excited about it! Would you happen to know whether or a not a pluggin exists for parsing ConfigMgr logs? Else I can always RTM and make one :)

    ReplyDelete
  2. I think you have to write a filter to achieve custom parsing.

    From the logstash point of view, a log consists of timestamp plus data. So if any log data including ConfigMgr logs which has this format may be collected using logstash process. The only thing you have to do is denote the log file path as input. But if ConfigMgr file format is more than this simple approach you have to write your own filter. Simple example for syslog is http://cookbook.logstash.net/recipes/syslog-pri/

    ReplyDelete

Post a Comment

Popular posts from this blog

Sending Jboss Server Logs to Logstash Using Filebeat with Multiline Support

In addition to sending system logs to logstash, it is possible to add a prospector section to the filebeat.yml for jboss server logs. Sometimes jboss server.log has single events made up from several lines of messages. In such cases Filebeat should be configured for a multiline prospector. Filebeat takes lines do not start with a date pattern (look at pattern in the multiline section "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}" and negate section is set to true ) and combines them with the previous line that starts with a date pattern. server.log file excerpt where DatePattern: yyyy-MM-dd-HH and ConversionPattern: %d %-5p [%c] %m%n Logstash filter:

Creating Multiple VLANs over Bonding Interfaces with Proper Routing on a Centos Linux Host

In this post, I am going to explain configuring multiple VLANs on a bond interface. First and foremost, I would like to describe the environment and give details of the infrastructure. The server has 4 Ethernet links to a layer 3 switch with names: enp3s0f0, enp3s0f1, enp4s0f0, enp4s0f1 There are two bond interfaces both configured as active-backup bond0, bond1 enp4s0f0 and enp4s0f1 interfaces are bonded as bond0. Bond0 is for making ssh connections and management only so corresponding switch ports are not configured in trunk mode. enp3s0f0 and enp3s0f1 interfaces are bonded as bond1. Bond1 is for data and corresponding switch ports are configured in trunk mode. Bond0 is the default gateway for the server and has IP address 10.1.10.11 Bond1 has three subinterfaces with VLAN 4, 36, 41. IP addresses are 10.1.3.11, 10.1.35.11, 10.1.40.11 respectively. Proper communication with other servers on the network we should use routing tables. There are three