For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.
Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.
logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.
Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.
Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.
I just discovered logstash myself and i'm very excited about it! Would you happen to know whether or a not a pluggin exists for parsing ConfigMgr logs? Else I can always RTM and make one :)
ReplyDeleteI think you have to write a filter to achieve custom parsing.
ReplyDeleteFrom the logstash point of view, a log consists of timestamp plus data. So if any log data including ConfigMgr logs which has this format may be collected using logstash process. The only thing you have to do is denote the log file path as input. But if ConfigMgr file format is more than this simple approach you have to write your own filter. Simple example for syslog is http://cookbook.logstash.net/recipes/syslog-pri/