Normally my posts are about the things I’ve done at home on personal projects, but here is one I did for a client recently.

Syslog is a protocol used by network gear, appliances, and most Unix distributions to handle their logs. Most importantly (for this discussion) the protocol is used by these devices to send their logs to a central server. The client is a very large organization and was sending data from 100’s of devices configured with high verbosity so that they can see any security events more clearly. They also had a lot of filters configured in Syslog-ng so that they can sort the events so their SEIM can consume them properly. Syslog-ng couldn’t keep up with all the events coming in.

They used Splunk as their SEIM. Using Splunk we collected netstat -s periodically from all servers and found that they were dropping nearly 1/3 of the UDP traffic coming in. Syslog default to using UDP (faster and less intensive for both client and server) on port 514 to send data in. Since syslog was the only UDP traffic expected this means that they were dropping 1/3 of all their events.

We needed to figure out where on the central server packets were dropping… At or below the kernel, or somewhere in Syslog-ng. We found that Syslog-ng was constantly at 100% CPU utilization which made it clear that Syslog-ng was choking. Since the version they were using was single threaded, Syslog-ng couldn’t go any faster at processing events since it was at 100% and was dropping packets. We tried optimize some lower level kernel tunables such as NIC ring buffer size and UDP memory but that didn’t help at all. Those tests did, however, cement our belief that Syslog-ng was the culprit.

At the end of this exercise we were able to get Syslog-ng to not only consume all the traffic it was sent but also have room to spare. In fact were were able to get it so that Splunk can’t consume traffic fast enough! But that is another story for another time. Below are the tricks we tried and the tricks we learned to optimize Syslog-ng.

Reduce what Syslog-ng has to do in order to process a packet.

The less work it has to do per event, the more events it can consume. The client had thousands of filters. We were able to reduce the filter count by 40% which gave Syslog-ng some room for growth.

Reduce what Syslog-ng has to do in order to process a packet. Part 2

Found an interesting trick in the syslog-ng documentation that really helps with this… If you have multiple log blocks with the same source then normally syslog-ng will process each log block’s filter rules in order to see if the packet should be written more than once. An example being that maybe you want sshd logs in your secure log file but also in your daemon log file. However, that may not be what you want. You can significantly reduce the amount of filters syslog-ng has to check by using the “final” flag in the log block. This will tell syslog-ng that if there is a match do not process the packet against any other log blocks. So, in my case where we had thousands of checks for a given packet this reduced the amount of checking by an incredible factor.


log first_output { source( udp_514 ); filter( filter_a ); destination( my_log_file ); flags( final ); };


Use the multithreaded version of Syslog-ng.

Older versions of syslog-ng were single threaded which means that it can only handle 1 event at a time. If events queued up too quickly syslog-ng wouldn’t take them off the queue and the server’s kernel would drop them. But the multithreaded version gives you the ability to process events simultaneously.

Separate data onto different ports where possible.

Syslog-Ng’s multithreading isn’t straightforward. If you send events in using TCP, then each event can get its own thread. However UDP events are given one thread per source port. If you have most of your traffic coming in on the same port via UDP, you will still only be able to handle one event at a time. But if you put your VMware logs on one port and Cisco on another port you’ll spread the load to separate threads and get better performance. Even if the bulk of the logs come in to one port, this will insulate the events coming in on different ports from getting stuck.

Load balance your traffic

If you can’t break up your traffic, or use TCP, then you may need to implement a load balancer into the mix. For the client, we chose to use iptables. However, you can also leverage a third party appliance to do it for you.

First, set up four sources in syslog-ng listening to different ports on the localhost interface. Configure them to write to separate files so that you don’t hit queue issues writing to disk. Don’t use the port you are trying to load balancer. It’ll keep the firewall rules simpler if you don’t have the same port on different interfaces doing different things. Don’t forget the various destinations and log entries you need.


source udp_515 { network( transport( "udp" ) ip( 127.0.0.1 ) port(515) ); };
source udp_516 { network( transport( "udp" ) ip( 127.0.0.1 ) port(516) ); };
source udp_517 { network( transport( "udp" ) ip( 127.0.0.1 ) port(517) ); };
source udp_518 { network( transport( "udp" ) ip( 127.0.0.1 ) port(518) ); };


Second, Configure iptables to load balance. This is the important part. Because syslog packets may be broken up (they aren’t supposed to according to the spec) I decided to do it in a way that allows all traffic from a host to always go to the same port. This way any broken up packets go to the same file nicely. If you notice the source, the net mask is a bit backwards. Basically that says that if the last 2 bit of the ip address are 0 go to port 518, 1 to port 515 and so on.


iptables -A PREROUTING -s 0.0.0.0/0.0.0.3 -i bond0 -p udp -m udp --dport 514 -j DNAT --to-destination 127.0.0.1:518
iptables -A PREROUTING -s 0.0.0.1/0.0.0.3 -i bond0 -p udp -m udp --dport 514 -j DNAT --to-destination 127.0.0.1:515
iptables -A PREROUTING -s 0.0.0.2/0.0.0.3 -i bond0 -p udp -m udp --dport 514 -j DNAT --to-destination 127.0.0.1:516
iptables -A PREROUTING -s 0.0.0.3/0.0.0.3 -i bond0 -p udp -m udp --dport 514 -j DNAT --to-destination 127.0.0.1:517


Third, update sysctl settings to enable the forwarding of packages. We need to allow forwarding packets to localhost by setting


echo 1 > /proc/sys/net/core/all/route_localnet


This gives us four threads processing incoming events and filtering them into the proper destinations. Because our servers have 12 cores (24 with hyper threading) each this allows us to spread the load to multiple cores and gives us plenty of room to spare.