HP iLo can’t send to syslog | Scott's Ramblings

iLo 2 and 3 don’t have the ability to send events to a syslog server by default. iLo 4 on the other hand, finally has the option to do it… But what about all my gen 7 servers? I needed a way to send all the event logs in to Splunk to help catch and monitor for hardware issues.

There are two logs kept in the iLo. First the iLo event log tracks events on the iLo such as login, NTP synchronization… Basically all things on the iLo. The IML log contains all server based events. POST errors, hardware issues, etc, go here. I want to capture both those logs. But, if I can’t get the iLo to send logs to Splunk, I need Splunk to pull the logs from the iLo.

Luckily, the iLo has an SSH interface. You can view the logs in the SSH interface by using the show command against the correct object: /map1/log1 for the IML log or /system1/log1 for the iLo log.

We need to get SSH ready for this. We’re going to use SSH keys. Instead of going through all the steps here, just follow the “Importing SSH keys from PuTTY” section of http://www.hp.com/ctg/Manual/c02237707.pdf . You can also look at any number of linux guides on how to create SSH keys if you want.

Now, that we can SSH to the iLo, we need a script to scrape the logs. There are a few problems we need to solve with this.

First, when I read the logs on the iLo all the logs that are in the iLo history are displayed. We can’t just send all the entries into Splunk as we’ll be recording the same, already indexed events, over and over again. I solve this by recording all events into a log file and then before I connect to the iLo I read and hash all the current events. As I read events from the iLo, if I see the entry already there I skip over it. Since events have a date stamp and an index number, they are unique and the hash method works well. Any new events get appended to the existing log file. One gotcha… If we ever want to rotate the log file on the server, we’ll again get duplicate logs since we won’t see them as duplicates. The trick here is to rotate the log file and then run a post rotate script that flushes the log on the iLo. One other note: You might think that we can use datestamps to recognize new entries but that actually won’t work. Often times the iLo doesn’t have the correct date UNTIL after NTP syncs up. As a result we may miss events.

Second, we have to figure out who we want to get the log entries into Splunk. We have two choices. First, we can run this via cron and have Splunk monitor the log file. Or we can have Splunk run the script periodically and output the new events to STDOUT. I chose to have Splunk do it all. So, my monitor lines look like:

[script://$SPLUNK_HOME/etc/apps/search/bin/get_ilo_log.pl]
disabled = 0
host = ilo
interval = 900
sourcetype = ilo_event_log

[script://$SPLUNK_HOME/etc/apps/search/bin/get_ilo_iml_log.pl]
disabled = 0
host = ilo
interval = 900
sourcetype = ilo_iml_log

I’m giving each log its own sourcetype. This way I can easily filter out where the event comes from. Since this runs on the Splunk indexer (at least in my case), I hard set the host to be the iLo, again making it easier to search with.

So, now what to do with the data? The most obvious thing is an alert for hardware issues:

( sourcetype=ilo_event_log OR sourcetype=get_iml_log ) NOT logout NOT login NOT "iLO clock has been synchronized"

I needed to exclude the SSH logins and normal log messages.

So, here is my script (its the same for the IML version):

#!/usr/bin/perl

# Read in the existing iLo log file and put the events into a hash
my $ilolog = {};
open( FILE, "/var/log/ilo.log" ) or die "Can't open ilo log for reading: $!\n";
while ( my $line =  ) {
    chomp( $line );
    $ilolog->{$line} = 1;
}
close( FILE );

# Grab all the events into an array
open( FILE, "ssh Administrator\@ilo show -a /map1/log1 |" ) or
    die "Can't ssh to ilo\n";
my @file = ;
close( FILE );

# Start appending to the iLo log
open( LOGFILE, ">>/var/log/ilo.log" ) or die "Can't open ilo log for writing: $!\n";
while( scalar( @file ) > 0 ) {
    my $line = shift( @file );
    # Skip any non log event lines
    next unless ( $line =~ /=/ );
    next if ( $line =~ /Stat/ );

    # Events are on multiple lines, so when we see the start of one pull in the
    # rest of the info
    if ( $line =~ /number/ ) {
        my $info = get_info( shift( @file ), shift( @file ), shift( @file ), shift( @file ) );
        my $output = "$info->{date} $info->{time} severity=\"$info->{severity}\" description=\"$info->{description}\"";
        # Check for matching line already in log file and if not there write out the log file
        if ( $ilolog->{$output} != 1 ) {
            print LOGFILE "$output\n";
            print "$output\n";
        }
    }
}

sub get_info {
    my $hash = {};
    foreach my $line ( @_ ) {
        $line =~ tr/\r/\n/;
        $line =~ /\s+(.*)=(.*)/;
        my $key=$1; my $value = $2;
        $hash->{$key} = $value; 
    }
    return $hash;
}