Passive Tests with the External Command File

Apart from active service and host checks, Nagios also makes use of passive tests (and combinations of both types of test). While the system itself defines the time for active checks when they are performed, and then initiates them, Nagios in passive mode only processes incoming results.

For this to work, an interface is required that allows test results from the outside to be passed on to Nagios, as well as commands that perform checks and feed in the results through the interface. Normally remote hosts send their test results, determined by shell scripts, via the Nagios Service Check Acceptor (NSCA), which is introduced in the next chapter (page 247), to the Nagios server.

Passive checks are used in particular with distributed monitoring, in which noncentral Nagios servers send all their results to a central Nagios instance. This subject is discussed in Chapter 15. Another field in which they are used is in the processing

239

13 Passive Tests with the External Command File

of asynchronous events, the time of which Nagios cannot define itself. One example of this is a backup script that sends a result to Nagios (OK or CRITICAL) when it has completed a data backup, and another example is processing SNMP traps (see Section 14.6).

13.1 The Interface for External Commands

The interface for external commands, known in Nagios jargon as External Command Files, consists of a named pipe (FIFO)1 in the subdirectory rw of the Nagios var directory:

user@linux:˜$ ls -lF /var/nagios/rw

The pipe, marked in the ls output with p, correctly sets up the make installcommandmode command during installation. For reasons of security it is essential that you ensure that only the group nagcmd can read from and write to the pipe. Anyone who has access here can control Nagios remotely via commands, and can, if they want, shut it down entirely.

Commands that Nagios accepts from the External Command File have the following form:

[timestamp] command;arguments

As the timestamp in square brackets, Nagios expects the current time in epoch seconds, that is the number of seconds which have elapsed in the UTC time zone since January 1, 1970. This is followed by a space, then a command followed by a matching number of arguments, separated by a semicolon.

The interface makes extensive use of this mechanism, allowing its users to make various settings via mouse click.2 In this chapter we will limit ourselves to the two processing commands with which computers deliver the results of passive checks to the Nagios server, PROCESS_SERVICE_CHECK_RESULT and PROCESS_HOST_ CHECK_RESULT.

For reasons of security, the processing of external commands must be explicitly switched on in the main configuration file nagios.cfg with the directive check_ external_commands=1:

1A named pipe is a buffer to which a process can write something, which can then be read by another process. Whatever is written first is also read first: First In, First Out (FIFO). Since this involves space in the main memory, a named pipe does not need any space on the hard drive.

2A detailed description of all possible commands is provided by the online documentation at http://localhost/nagios/docs/extcommands.html or file:/usr/local/nagios/share/ docs/extcommands.html.

240

13.2 Passive Service Checks

# /etc/nagios/nagios.cfg

...

check_external_commands=1 command_check_interval=-1 command_file=/var/nagios/rw/nagios.cmd

...

The command_check_interval determines that Nagios checks the interface for existing commands every so many seconds. -1 means “as often as possible.” command_file specifies the path to the named pipe.

13.2 Passive Service Checks

In order for Nagios to be able to accept passive service checks via the interface, this must be explicitly allowed in the global configuration and in the corresponding service definition. The corresponding entry in nagios.cfg is

# /etc/nagios/nagios.cfg

...

accept_passive_service_checks=1

...

In the service definition you can select whether you want to perform active checks in parallel to the passive ones. Active checks are only possible, of course, if Nagios can query the information itself. The following example allows passive checks and stops all active ones:

An exception is normally made for freshness checks (see Section 13.4 from page 243)—here Nagios makes use of the command defined in check_command. To ban active checks entirely, the check_period parameter is set to none. The check command does not play a role in this case, so you can just enter a dummy check here, for example (which like all other commands has to be defined, of course).

On the computer to be tested passively (in this example, linux01) you must ensure, via NSCA (see Chapter 14), that it contacts the Nagios server through the interface

241

13 Passive Tests with the External Command File

for external commands. There it writes the command for passive service checks in the following one-line form:

[timestamp] PROCESS_SERVICE_CHECK_RESULT;host-name;service; return value;plugin output

The timestamp can be created in a shell script, for example with date:

user@linux:˜$ date +%s 1112435763

A simple script that passes on the result of a passive service check on the Nagios server itself to the Nagios installed there, could look like this:

#!/bin/bash

EXTCMDFILE="/var/nagios/rw/nagios.cmd" TIME=‘date +%s‘

HOST=$1

SRV=$2 RESULT=$3 OUTPUT=$4

CMD="[$TIME] PROCESS_SERVICE_CHECK_RESULT;$HOST;$SRV;$RESULT;$OUTPUT"

/bin/echo $CMD >> $EXTCMDFILE

When it is run it expects the parameters in the correct sequence:

name_of_script linux01 Disks 0 ’Disks ok: everything in order :-)’

After the host and service names, the test status follows as a digit, and finally the output text. If the service name contains spaces, then it should also be set in quotation marks.

13.3 Passive Host Checks

Passive host checks follow the same principle as passive service checks, except that they involve computers and not services. To allow them globally, the accept_passive_host_checks parameter is set in nagios.cfg to 1:

# /etc/nagios/nagios.cfg

...

accept_passive_host_checks=1

...

242

13.4 Reacting to Out-of-Date Information of Passive Checks

In addition, the host definition for the computer to be monitored passively must allow this kind of host check:

In this example it simultaneously forbids active checks.

The command to be sent through the external interface with which the computer delivers its test results differs here only marginally from the syntax used in the service check command already introduced:

[timestamp] PROCESS_HOST_CHECK_RESULT;hostname;return value; plugin output

Active and passive host checks differ in one important respect: with passive checks, Nagios is no longer in a position to distinguish between DOWN and UNREACHABLE (see Section 4.1 from page 72). If you still want to take account of network topology dependencies when making notifications and to give specific information on the actual host that is down, you must make use of host dependencies in this case (see Section 12.6.2 from page 238).

13.4Reacting to Out-of-Date Information of Passive Checks

It lies in the nature of passive checks that Nagios is content with the information delivered. Nagios has no influence over when and at what intervals the remote host delivers them. It may even be the case that the information does not arrive at all.

In order to classify the “knowledge state” of the server as out of date, Nagios has the ability to become active itself, with a freshness check. Like passive checks, freshness checking must be enabled both globally and in the relevant serviceable host object. To do this, you need to set the following global parameters in the file nagios.cfg:

# /etc/nagios/nagios.cfg

...

check_service_freshness=1

243

13 Passive Tests with the External Command File

service_freshness_check_interval=60 check_host_freshness=0 host_freshness_check_interval=60

...

The value 0 in check_host_freshness and the value 1 in check_service_freshness ensure that Nagios carries out freshness checks only for services, and not for hosts. The check interval defines the intervals at which the server updates its information, in this case, every 60 seconds. When Nagios really becomes active in the case of a specific service or host depends on the threshold value, which you can set in the appropriate service or host definition with the freshness_threshold parameter:3

So in this example Nagios performs the freshness check for this service only if the last transmitted value is older than 3600 seconds (one hour). Then Nagios starts the command defined in check_command, even if active checks have been switched off in the corresponding host or service definition, or even globally.

If you define the command named here in the example, service_is_stale, so that Nagios really does check the service or host, then Nagios will perform active tests even if active checking is switched off, but always only if passive results are overdue for longer than the threshold value set.

If active checks are not possible or not wanted, you can ensure, using a pseudotest, that Nagios will explicitly signal an error status, so that the administrator’s attention is drawn to it. Otherwise Nagios will always display the last status to be received. If this was OK, then it will not necessarily be noticed that current results have not been arriving for some time. The following pseudo-test script delivers an appropriate error message with echo, and with exit 2 delivers the return value for CRITICAL, so that the administrator can react accordingly:

#!/bin/bash

/bin/echo "CRITICAL: no current results of the service" exit 2

3If you do not explicitly specify freshness_threshold, the value set for normal_check_interval will be used in the hard state, and if there is a soft state, the value retry_check_interval will serve as the default.

244

13.4 Reacting to Out-of-Date Information of Passive Checks

If you start the script from the plugin directory as service_is_stale.sh, the Nagios command service_is_stale will be defined as follows:

define command{

command_name service_is_stale command_line $USER1$/service_is_stale.sh

}

If the results for the service Disks on linux01 fail to appear for longer than one hour, Nagios will run the script service_is_stale.sh, which always returns CRITICAL, irrespective of what data linux01 last sent. This CRITICAL status is only ended when the host passes on new and more positive results to the server through a passive check.

245

The Nagios Service Check

Acceptor (NSCA)

In order to send service and host checks across the network to the central Nagios server, a transmission mechanism is required. This is provided by the Nagios Service Check Acceptor (NSCA). It consists of two components: a client program send_nsca, which accepts the results of a service or host check on the remote host and sends them to the Nagios server, and the NSCA daemon nsca, which runs on the server, receives data from the client, processes this for the External Command File interface (see Section 13.1), and passes this data on to it (Figure 14.1).

The Nagios Service Check Acceptor was originally developed to enable distributed monitoring in which decentralized Nagios servers can send their results to a central Nagios server (see Chapter 15 from page 265). In principle, the data that send_nsca sends to the Nagios server can come from any applications you like.

247

14 The Nagios Service Check Acceptor (NSCA)

Sending commands across the network to the central Nagios instance is not insignificant, from a security point of view, since Nagios could be completely switched off using the External Command File. This is why NSCA sends the data in encrypted form, and clients must have the correct key to obtain access to the interface. This prevents an arbitrary network participant from being able to run any commands at all on the Nagios server.

Figure 14.1:

How the NSCA

functions

14.1 Installation

NSCA version 2.4, current at the time of going to press, was published in the summer of 2003; the chances are therefore quite high that the distribution you are using contains a current package. The source code1 is quite easy to compile yourself, however. As a prerequisite, you need to have the library libmcrypt installed, together with the relevant header files,2 or else the integrated encryption cannot be used.

In the unpacked source directory, you should run the included configure script, specifying the Nagios configuration and var directories:

linux:local/src # tar xvzf /path/to/nsca-2.4.tar.gz

...

linux:local/src # cd nsca-2.4

linux:src/nsca-2.4 # ./configure --sysconfdir=/etc/nagios \

--localstatedir=/var/nagios

...

*** Configuration summary for nsca 2.4 07-23-2003 ***:

248

14.2 Configuring the Nagios Server

General Options:

-------------------------

NSCA port: 5667 NSCA user: nagios NSCA group: nagios

...

At the end it displays output, showing the permissions with which the NSCA user starts by default, if not otherwise specified in the configuration. Normally the NSCA daemon waits on TCP port 5667.

A final make all compiles the two programs nsca and send_nsca. They are now located in the subdirectory src and need to be copied manually to a suitable directory:

linux:src/nsca-2.4 # cp src/nsca /usr/local/sbin/.

nsca is copied to the Nagios server, preferably to the directory /usr/local/sbin. send_nsca belongs on the remote host that is to send its test results to the Nagios server. If this computer has a different operating system version or platform, it is possible that the client to run there will need to be recompiled.

Both programs each require their own configuration file, which is best stored in the directory /etc/nagios:

linux:src/nsca-2.4 # cp nsca.cfg /etc/nagios/. linux:src/nsca-2.4 # scp send_nsca.cfg remote_host:/etc/nagios/.

14.2 Configuring the Nagios Server

14.2.1The configuration file nsca.cfg

For NSCA to work, the External Command File interface on the Nagios server must be activated in the configuration file /etc/nagios/nagios.cfg (Section 13.1, page 240) and the corresponding data entered in the NSCA configuration file nsca.cfg:

# /etc/nagios/nsca.cfg server_port=5667 server_address=192.168.1.1 allowed_hosts=127.0.0.1 nsca_user=nagios nsca_group=nagios

debug=0 command_file=/var/nagios/rw/nagios.cmd

249

14 The Nagios Service Check Acceptor (NSCA)

alternate_dump_file=/var/nagios/rw/nsca.dump aggregate_writes=0

append_to_file=0 max_packet_age=30 password=verysecret decryption_method=10

The parameters server_port, server_address, allowed_hosts, nsca_user, and nsca_ group take effect only if nsca is started as a daemon. If it is started as an inet daemon, the values set in its configuration apply to the NSCA server address and the port on which the NSCA is listening, the IP addresses of the hosts that are allowed to access the interface,3 and the users and group with whose permissions the Service Check Acceptor runs.

The debug parameter makes it easier to search for errors, but it should normally be switched off (value 0). If it is set to 1, NSCA writes debugging information in the syslog.

The named pipe is defined by the entry command_file. If you specify an alternative output file, with alternate_dump_file, this serves as a fallback in case the named pipe given does not exist. Before version 2.0, Nagios removed the pipe each time it was shut down, but this should not happen anymore.

If it is set to 1, aggregate_writes ensures that NSCA collects all the incoming commands just once and then passes these on to the interface as a block. If the value at this position is 0, then NSCA sends on each incoming command immediately to the External Command File.

append_to_file can have the values 0 (opens the External Command File in write mode) or 1 (opens it in the append mode), and it should always be set to 0.4

Client messages older than max_packet_age seconds are discarded by NSCA, to avoid replay attacks. This value may not be larger than 900 seconds (15 minutes) and should be as small as possible.

The last two parameters refer to the encryption of the communication. password contains the actual key, which is identical for clients, and which must be entered in the configuration for the clients (cf. Section 14.3 on page 252). Because the key is written in the file in plain text, nsca.cfg should be readable only for the user with whose permissions the NSCA is running, which in our case is nagios:

linux:/etc/nagios # chown nagios.nagios nsca.cfg linux:/etc/nagios # chmod 400 nsca.cfg

3If you want to define more than one IP address for allowed_hosts, they are separated by a comma.

4The append mode only makes sense if the External Command File is replaced for debugging purposes with a simple file.

250

14.2 Configuring the Nagios Server

Finally, decryption_method defines the encryption algorithm. The default is 1 (XOR), which is almost as insecure as 0 (no encryption). 10 stands for LOKI97, which is regarded as secure.5 The list of all possible algorithms is contained in the supplied configuration file, which contains many old algorithms and some newer ones, such as DES (2), Triple-DES (3), Blowfish (8), and Rijndael (AES).6

14.2.2 Configurung the inet daemon

If you want to start nsca with the inet daemon, the following entry is added in the file /etc/services:

nsca 5667/tcp # Nagios Service Check Acceptor (NSCA)

xinetd configuration

If the newer xinetd is used, the file nagios-nsca is created in the directory /etc/ xinetd.d with the following contents:

#/etc/xinetd.d/nrpe

#description: NRPE

#default: on service nrpe

{

}

The values printed in bold type for the user and group with whose permissions the NSCA should run, and the path to the NSCA daemon nsca (parameter server) and the corresponding configuration file, are adjusted if necessary to your own environment. The line only_from, as an equivalent to the nsca.cfg parameter allowed_hosts, takes in all the IP addresses, separated by spaces, from which the NSCA may be addressed. Distributions that include NSCA as a finished package and install xinetd by default, include a ready-to-use xinetd configuration file, where you only need to adjust this last parameter.

251

14 The Nagios Service Check Acceptor (NSCA)

In order for the new configuration to become effective, the xinetd init script is run with the reload argument:

linux:˜ # /etc/init.d/xinetd reload

inetd configuration

If the standard inetd command is run, the following line is added (line-wrapped for the printed version) in the configuration file /etc/inetd.conf:

nsca stream tcp nowait nagios /usr/sbin/tcpd

/usr/local/sbin/nsca -c /etc/nagios/nsca.cfg --inetd

If you want to leave out the TCP wrapper tcpd, you just omit the string /usr/sbin/ tcpd. In this case you must also explicitly specify the user (nagios) with whose permissions the NSCA starts, the complete path to the binary nsca, and the configuration file with its absolute path. So that the Internet daemon can take account of the modification, its configuration must be reloaded:

linux:˜ # /etc/init.d/inetd reload

14.3 Client-side Configuration

The configuration file send_nsca.cfg on the client side must contain the same encryption parameters as the file on the Nagios server:

password=verysecret decryption_method=10

Since the key is also written here in plain text, it should not be readable for just any user. For this reason it is best to create a user nagios and a group nagios on the client side:

linux:˜ # groupadd -g 9000 nagios

linux:˜ # useradd -u 9000 -g nagios -d /usr/local/nagios \

-c "Nagios Admin" nagios

You should now protect the file send_nsca.cfg so that only the user nagios can read it, and ensure, using the SUID mechanism, that the program send_nsca always runs under the user ID of this user. If you now grant execute permission to the group nagios, only its members may execute the NSCA client program:

252

14.4 Sending Test Results to the Server

linux:˜ # chown nagios.nagios /etc/nagios/send_nsca.cfg linux:˜ # chown nagios.nagios /usr/bin/send_nsca linux:˜ # chmod 400 /etc/nagios/send_nsca.cfg

linux:˜ # chmod 4710 /usr/bin/send_nsca linux:˜ # ls -l /usr/bin/send_nsca

-rws--x--- 1 nagios nagios 83187 Apr 2 17:56 /usr/local/bin/send_nsca

14.4 Sending Test Results to the Server

The client program send_nsca reads the details of a host or service check from the standard input, which the administrator must format as follows:7

host-name\tservice\treturn value\toutput host-name\treturn value\toutput

send_nsca sends this to the Nagios server. The first line describes the format for service checks and the second line, that for host checks. The placeholder return value is replaced by the status determined, that is, 0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 for UNKNOWN. By output, a one-line text is meant, of the type that plugins provide as a support for the administrator. As the separator, a tabu sign is used (\t).

In order to make a complete command from this that can be understood by the external command, the NSCA daemon first prefixes the timestamp and the matching command (PROCESS_SERVICE_CHECK_RESULT or PROCESS_HOST_CHECK_RESULT). This is why only these two commands can be sent using NSCA.

send_nsca itself has the following options:

-H address

This is the host name or IP address of the Nagios server to be addressed by NSCA.

-d delimiter

This is the delimiter for the input; the default is a tab sign. The following example page uses the semicolon as a delimiter.

-c path/to the/configuration file

This parameter specifies the path to the configuration file send_nsca.cfg. Since no path has been compiled into the client, send_nsca expects by default to find the file in the current directory. For this reason it makes sense to specify the absolute path with this option.

7Normally you have to ensure that test scripts you have written yourself produce the correct output; if you use Nagios plugins, you must reformat their output accordingly. Since the latter can be run much better directly with NRPE, this should be the exception to the rule.

253

14 The Nagios Service Check Acceptor (NSCA)

-p port

This defines an alternative port if the default, the TCP port 5667, is not used.

-to timeout

After timeout seconds (by default, 10) send_nsca aborts the connection attempt to the NSCA daemon, if no connection is established.

With simple test scripts such as the following one, the functionality of the NSCA can be tested. A service is chosen as the test object, which is in a state other than UNKNOWN (e.g., OK), in this case, nmbd on the host linux01:

#!/bin/bash CFG="/etc/nagios/send_nsca.cfg"

CMD="linux01;nmbd;3;UNKNOWN - just one NSCA test"

/bin/echo $CMD | /usr/local/bin/send_nsca -H nagios -d ’;’ -c $CFG

The script puts it, from Nagios’s point of view, into the UNKNOWN status. After it is run, you should discover if the transfer was successful:

nagios@linux:˜$ bash ./test_nsca

1 data packet(s) sent to host successfully.

As soon as Nagios processes the command and you have reloaded the page in your browser, the Web interface displays the UNKNOWN status for the selected service. With the next active check, the previous status will be recovered.

Because it is so simple to send Nagios check results with send_nsca, it is essential that you protect the NSCA from misuse, as already demonstrated. On the client, you should restrict access to the client program send_nsca and to its configuration file and you should make sure that you have secure encryption, and on the server explicitly define the sender and IP addresses that are to be allowed.

14.5Application Example I: Integrating syslog and Nagios

Linux and Unix systems as a rule log system-relevant events through syslog. Sooner or later you will probably want Nagios to also inform the administrator of important syslog events. To do this, you require passive service checks, NSCA for transmitting the results to the Nagios server, and a method of filtering individual block entries.

If you are using syslog-ng8 instead of the standard BSD syslog, you can make use of its ability to set filters and to format the output using templates. The use of

8 The “ng” stands here for next generation.

254

14.5 Application Example I: Integrating syslog and Nagios

NSCA compensates for the fact that the program cannot itself transmit data in encrypted form.

This connection to Nagios is supplemented by programs to evaluate log files, such as logcheck,9 which is contained in almost every Linux distribution, but it does not replace them. This is because Nagios can send individual e-mails for each event, but not for a summary of events, as logcheck does (usually once per hour). In addition to this, the Web interface always displays the last event in each case.

14.5.1 Preparing syslog-ng for use with Nagios

Apart from the source code, the syslog-ng homepage10 also provides a detailed manual, which is why we shall only discuss the basic principle at this point. The software differentiates between the source, filter, and destination. All three objects can be combined in any form; they are defined in the configuration file

/etc/syslog-ng/syslog-ng.conf:

# /etc/syslog-ng/syslog-ng.conf source local {

unix-stream("/dev/log"); internal();

file("/proc/kmsg" log_prefix("kernel: "));

};

destination console_10 { file("/dev/tty10");

};

filter f_messages {

not facility(auth, authpriv) and level(info .. alert);

};

log { source(local);

filter(f_messages); destination(console_10);

};

This example defines three sources at the same time: unix-stream reads from the socket /dev/log, through which most programs send their messages to the syslog. internal is the name of the source syslog-ng feeds with internal messages, and from the file /proc/kmsg syslog receives kernel messages. These are given the kernel: prefix, so that they can be be distinguished from normal log entries.

9 http://sourceforge.net/projects/logcheck/

10 http://www.balabit.com/products/syslog_ng/

255

14 The Nagios Service Check Acceptor (NSCA)

The destination definition ensures that all syslog output appears on the console tty10 (this can be displayed with Alt - F10 ).

filter defines what messages should reach this destination, if any. In the case of the f_messages filter, this is all messages matching the category (the level) info and that syslog does not provide with the stamp (the facility; see man syslog.conf and man 3 syslog) auth or authpriv. Alternatively syslog-ng filters according to a search pattern, with the instruction match(”pattern”), according to the program doing the logging (program(”program name”)) and according to the source host (host(”hostname”)).

Finally the keyword log links the source, filter, and destination. Multiple specifications are possible here, so several sources and destinations can be specified in a single statement:

log {

source1); source2; ...

filter1; filter2; ...

destination1; destination2; ...

}

To integrate this into Nagios, use is made of the option of defining a program as a target, which is called for every event:

destination d_nagios_warn { program("/usr/local/nagios/misc/send_syslog.sh" template("$HOST;syslog-ng;1;WARNING: $MSG\n") template_escape(no));

};

destination d_nagios_crit { program("/usr/local/nagios/misc/send_syslog.sh" template("$HOST;syslog-ng;2;CRITICAL: $MSG\n") template_escape(no));

};

The template directive formats the output so that it is suitable for send_nsca, using a semicolon as the delimiter: host and service names (syslog-ng) are followed by the state (1 = WARNING; 2 = CRITICAL), and then the actual output text is given. Apart from $HOST and $MSG, syslog-ng has a series of further macros, which are described individually in the documentation on the homepage. The parameter template_escape protects quotation marks in the text and is intended principally for SQL commands, so in this case it can be set to no.

The following script send_syslog.sh uses the bash function read to read from the standard input line by line, and for each line read it calls up send_nsca, which sends on the data—as described in this chapter—as a passive test result to Nagios:

256

14.5 Application Example I: Integrating syslog and Nagios

#!/bin/bash

while read -r line; do

echo $line | /usr/bin/send_nsca -H nagsrv -d ’;’ \ -c /etc/nagios/send_nsca.cfg \

1>/usr/local/nagios/var/send_syslog.log 2>&1

done

Because a semicolon is used as a delimiter, we specify this explicitly with the option -d. The status report that each send_nsca command displays on the standard output is diverted by the script into a separate log file (/usr/local/nagios/var/send_syslog.log).

Thanks to the program instruction in the syslog configuration, syslog-ng starts the script automatically. This is also the reason that the send_nsca command is in an endless loop: this means that syslog-ng does not run an external program every time there is a relevant event.

14.5.2 Nagios configuration: volatile services

In Nagios slang, “volatile” refers to services that show an error state only once. This refers to devices, for example, that automatically reset the state when an error is queried—which means that the error cannot be reproduced. The same applies for syslog entries: if a check following an error state returns an error, this will always be a second event. So we don’t have a continuing error state here, but a problem that has again occurred.

For continuing error states, Nagios normally does not send any further messages for the time being. With the is_volatile parameter, however, it treats every error as if it had just occurred. Nagios logs the state, sends a notification, and implements the event handler—provided it is defined—(see Appendix B from page 409).

For syslog-ng, this means that each entry is seen as an independent event. In order that Nagios sees things in this way as well, the corresponding service definition contains the is_volatile parameter:

257

14 The Nagios Service Check Acceptor (NSCA)

notification_options w,c,u notification_interval 480 notification_period 24x7

}

Since the Nagios server should not test anything on its own, active_checks_enabled 0 switches off active service checks. However, freshness checking (see Section 13.4 from page 243) can always cause Nagios to perform active tests. To prevent this, we set the check_freshness parameter in this case explicitly to 0.

This service definition does not really require the parameters check_command and check_period, but since these are mandatory parameters, they must still be specified: as check_command, the plugin check_dummy (see Section 7.13 on page 154) is used.

It is also important that max_check_attempts is set to 1, so that a transmitted error state immediately triggers a hard state. With a value larger than 1, Nagios would wait for further error results here before categorizing the problem state as a hard state.

The notification_options parameter ensures that the system informs the specified contact group of all error states (WARNING, CRITICAL, and UNKNOWN). The notification_interval, which defines the interval between two notifications for a continuing error state, is actually superfluous, since Nagios, thanks to is_volatile 1, provides notification of every event immediately, irrespective of what the previous state looked like. But since it is a mandatory parameter, notification_interval still has to be specified.

14.5.3 Resetting error states manually

Events that are taken into account by the syslog filter always inform you of only one current state, which is why the syslog service in Nagios never displays an OK state on its own (Figure 14.2). This problem can be solved with the Web interface, which allows a passive check result to be generated manually.

Figure 14.2: The syslog-ng service in an error state

If you click on the service name in Figure 14.2, the extended status information will be shown (Figure 14.3). There you will find the entry Submit passive check result for this service, with which a test result can be sent manually (Figure 14.4). In this way the syslog-ng service can be reset to its normal state. Since the Web

258

14.5 Application Example I: Integrating syslog and Nagios

interface always shows only the most recent error state, but not individual error messages, you must look through the e-mail messages to see whether other errors have occurred apart from those errors displayed by Nagios in the Web interface.

Figure 14.3:

The arrow points to the possibility of “generating” a passive test result for the syslog-ng service

You can also define your own service for each syslog event, of course. This may sometimes be quite time-consuming, but it does allow you to separate various messages and their processing states in the Web interface. If the filter in syslog-ng is restricted so that a syslog service object always refers to just one resource to be monitored, you can also leave out the is_volatile parameter.

Figure 14.4: Creating a passive check result syslog-ng

259

14The Nagios Service Check Acceptor (NSCA)

14.6Application Example II: Processing SNMP Traps

Asynchronous messages that are sent by an SNMP agent (see Section 11.1 from page 178) to a central management unit, called traps in SNMP jargon, can be processed by Nagios in a way similar to the Nagios Service Check Acceptor (NSCA). In addition, it allows SNMP traps to be accepted on a host other than the Nagios server itself.

Processing SNMP traps with Nagios is particularly worthwhile if the system monitors the network almost completely, and only a few devices or services restrict their communication just to SNMP and SNMP traps. Nagios, or the Open Source tool OpenNMS,11 are no substitutes for real commercial SNMP management systems.

In many cases, SNMP traps are vendor-specific, so that you cannot avoid getting to grips with the appropriate documentation and the vendor-specific MIB (Management Information Base; see Section 11.1.1 from page 179).

14.6.1 Receiving traps with snmptrapd

In order to receive SNMP traps, you require a special Unix/Linux daemon that generates messages for Nagios from them. The software package NET-SNMP, described in Section 11.2.2 from page 187, includes the daemon snmptrapd.

In the following scenario, snmptrapd is installed on a third host (neither the computer generating the trap, nor the Nagios server). It evaluates the information received by means of a script and forwards it with NSCA to the Nagios server.12

In the snmptrapd configuration file /etc/snmp/snmptrapd.conf, each trap type is given a separate entry, the syntax of which corresponds to one of the following lines:

traphandle oid program traphandle oid program arguments traphandle default program

traphandle default program arguments

The keyword traphandle is followed either by the object identifier of the desired trap, or by the keyword default. In the second case the entry applies to all traps that do not have their own configuration entry. Finally the program that should run if a relevant trap arrives is specified.

11http://www.opennms.org/

12If you install the snmptrapd on the Nagios server itself, you do not need NSCA and you can send a correspondingly formatted command, as described in Section 13.2 from page 241 directly to the interface for external commands.

260

14.6 Application Example II: Processing SNMP Traps

In addition you can also include arguments used with this program. But you must be a bit careful when doing this. Quotation marks are passed on by snmptrapd as characters and spaces are always used as delimiters. This means that you cannot pass on any arguments containing spaces, which you should bear in mind when assigning name services in Nagios.

snmpdtrapd gives this program information via the standard output in the following format:

hostname ip-address oid value

...

The first line contains the fully qualified domain name of the host that sends the message and the second, its IP address. Then one or more OID-value pairs are given, each on a separate line. A particular event is very often linked to a unique OID-value pair, so that the program can often omit the evaluation of the OID-value pair entirely.

In the following snmptrapd.conf example, the lines are wrapped for readability. Each traphandle instruction must be entered on a single line:

# snmptrapd.conf

traphandle SNMPv2-MIB::coldStart /usr/local/nagios/libexec/eventhandler/ handle-trap SNMP cold-start

traphandle NET-SNMP-AGENT-MIB::nsNotifyRestart /usr/local/nagios/libexec /eventhandler/handle-trap SNMP restart

traphandle NET-SNMP-AGENT-MIB::nsNotifyShutdown /usr/local/nagios/libexe c/eventhandler/handle-trap SNMP shutdown

traphandle default /usr/local/nagios/libexec/eventhandler/handle-trap SN MP unknown

The traps used here are sent by the SNMP agent snmpd from the NET-SNMP package by default, as long as a destination was specified in snmpd.conf:

# snmpd.conf

trapsink name_or_ip_of_the_nagios-server

If a trap arrives with the OID SNMPv2-MIB::coldStart, for example, snmptrapd starts the script handle-trap with the argument cold-start. In this way it does not have to search first for the necessary information from the OID-value pairs. However, this shortcut only works with trap OID names that describe their function.

14.6.2 Passing on traps to NSCA

The script handle-trap, which is run by snmptrapd, breaks down the information passed on and hands it over, correctly formatted, to send_nsca:

261

14 The Nagios Service Check Acceptor (NSCA)

#!/bin/bash

NAGIOS="nagsrv"

LOGFILE="/usr/local/nagios/var/handle-trap.log"

read HOST && echo "host: $HOST" >> $LOGFILE read IPADDR && echo "ip: $IPADDR" >> $LOGFILE

case $IPADDR in 192.168.201.4)

HOSTNAME="irouter"

;;

*)

# silent discard from unknown hosts exit 0

;;

esac

if [ -z "$1" ]; then

echo "usage: $0 <service> <key>"

echo "usage: $0 <service> <key>" >> $LOGFILE exit 1

else SERVICE="$1"

fi

if [ ! -z "$2" ]; then SWITCH="$2"

fi

case $SWITCH in "cold-start")

OUTPUT="snmpd: Cold Start" STATE=0

;;

restart)

OUTPUT="snmpd: Restart" STATE=1

;;

shutdown)

OUTPUT="snmpd: Shutdown" STATE=2

;;

*)

OUTPUT="Unknown Trap" STATE=1

;;

esac

CMD="$HOSTNAME;$SERVICE;$STATE;$OUTPUT"

262

14.6 Application Example II: Processing SNMP Traps

echo "$CMD" >> $LOGFILE

echo "$CMD" | /usr/bin/send_nsca -H $NAGIOS -d ’;’ \

-c /etc/nagios/send_nsca.cfg >> $LOGFILE 2>&1

First it saves the log file and the name of the Nagios server nagsrv, each in a separate variable. The first case statement specifies the host name used by Nagios for the IP address passed on (and temporarily stored in IPADDR). HOST normally contains the fully qualified domain name, which also cannot be used directly, and sometimes also just contains one IP address, so that it is better to use the latter here. The explicit test also allows it to discard traps from undesired hosts. Finally, matching traps land without further authentication on the Nagios server.13

The following if statement determines whether a service name was also given to the script. If this is the case, then it is saved in the SERVICE variable. If there was a second argument, the procedure is similar. Depending on the value, the next “case $SWITCH” instruction defines the output text and the desired status for Nagios.

The command for NSCA is finally assembled and the CMD variable is passed on by the script to send_nsca. As in previous examples, a semicolon is used as the delimiter, which must be specified in send_nsca with the option -d.

14.6.3 The matching service definition

As in the syslog-ng example (page 257), we again define the service on the Nagios server as a purely passive one:

Since soft states do not make any sense in a single trap message, we should set max_check_attempts back to 1. Whether the parameter is_volatile is used or not depends on the purpose to which the service is put. As long as you define a separate service for each error category, there is no problem in omitting is_volatile. But if you form different error categories using a single service, you should set is_volatile 1, because in this case the previous error will seldom have anything to do with the new one. Section 14.5.2 on page 257 is devoted to the subject of volatile services.

13Although SNMPv3 does provide authentication for SNMP traps, this would go beyond the scope of this book.

263