References

f Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10

and OpenSolaris; Brendan Gregg, Jim Mauro, Richard McDougall; Prentice Hall;

Isbn-13: 978-0131568198

f DTraceToolkit website at http://www.brendangregg.com/dtracetoolkit.html

f Dtrace.org website at http://dtrace.org/blogs/

9

Configuring the

Syslog and Monitoring

Performance

In this chapter, we will cover the following topics:

f Configuring the syslog

f Monitoring the performance on Oracle Solaris 11

Introduction

In this chapter, we will learn about two important topics: syslog and performance monitoring. The former is an essential task for daily administration and is very appropriate for resolving the following possible events and problems that occur in Oracle Solaris 11. Configuring syslog is very similar to other UNIX flavors, but there will be particular details that are exclusively related to Oracle Solaris.

Talking about the syslog framework means discussing a very important part of the system that is responsible for event messages. Any security problem, hardware change and problem, kernel event, or general issues will be recorded in logfiles. Additionally, applications will log their messages in logfiles. The syslog framework plays a special role if we are working with forensic analysis. Syslog framework has a central role. Logs are also important when we investigate

a malware's attack. If we have to create a troubleshooting process, once more, the records saved and managed by the syslog framework are vital. This is the real importance of the syslog framework because its responsibility is to forward any kind of message to the logfiles, according to the category and severity of the message.

Configuring the Syslog and Monitoring Performance

The latter topic, performance monitoring, introduces us to a complete and new world where it would be possible to write a whole book on the subject. The idea here is to learn about the main fundamentals and commands to help find out performance problems in the system.

The gathered metrics can be used for a tuning task where the main goal is to improve the performance and try to keep the same hardware. This is useful because managers do not want to spend money buying an unnecessary and expensive hardware when eventually, only some modifications in the system will be enough.

Configuring the syslog

The syslog framework is one of the most important features of Oracle Solaris 11, because its goal is to log all the events that occur in each second. These records can be used to investigate any suspicious behavior on the system. Like most books, we will not delve into unnecessary details and theory about syslog. The main idea here is to show how the syslog can be configured, monitored, and used.

Getting ready

This recipe requires two virtual machines (VirtualBox or VMware) named solaris11-1 and solaris11-2, both running Oracle Solaris 11 with at least 2 GB RAM, and a network interface.

How to do it…

The syslog framework is composed of a main daemon (syslogd) and its respective configuration file (/etc/syslog.conf). To gather details about the associated syslog service, we have to execute the following SMF administration commands:

root@solaris11-1:~# svcs -l svc:/system/system-log:default

fmri svc:/system/system-log:default

name system log

enabled true

state online next_state none

state_time May 19, 2014 01:29:14 AM BRT

logfile /var/svc/log/system-system-log:default.log restarter svc:/system/svc/restarter:default contract_id 117

manifest /root/chapter5/myprofile.xml manifest /etc/svc/profile/generic.xml

Chapter 9

manifest /lib/svc/manifest/system/system-log.xml

dependency require_all/none svc:/milestone/self-assembly-complete (online)

dependency require_all/none svc:/system/filesystem/local (online) dependency optional_all/none svc:/system/filesystem/autofs (online) dependency require_all/none svc:/milestone/name-services (online) root@solaris11-1:~# svcs -x svc:/system/system-log:default svc:/system/system-log:default (system log)

State: online since May 19, 2014 01:29:14 AM BRT See: syslogd(1M)

See: /var/svc/log/system-system-log:default.log Impact: None.

As we mentioned about the syslog service, there's a configuration file named

/etc/syslog.conf, as shown in the following command:

root@solaris11-1:~# more /etc/syslog.conf

#

#ident "%Z%%M% %I% %E% SMI" /* SunOS 5.0 */

#

# Copyright (c) 1991-1998 by Sun Microsystems, Inc.

# All rights reserved.

#

# syslog configuration file.

#

# This file is processed by m4 so be careful to quote (`') names

# that match m4 reserved words. Also, within ifdef's, arguments

# containing commas must be quoted.

#

*.err;kern.notice;auth.notice /dev/sysmsg

*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages

*.alert;kern.err;daemon.err alexandre

*.alert root

Configuring the Syslog and Monitoring Performance

*.emerg *

# if a non-loghost machine chooses to have authentication messages

# sent to the loghost machine, un-comment out the following line:

#auth.notice ifdef(`LOGHOST', /var/log/authlog, @loghost) mail.debug ifdef(`LOGHOST', /var/log/syslog, @loghost)

#

# non-loghost machines will use the following lines to cause "user"

# log messages to be logged locally.

#

ifdef(`LOGHOST', ,

user.err /dev/sysmsg

user.err /var/adm/messages user.alert `root, operator'

user.emerg *

)

This configuration file is straight and has only two columns, selector and target, both separated by tabs (not spaces).

The selector column is composed of two components in the facility.level format, and the syntax is defined as follows:

<facility>.<level> <target>

The facility component determines the class or category of message (KERN, USER, MAIL, DAEMON, AUTH, NEWS, UUCP, CRON, AUDIT, LOCAL 0-7, and *), and the level component means the priority (EMERG, ALERT, CRIT, ERROR, WARNING, NOTICE, INFO, and DEBUG, in the descending order). Additionally, the target column is the destination of the message, where the destination can be a device file, file, user, or host.

We will now see some practical examples of the /etc/syslog.conf configuration file with its respective syntax:

f *.err;kern.notice;auth.notice /dev/sysmsg: All messages with an error (err) priority (the facility doesn't matter), any kernel facility messages with a priority equal to or higher than notice (notice, warning, error, critical, alert, and emergency), and any authentication (auth) facility message with a priority equal to or higher than notice are sent to /devsysmsg

Chapter 9

f *.err;kern.debug;daemon.notice;mail.crit /var/adm/messages: All messages with a debug priority (the facility doesn't matter), any kernel facility message with a debug level or higher, any daemon facility message with notice priority or higher, and all mail facility messages with critical priority or higher are sent to the /var/adm/messages file

f *.alert;kern.err;daemon.err alexandre: In this example, all messages with priority equal to or higher than alert, messages with facility equal to kernel and priority error (err) or higher, and messages with facility equal to daemon and priority error (err) or higher are sent to the alexandre user

f *.emerg *: In this line, all messages with priority level equal to or higher than emerg are sent to every user that is logged on

f local7.alert @solaris11-2: Any message with the local7 facility and priority level equal to or higher than alert is sent to another host (solaris11-2)

f mail.debug ifdef(`LOGHOST', /var/log/syslog, @loghost): This is a nice example because any message with the facility equal to mail and priority level equal to debug or higher can be sent to two different destinations specified in the /etc/hosts file

If the LOGHOST variable (as shown earlier) is set (defined in the same line) to the localhost, the mail.debug messages are sent to the /var/log/syslog file. However, if the LOGHOST keyword is set to another host (for example, solaris11-2 machine), then the mail.debug message is sent to the solaris11-2 machine.

As the /etc/hosts file is used to specify these special hostnames, we can verify an example as follows:

root@solaris11-1:~# more /etc/hosts

#

# Copyright 2009 Sun Microsystems, Inc. All rights reserved.

# Use is subject to license terms.

#

# Internet host table

#

::1 solaris11-1 localhost

127.0.0.1 solaris11-1 localhost loghost 192.168.1.144 solaris11-1 solaris11-1.example.com 192.168.1.155 solaris11-2 solaris11-2.example.com

In this case, loghost is configured to the localhost (solaris11-1), so any message

with facility equal to mail and priority level equal to debug must be sent to the /var/adm/ message file.

Configuring the Syslog and Monitoring Performance

From these examples, you can note some of the following points:

f A message can be sent to two or more different places, as seen in the first two examples

f If a message is sent to another host, such as the last configuration line's example, the target host must have a similar line to handle the arriving message

f Any change in the /etc/syslog.conf file requires restarting the syslog service (svcadm restart svc:/system/system-log:default and svcadm restart svc:/system/system-log:default)

Let's create a real test. In the solaris11-1 system, edit the /etc/syslog.conf file and add the following line:

local7.emerg @solaris11-2

Add the solaris11-2 system in the /etc/hosts file on the solaris11-1 machine, and make sure that it's accessible from the solaris11-1 system, as shown in the following commands:

root@solaris11-1:~# ping solaris11-2

solaris11-2 is alive

root@solaris11-1:~# more /etc/syslog.conf (truncated output)

# non-loghost machines will use the following lines to cause "user"

# log messages to be logged locally.

#

ifdef(`LOGHOST', ,

user.err /dev/sysmsg

user.err /var/adm/messages user.alert `root, operator'

user.emerg *

)

local7.emerg @solaris11-2

On the solaris11-1 system, refresh the syslog service by executing the following command:

root@solaris11-1:~# svcadm refresh svc:/system/system-log:default

Chapter 9

If the syslog configuration doesn't take effect for some reason, you can restart it by running the following command:

root@solaris11-1:~# svcadm restart svc:/system/system-log:default

root@solaris11-1:~# svcs svc:/system/system-log:default

STATE STIME FMRI

online 4:58:45 svc:/system/system-log:default

On another system (solaris11-2), we have to include the following line at end of the

/etc/syslog.conf file:

local7.emerg /var/adm/new_messages

As this file doesn't exist, we can create it as shown in the following command:

root@solaris11-2:~# touch /var/adm/new_messages

There is a property from the system-log:default service named log_from_remote, and it should be set to true to allow remote hosts (solaris11-1) to log any message into the solaris11-2 system. Nonetheless, the big issue is that this parameter is usually configured to false. Additionally, a configuration file (/etc/default /syslog) also controls the remote logging behavior, as shown in the following command:

root@solaris11-2:~# more /etc/default/syslogd

#ident "%Z%%M% %I% %E% SMI"

#

# Copyright 2006 Sun Microsystems, Inc. All rights reserved.

# Use is subject to license terms.

#

# /etc/default/syslogd

#

# Legacy configuration file for syslogd(1M). See syslogd(1M).

#

# This file should no longer be directly used to configure syslogd.

# These settings are kept here for backwards compatibility only.

# Please use svccfg(1M) to modify the properties of syslogd(1M).

#

# The LOG_FROM_REMOTE setting used to affect the logging of remote

# messages. Its definition here will override the svccfg(1M) settings

# for log_from_remote.

#

#LOG_FROM_REMOTE=YES

Configuring the Syslog and Monitoring Performance

Now, let's take a look at the details. If this LOG_FROM_REMOTE variable (from the /etc/ default/syslogd file) is set to YES or NO, the log_from_remote property (from the system-log:default service) is enabled or disabled, respectively. However, if the LOG_FROM_REMOTE variable is commented out (as shown in the previous file), the value of the log_from_remote property (from the system-log:default service) takes effect.

Therefore, to make our lives easier, we are going to enable the log_from_remote property, without touching the /etc/default/syslogd configuration file, and restart the service, as shown in the following command:

root@solaris11-2:~# svccfg -s svc:/system/system-log setprop config/log_ from_remote = true

root@solaris11-2:~# svcadm restart svc:/system/system-log:default

root@solaris11-2:~# svcs svc:/system/system-log:default

STATE STIME FMRI

online 13:38:17 svc:/system/system-log:default

On the same solaris11-2 system, we have to follow /var/adm/new_messages to confirm if the message from solaris11-1 arrives, using the next command:

root@solaris11-2:~# tail -f /var/adm/new_messages

On the solaris11-1 system, it is time to test the configuration, and we can use the logger command that generates a message with the facility and level specified, using the -p option. In this case, we are going to generate the Alexandre Borges message that will be classified as local7 and with priority level emerg. According to the /etc/syslog.conf configuration file, the message will be sent to the solaris11-2 host. Once it is there, the message will be sent to the /var/adm/new_messages file, as shown in the following command:

root@solaris11-1:~# logger -p local7.emerg Alexandre Borges

And we're done! Returning to the solaris11-2 host, we are able to confirm that the message has arrived by executing the following command:

root@solaris11-2:~# tail -f /var/adm/new_messages

May 19 13:41:44 solaris11-1.example.com root: [ID 702911 local7.emerg] Alexandre Borges

This is perfect! Everything worked as expected!

Chapter 9

Proceeding with the explanation about logging, some network services have their own log configuration, and the best way to understand this is by taking a look at another practical example. For example, pick the telnet service and examine its configuration using the following command:

root@solaris11-1:~# inetadm -l telnet

SCOPE NAME=VALUE

name="telnet" endpoint_type="stream" proto="tcp6" isrpc=FALSE

wait=FALSE exec="/usr/sbin/in.telnetd" user="root"

default bind_addr="" default bind_fail_max=-1

default bind_fail_interval=-1 default max_con_rate=-1 default max_copies=-1

default con_rate_offline=-1 default failrate_cnt=40 default failrate_interval=60 default inherit_env=TRUE

tcp_trace=FALSE default tcp_wrappers=FALSE default connection_backlog=10 default tcp_keepalive=FALSE

As we are able to see in the previous output, the tcp_trace property is set to false, and this way, no telnet message is sent to the syslog service. It is possible to change this default behavior by running the following commands:

root@solaris11-1:~# inetadm -m telnet tcp_trace=true root@solaris11-1:~# inetadm -l telnet | grep tcp_trace tcp_trace=TRUE

root@solaris11-1:~# inetadm | grep telnet

enabled online svc:/network/telnet:default

Configuring the Syslog and Monitoring Performance

To verify the telnet events, we must execute a telnet operation from the solaris11-2 system and check the /var/adm/messages file in the solaris11-1 host, as shown in the following command:

root@solaris11-2:~# telnet solaris11-1

Trying 192.168.1.144...

Connected to solaris11-1.example.com. Escape character is '^]'.

login: borges

Password: hacker123!

Oracle Corporation SunOS 5.11 11.1 September 2012

-bash-4.1$

On the solaris11-1 host, verify the /var/adm/message's file log content by executing the following command:

root@solaris11-1:~# tail -3 /var/adm/messages

May 19 15:03:44 solaris11-1 mDNSResponder: [ID 702911 daemon.warning] SendResponses: No active interface to send: 33 _OSInstall._tcp.local. PTR borges_ai._OSInstall._tcp.local.

May 19 15:03:44 solaris11-1 mDNSResponder: [ID 702911 daemon.warning] SendResponses: No active interface to send: 36 _OSInstall._tcp.local. PTR default-i386._OSInstall._tcp.local.

May 19 15:15:02 solaris11-1 inetd[829]: [ID 317013 daemon.notice]

telnet[2677] from 192.168.1.155 40498

It worked! However, why do messages from services that were configured using tcp_trace=true go to the /var/adm/message file? Because all the messages that originated from this attribute are classified as daemon.notice (remember the facility.severity syntax), and according to the /etc/syslog.conf file, we have the following command:

root@solaris11-1:~# cat /etc/syslog.conf | grep /var/adm/messages

*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages user.err /var/adm/messages

Instead of configuring the logging capacity in each network service, we can configure the logging feature for all network services, using a simple command:

root@solaris11-1:~# inetadm -M tcp_trace=true

Chapter 9

Now, all the network services that are controlled by the inetadm framework are configured to log to the system-log:default service according to the /etc/syslog.conf configuration file.

An overview of the recipe

We learned how to configure the logging service in Oracle Solaris 11 using the

system-log:default service (the syslogd daemon) and its respective configuration file (/etc/syslog.conf). Additionally, we saw how to configure the logging feature for network services that are controlled by the inetadm framework.

Monitoring the performance on Oracle

Solaris 11

When we are working in an environment with many available resources, without doubt, it is easier to administer all systems. However, how can we handle critical systems that run Oracle Solaris 11 with few free resources? How can we find and monitor these rare resources on Oracle Solaris 11?

The performance and tuning subject on Oracle Solaris is a very long and dense topic to be explained in a complete way; it deserves an entire book dedicated to all its details. However, we will learn enough monitor details and commands that will motivate you to study this topic deeply.