Getting ready
This recipe requires a virtual machine (VirtualBox or VMware) that runs Oracle Solaris 11 with 2 GB RAM at least. It is recommended that the system has two or more processors or cores.
How to do it…
Fundamentally, Oracle Solaris 11 is composed of CPU, RAM, and I/O devices, and there are many ways to monitor the system. Furthermore, there are some parameters that are very important, so it's appropriate to start our studies by examining the memory subsystem.
The first step is to enable the system to collect the sar statistics, as shown in the following command:
root@solaris11-1:~# svcs -a | grep sar
disabled 0:37:02 svc:/system/sar:default root@solaris11-1:~# svcadm enable svc:/system/sar:default root@solaris11-1:~# svcs -a | grep sar
online 4:34:57 svc:/system/sar:default
Configuring the Syslog and Monitoring Performance
Using either the prtconf or the lgrpinfo command, we can find out the total installed memory. In addition, by executing the pagesize command, we can find the page size of a page in memory, and finally, we can use sar –r to get the free memory and swap space, as shown in the following command:
root@solaris11-1:~# prtconf | grep -i memory
Memory size: 4096 Megabytes root@solaris11-1:~# lgrpinfo lgroup 0 (root):
Children: none CPU: 0
Memory: installed 4.0G, allocated 1.3G, free 2.7G Lgroup resources: 0 (CPU); 0 (memory)
Load: 0.297
Latency: 0
root@solaris11-1:~# pagesize
4096
root@solaris11-1:~# sar -r 1 3
SunOS solaris11-1 5.11 11.1 i86pc 05/21/2014
-
01:45:09
freemem
freeswap
01:45:10
632394
5876128
01:45:11
632439
5877184
01:45:12
632476
5876128
Average
632436
5876480
In the preceding command, the freemem column is the average number of available pages (4K in this case), and the freeswap column means the average number of disk blocks designed for page swapping.
The free memory (in pages of 4 KB) can also be obtained using a very smart command:
root@solaris11-1:~# kstat -p unix:0:system_pages:freemem
unix:0:system_pages:freemem 632476
Chapter 9
A typical way to get the free swap space is using the following command:
root@solaris11-1:~# swap –l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 285,2 8 2097144 2097144
In this case, we should remember that free space is shown in sectors (512 bytes).
Taking a different way, free swap information can be obtained from Modular Debugger (MDB):
root@solaris11-1:~# echo ::swapinfo | mdb –k
ADDR VNODE PAGES FREE NAME
ffffc1000743e378 ffffc10007df3d00 262143 262143 /dev/zvol/dsk/ rpool/swap
Furthermore, the same MDB can provide us with lots of information about the memory status by executing the following command:
-
root@solaris11-1:~# echo ::memstat
Page Summary Pages
| mdb –k
MB
%Tot
------------ ----------------
----------------
----
Kernel 215458
841
21%
ZFS File Data 132510
517
13%
Anon 101485
396
10%
Exec and libs 4105
16
0%
Page cache 17361
67
2%
Free (cachelist) 14411
56
1%
Free (freelist) 563133
2199
54%
Total 1048463
4095
This output shows the amount of memory used by kernel (Kernel), amount of memory used by data from ZFS File Data (ZFS), and the number of anonymous pages (a sum of heap, stack, shared memory, and copy on write pages) in memory.
The page cache (stored on virtual memory) is made by all the recently read and written regular filesystem data (file and directory data) other than ZFS (the ZFS data is on Adaptive Replacement Cache (ARC)). As we mentioned earlier, regular ZFS filesystem data is stored on the page cache because mmap ZFS data also stays there.
Configuring the Syslog and Monitoring Performance
Free (freelist) is the real amount of free memory without any connection to the processes and files. The cache list is the number of unmapped file pages on the free list.
The basic and rough working of page cache is that any necessary filesystem data is fetched on
the segmap cache.
The Segmap cache is a kind of first-level cache or staging area, where recent pages that were read from the filesystem (UFS, VxFS, NFS, and QFS) are kept into pages of kernel's virtual memory to be copied to user space buffers. Nevertheless, if the information is not found on the segmap cache, the kernel tries to find the requested data on cachelist (unmapped filesystem pages). Additionally, the segmap cache is not used by the ZFS filesytem. An interesting concept is that freelist is linked to cachelist, showing that when some free page of memory is necessary, first, the kernel tries to take pages from freelist, but if it isn't possible, the kernel takes a page of memory from cachelist.
By the way, only for completeness, the segmap cache statistics could be found by running the following command:
root@solaris11-1:~# kstat -n segmap
module: unix instance: 0
-
name: segmap
class:
vm
crtime
0
fault
69
faulta
0
free
0
free_dirty
0
(truncated output)
The kstat command is also appropriate to show complementary page system information. Remember that a page size in memory is 4 KB, and it can be found by executing the following command:
root@solaris11-1:~# kstat -n system_pages
module: unix instance: 0
name: system_pages class: pages availrmem 696041
crtime 0
desfree 8159
desscan 25
econtig 4229439488
fastscan 473831
freemem 585862
-
kernelbase lotsfree
0
16318
Chapter 9
minfree
4079
nalloc
44993593
nalloc_calls
19577
nfree
42000307
nfree_calls
13223
nscan
0
pagesfree
585862
pageslocked
348325
pagestotal
1044366
physmem
1044366
pp_kernel
362807
slowscan
100
snaptime
6181.186029253
An additional and interesting note: availrmem is the amount of unlocked memory available for allocation. Furthermore, if we take the same kstat command, it is possible to get system-wide page statistics, as shown in the following command:
root@solaris11-1:~# kstat -n vm
module: cpu instance: 0
name: vm class: misc
anonfree 0
anonpgin 0
anonpgout 0
as_fault 941681
cow_fault 151186
crtime 42.291984164
dfree 0
execfree 0
execpgin 0
execpgout 0
fsfree 0
fspgin 32
fspgout 0
hat_fault 0
kernel_asflt 0
maj_fault 5
pgfrec 149071
Configuring the Syslog and Monitoring Performance
-
pgin
6
pgout
0
pgpgin
32
pgpgout
0
pgrec
149071
pgrrun
4
pgswapin
0
pgswapout
0
prot_fault
162132
rev
0
scan
97276
snaptime
6715.331061273
softlock
17396
swapin
0
swapout
0
zfod
399824
From this huge output, some parameters stand out:
f anonfree: This defines heap and stack pages that were released after these pages have been paged out to the disk
f anopgin: This defines heap and stack pages paged in from the disk
f anonpgout: This defines heap and stack pages paged out from the swap
f maj_fault: This defines the number of operations where the page has been found on the disk because it wasn't on memory
f pgswapin: This defines the number of pages swapped in
f pgswapout: This defines the number of pages swapped out
Returning to general memory statistics, there is a known command named vmstat (which uses the –p option to report paging activity) that can disclose useful details, as shown in the following command:
root@solaris11-1:~# vmstat -p 1 5
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo
fpf
3243004 2845352 237 1317 0 0 266 0 0 0 0 0 0 0 0
0
2844188 2438808 14 57 0 0 0 0 0 0 0 0 0 0 0
0
Chapter 9
2843132 2438060 0 2 0 0 0 0 0 0 0 0 0 0 0
0
2843132 2437664 0 2 0 0 0 0 0 0 0 0 0 0 0
0
2842604 2437128 0 25 0 0 0 0 0 0 0 0 0 0 0
0
This output brings to us some interesting information about swap (the available swap space in KB) and free (amount of free memory). There are other critical parameters such as sr (number of pages scanned per second during an operation to find enough free memory), api
(anonymous page-ins), and apo (anonymous page-outs). Usually, an sr value (scan rate) above zero indicates problems with lack of memory, and a high value of either the api or apo value indicates low memory and a high number of operations to and from the swap. Additionally, as anonymous page-in operations have a bad impact on the system's performance, we could use the DTrace tool to find all the executables that make many page-in operations by running either of the following commands:
f root@solaris11-1:~# dtrace -n 'vminfo:::anonpgin { @[pid, execname] = count(); }'
f root@solaris11-1:~# dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'
After we find what executable is causing a performance impact, it is time to decide what we can do. Eventually, it could be necessary to move the offending application to another system.
There is an interesting way to verify that a process is facing problems with memory, using the
prstat command as shown:
root@solaris11-1:~# prstat -mLc 1 1
The previous command gives the following output:
Configuring the Syslog and Monitoring Performance
First, the options we used here were as follows:
f -m: This reports microstate process accounting information.
f -c: This prints new reports below previous reports instead of overprinting them.
f -L: This reports statistics for each light-weight process (LWP). By default, the
prstat command reports only the number of LWPs for each process.
The DFL column represents the percentage of time the process has spent processing data page faults, in other words, lack of enough memory. Ideally, this value should be zero.
For daily administration, we are used to executing the vmstat command to gather information about virtual memory, as shown in the following command:
root@solaris11-1:~# vmstat 1 5
kthr memory page disk faults cpu
-
r sy
b w id
swap
free
re mf pi po
fr
de
sr
s0
s1
s2
--
in
sy
cs
us
0
0 0
2855860
2439648 28 159 0 0
0
0
19
12
0
0
0
568
2519
741
2
4 94
0 0
0
2773564
2364844
14
57
0
0
0
0
0
0
0
0
0
552
793
587
1
3 96
1 0
0
2773564
2364764
0
0
0
0
0
0
0
0
0
0
0
583
677
590
1
2 97
0 0
0
2773564
2364764
0
0
0
0
0
0
0
0
0
0
0
548
662
567
1
4 95
0 0
0
2773564
2364764
0
0
0
0
0
0
0
0
0
0
0
566
655
574
2
2 96
Some cool columns are w (number of swapped-out threads), swap (free swap space in KBs), free (free memory, including page cache and free lists in KB), re (number of reclaimed pages from page cache), pi and po (KB of page paged in and out, respectively), and sr (pages scanned in memory for available pages).
A nice variation from the preceding command is vmstat –s (displays the total number of various system events since boot), as shown in the following commands:
root@solaris11-1:~# vmstat –s
0 swap ins
0 swap outs
0 pages swapped in
0 pages swapped out
762167 total address trans. faults taken
7 page ins
Chapter 9
0 page outs
7 pages paged in
0 pages paged out
135490 total reclaims
135490 reclaims from free list
0 micro (hat) faults 762167 minor (as) faults
7 major faults
148638 copy-on-write faults 257547 zero fill page faults
148476 pages examined by the clock daemon
0 revolutions of the clock hand
0 pages freed by the clock daemon 967 forks
1670 vforks
2876 execs
2840211 cpu context switches 1877292 device interrupts
925020 traps
8412869 system calls
2901338 total name lookups (cache hits 86%) 9114 user cpu
16207 system cpu
298592 idle cpu
0 wait cpu
I have highlighted the main statistics from this vmstat –s output as follows:
f pages swapped in: This refers to the number of pages swapped in (from swap to memory). The ideal value is zero.
f pages swapped out: This refers to the number of pages swapped out (from memory to swap). The ideal value is zero.
f reclaims from free list: This refers to the total of the reclaimed pages from the free page cache inside the free list. Reclaimed pages are pages of memory that were released because of space shortage, but they still were not used for other processes nor paged out to swap. A high value can evince lack of memory.
f major faults: This refers to the number of pages not found on physical memory;
these pages were fetched on disk. The ideal value is close to zero.
Configuring the Syslog and Monitoring Performance
f total name lookups: Every time a file is opened, its pathname is stored in a special place named Directory Name Lookup Cache (DNLC). These statistics show us how many times the kernel found the directory path in cache (DNLC), and it does not have to fetch this information on disk. Values above 90 percent are great! Another way to get information about DNLC is using kstat.
It is possible to gather specialized DNLC cache information by executing the following command:
-
root@solaris11-1:~# kstat -n
module: unix name: dnlcstats
dnlcstats
instance: class:
0
misc
crtime
38.737278004
dir_add_abort
0
dir_add_max
0
dir_add_no_memory
0
dir_cached_current
0
dir_cached_total
0
dir_entries_cached_current
0
dir_fini_purge
0
dir_hits
0
dir_misses
0
dir_reclaim_any
0
dir_reclaim_last
0
dir_remove_entry_fail
0
dir_remove_space_fail
0
dir_start_no_memory
0
dir_update_fail
0
double_enters
40
enters
112579
hits
2439710
misses
408555
negative_cache_hits
89113
pick_free
0
pick_heuristic
0
pick_last
0
purge_all
0
purge_fs1
0
purge_total_entries purge_vfs
436
60
10
Chapter 9
purge_vp 40
snaptime 1136.042407346
It is possible to calculate the efficiency of DNLC, which is (hits/(hits + misses)) * 100. Therefore, according to the previous output, we have the following:
DNLC's efficiency = (2439710/ (2439710 + 408555) * 100 = 85,67 percent
As an interesting interpretation from the previous output, every hundred times we start searching for directory path information on disk, 85 times, this information is found in a DNLC cache.
Another clever method to get DNLC statistics is using the sar command as follows:
root@solaris11-1:~# sar -a 1 5
SunOS solaris11-1 5.11 11.1 i86pc 05/21/2014 04:37:12 iget/s namei/s dirbk/s
04:37:13 19 14 6
04:37:14 5 14 4
-
04:37:15
10
18
9
04:37:16
13
10
11
04:37:17
8
13
12
Average 11 14 8
The iget/s column shows us how many requests for the inode directory path were not found on DNLC. Zero is an ideal value for this column. Nonetheless, if the iget/s value is not equal to zero, we can change the ncsize parameter to improve this statistic by changing the
/etc/system file, as shown in the following command:
set ncsize = value
The ncsize parameter defines the number of entries in the directory name look-up cache (DNLC), and this parameter is used by UFS, NFS, and ZFS to cache elements of path names that have been resolved.
By default, the value is dynamically calculated using the formula (4 * (v.v_proc + maxusers)
+ 320) + (4 * (v.v_proc + maxusers) + 320) / 100. Additionally, the current value is found by executing the following command:
root@solaris11-1:~# echo ncsize/D | mdb -k
ncsize:
ncsize: 129797
Configuring the Syslog and Monitoring Performance
When talking about DNLC, another hot topic arises; this is the buffer cache. The buffer cache holds the metadata for inodes, which have directory path information on DNLC. The buffer cache statistics are obtained by running a command, as shown in the following screenshot:
The %rcache and %wcache columns tell us about the percentage of times metadata information was found in the buffer cache for the read/write operations, respectively. Very good values are %rcache, which should be more than 90 percent, and %wcache, which should be more than 70 percent.
While managing memory performance, a final critical issue is to know if a physical error occurred, such as fault memory or an ECC error. In this case, we can verify that the memory and all other hardware components in system are working well by running the following command:
root@solaris11-1:~# fmadm faulty
root@solaris11-1:~# fmstat -s -m cpumem-retire
NAME >N T CNT
DELTA STAT
Fortunately, there are no errors on my machine.
A huge concern when trying to tune the performance on Oracle Solaris 11 is the CPU. Eventually, the potential performance problem in the system is that the CPU is not able to attend all requests on time. For example, the vmstat command helps us identify if the operating system shows a CPU bottleneck by executing the following command:
root@solaris11-1:~# vmstat 1 5
kthr memory page disk faults cpu
Chapter 9
r b w swap free re mf pi po fr de sr s0 s1 s2 -- in sy cs us sy id
5 3 0 2876068 2470716 77 324 0 0 0 0 89 21 0 1 0 580 2862 904 3
5 92
-
3
3
0
2764212
2357504
14
58 0
0
0
0
56 14
0
0
0
547 758 582
2
3
95
7
4
0
2763576
2357380
67
88
0
0
0
0 72
13
0
0
0 601 1445 691
4
4
92
2
4
0
2763576
2357380
34
65
0
0
0
0 72
16
0
0
0 586 1595 700
4
5
91
5
2
0
2763576
2357380
25
64
0
0
0
68 65
11
0
0
0 614 1904
752
5
5 90
The kthr:r column means the total number of ready threads on the run queue (sum of the dispatches queues) that wait for CPUs. A constant value above the sum of the core processors or CPUs can represent a processor bottleneck, and dividing the kthr:r/number of CPUs or cores is a good way to compare CPU performance among servers.
Using the DTraceToolkit, we have the dispqlen.d script that shows each CPU or core dispatch queue that indicates any case of CPU saturation. Fortunately, it is not the case in the following command:
root@solaris11-1:~/DTraceToolkit-0.99/Bin# ./dispqlen.d
Sampling... Hit Ctrl-C to end.
^C^C
-
CPU
2
value
------------- Distribution -------------
< 0
|
0
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
1
|@
2
|
3
|
CPU
0
value
------------- Distribution -------------
< 0
|
0
0
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
75157
1
|@
2431
2
|
52
3
|
0
439
Configuring the Syslog and Monitoring Performance
CPU 1
-
value
< 0
------------- Distribution -------------
|
count 0
0
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
75097
1
|@
2467
2
|
67
3
|
8
4
|
1
5
|
0
CPU 3
-
value
< 0
------------- Distribution -------------
|
count 0
0
|@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
75051
1
|@
2540
2
|
49
3
|
0
A similar command that shows a similar output to the kthr:r column from vmstat is the
runq-sz column from sar –q:
root@solaris11-1:~# sar -q 1 4
SunOS solaris11-1 5.11 11.1 i86pc 05/21/2014
-
21:52:45
runq-sz
%runocc
swpq-sz
%swpocc
21:52:46
0.0
0
0.0
0
21:52:47
2.0
100
0.0
0
21:52:48
1.0
100
0.0
0
21:52:49
1.0
100
0.0
0
Average
1.3
75
0.0
0
The %runocc file explains the average run queue occupancy that helps us identify the eventual burst in the run queue.
Returning to the vmstat output, other useful fields are cpu: us (user time—how much time the CPU spends processing user threads), cpu:sy (system time—how much time the CPU spends processing kernel threads and system calls), and cpu:id (idle time—percentage of time that CPUs are waiting for runnable threads). A practical way to evaluate potential CPU problems is by considering that a good balance between user time (cpu:us) and system time (cpu:sy) is about 90/10 (depending on applications running on the system). Additionally,
an upper limit is 70/30, at maximum (limit). Any system showing values different from these ranges deserves an investigation.
Chapter 9
Most of the previous columns can be viewed in a similar way by executing commands such as the following one:
root@solaris11-1:~# sar 1 3
-
SunOS solaris11-1
5.11 11.1
i86pc
05/21/2014
20:39:05 %usr
%sys
%wio
%idle
20:39:06 3
5
0
92
20:39:07 2
2
0
96
20:39:08 3
3
0
94
Average 3
3
0
94
Keeping the focus on system time and user time, the next command traces what processes are on the CPU, what user code they are running, and what kernel functions are running on the CPU (system time). Therefore, if we need to know what processes are running on the CPU, execute the following command:
root@solaris11-1:~# dtrace -n 'profile-993Hz {@[pid,execname] = count ();
}'
dtrace: description 'profile-993Hz ' matched 1 probe
^C
-
13
svc.configd
1
928
fmd
1
1817
gnome-settings-d
1
1824
nautilus
1
1847
updatemanagernot
1
1854
xscreensaver
1
1858
nwam-manager
1
1839
gnome-power-mana
2
849
VBoxService
3
1899
dtrace
3
1820
metacity
4
1829
wnck-applet
4
1821
gnome-panel
5
6
kmem_task
7
1873
clock-applet
7
3
fsflush
12
Configuring the Syslog and Monitoring Performance
-
1896
gnome-terminal
25
1162
Xorg
27
1840
java
94
0
sched
14985
In this case, the sched process (the Oracle Solaris scheduler) is taking most of the CPU's time. Additionally, the Java and Xorg processes also take a considerable amount of
the CPU's time.
To find which processes are taking more user time (to run the user code) from the CPU, execute the following command:
root@solaris11-1:~# dtrace -n 'profile-993hz /arg1/ { @[pid,execname] = count(); }'
dtrace: description 'profile-993hz ' matched 1 probe
^C
-
1919
dtrace
1
1152
Xorg
6
1892
gnome-terminal
9
1830
java
34
1901
firefox
49
According to the output, the firefox process takes more of the CPU's time.
Following the same line, it is feasible to obtain the top kernel functions that are on the CPU (the %sys time) by executing the following command:
root@solaris11-1:~# dtrace -n 'profile-993hz /arg0/ { @[func(arg0)] = count() ;}'
dtrace: description 'profile-993hz ' matched 1 probe
(truncated output)
genunix`fsflush_do_pages 14
unix`ddi_get32 31
unix`i86_monito 76
unix`cpu_idle_enter 125
unix`tsc_read 152
unix`dispatch_softint 263
unix`i86_mwait 24424
Chapter 9
The CPU saturation is also examined when managing processing through the
prstat command:
root@solaris11-1:~# prstat -mLc 1 1
Please wait...
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/ LWPID
-
2618
root
15
84
0.0
0.0
0.0
0.0
0.0
0.3
0
13
8K
0
prstat/1
1953 pkg5srv htcacheclean/1
0.2
1.4
0.0
0.0
0.0
0.0
98
0.4
19
0
147
0
2530
root
0.6
0.6
0.0
0.0
0.0
98
0.0
0.7
39
0
60
0
java/22
2530
root
0.4
0.4
0.0
0.0
0.0
99
0.0
0.6
20
0
40
0
java/21
2563
root
0.2
0.5
0.0
0.0
0.0
0.0
99
0.4
38
0
76
0
(truncated output)
In the preceding command, we can see a total of 120 processes; 830 lwps; and 0.12, 0.11,
0.13 load averages. The LAT (latency) column means the amount of time that processes are waiting for the CPU, and a constant value above 1 deserves a detailed investigation. If some process or thread has an inappropriate value, Oracle Solaris offers ways to delve into the details of the problem. For example, the java process presents a latency (LAT) value equal to 0.7 (this is a very low value, and it would not be worth investigating in a real case), but if we want to gather details about all its threads, execute the following command:
root@solaris11-1:~# prstat -mL -p 2530
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/ LWPID
-
2530
root
0.5
0.5
0.0
0.0
0.0
98
0.0
0.9
201
0
300
0
java/22
2530
root
0.4
0.4
0.0
0.0
0.0
99
0.0
0.5
100
0
201
0
java/21
2530
root
0.2
0.2
0.0
0.0
0.0
100
0.0
0.1
99
0
99
0
java/14
2530
root
0.0
0.0
0.0
0.0
0.0
0.0
100
0.0
10
0
30
0
java/17
(truncated output)
It would be possible to verify the stack for a particular thread by executing the following command:
root@solaris11-1:~# pstack 2530/22
2530: java -Djava.security.policy=/usr/share/vpanels/java.policy com. oracle.
----------------- lwp# 22 / thread# 22 --------------------
fe6893a5 lwp_cond_wait (8966640, 8966628, f527e9d8, 0)
fe65e2a4 _lwp_cond_timedwait (8966640, 8966628, f527ea20, fe10f9f1) + 37 fe10fd2d 1cGParkerEpark6Mbx_v_ (8966628) + 34d
fe272980
Unsafe_Park (8965d28, f527eab4, 0, 2e95966, 0, af8a18f0) + 208
fa2ce072
* *sun/misc/Unsafe.park(ZJ)V [compiled]
Configuring the Syslog and Monitoring Performance
fa330790 * *java/util/concurrent/locks/LockSupport.parkNanos(Ljava/lang/ Object;J)V [compiled] +21 (line 449)
fa330790 * *java/util/concurrent/locks/AbstractQueuedSynchronizer$Condit ionObject.awaitNanos(J)J+69 (line 4153)
fa330790 * *java/util/concurrent/DelayQueue.take()Ljava/util/concurrent/ Delayed;+133 (line 484)
00000000 ???????? (da647c20, da647c20, af8ac070, 0, 2d, ab5463b8) +
ffffffff01d8d888
ab7a4350 ???????? () + ffffffffad531bd8
The lwp_cond_wait and _lwp_cond_timedwait functions usually wait for the occurrence of a condition represented by an LWP condition variable. In this case, both are looking for a CPU.
Eventually, the mpstat command can help us distinguish the load among CPUs, as shown in the following command:
root@solaris11-1:~# mpstat 1 1
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 331 0 21 490 127 714 10 66 39 1 2286 5 12 0
84
1 386 0 17 236 36 609 9 60 38 1 1988 4 9 0
86
2 264 0 22 281 114 566 9 58 35 1 1817 4 11 0
85
3 299 0 16 227 37 669 9 65 37 1 1930 4 9 0
87
Here, minf (minor fault—pages that were not found on cache and were fetched on memory), mjf (major fault—pages that were not found on memory and were fetched on disk), xcal (cross call), and intr (number of interrupts received by the CPU). It is appropriate to say that cross-calls are calls between CPUs or cores that execute a specific low-level function.
Additionally, cross-calls are also necessary to keep the cache coherent due to a stale entry in a cache from a CPU. Usually, cross-calls are originated from a requirement of releasing memory as performed by functions such as kmen_free (). An interrupt (the intr column) is used by the kernel when it needs another processor to perform work on its behalf, such as preempting a dispatcher (a thread signal, a thread that runs on another processor to enter the kernel mode) to deliver a signal to interrupt a thread on another processor and to start/stop a /proc thread on a different processor. The mpstat command itself doesn't show us the performance bottleneck, but it helps us have a general understanding of a system, as shown earlier. For example, continuing the preceding example, it's possible to list how many cross calls a process executed by running the following command:
root@solaris11-1:~# dtrace -n 'sysinfo:::xcalls { @[execname] = count();
}'
Chapter 9
dtrace: description 'sysinfo:::xcalls ' matched 1 probe
^C
-
pargs
36
sched
2156
dtrace
2607
The procedure of running common monitor commands such as mpstat, sar, vmstat, and iostat followed by a detailed DTrace investigation is a typical approach to finding what is the offending application or process.
Now, we will change the focus to I/O performance. Perhaps the most fundamental command to analyze potential problems with the I/O is the iostat command:
root@solaris11-1:~# iostat -xnze 1
extended device statistics ---- errors
w/s |
kr/s |
kw/s |
wait |
actv |
wsvc_t |
asvc_t |
%w |
%b |
s/w |
h/w |
trn |
0.4 |
18.2 |
5.1 |
0.0 |
0.0 |
12.7 |
11.7 |
1 |
2 |
0 |
0 |
0 |
0.0 |
0.1 |
0.0 |
0.0 |
0.0 |
0.1 |
9.2 |
0 |
0 |
0 |
0 |
0 |
0.0 |
0.4 |
0.1 |
0.0 |
0.0 |
7.1 |
14.9 |
0 |
0 |
0 |
0 |
0 |
---
r/s
tot device
1.7
0 c7t0d0
0.0
0 c7t1d0
0.1
0 c7t2d0
(truncated output)
The output shows statistics for each disk. The used options are –n (uses logical names), -x (shows extended statistics), -e (shows error statistics), and –z (does not show lines without activity). Furthermore, some columns are very important, such as wait (average number of transactions that are in queue and waiting for the disk), actv (number of transactions being processed), wsvc_t (average time that a transaction spends on the I/O wait queue), and %b (percentage of time that the disk is active). From this explanation, the wait column deserves attention because it is a metric of disk saturation, and ideally, it should always be equal to zero.
A really good tool (from the DTraceToolkit) is the iotop.d script that prints I/O details ordered by processes and shows I/O sizes (BYTES column), as shown in the following command:
root@solaris11-1:~# cd /usr/dtrace/DTT/Bin/
root@solaris11-1:/usr/dtrace/DTT/Bin# ./iotop -PC
2014 May 22 05:18:15, load: 0.38, disk_r: 559 KB, disk_w: 4053 KB
Configuring the Syslog and Monitoring Performance
-
UID
PID
PPID
CMD
DEVICE
MAJ
MIN
D
BYTES
0
2768
1
firefox
sd0
217
1
R
572928
0
5
0
zpool-rpool
sd0
217
1
W
4282880
(truncated output)
We could remove the –P option to prevent the output from rolling and refreshing the screen.
Finally, we have to monitor network interfaces and look for network bottleneck, so there are good tools to accomplish this task. For example, the netstat command is a simple and effective command to gather network information and analyze if collision is happening, as shown in the following command:
root@solaris11-1:~# netstat -i 1
input net0 output input (Total) output
-
packets
errs
packets
errs
colls
packets
errs
packets
errs
colls
338712
0
180791
0
0
339832
0
181911
0
0
4
0
1
0
0
4
0
1
0
0
5
0
1
0
0
5
0
1
0
0
6
0
1
0
0
6
0
1
0
0
4
0
1
0
0
4
0
1
0
0
4
0
1
0
0
4
0
1
0
0
(truncated output)
There is another fantastic tool named nicstat that can help us find potential bottleneck on network. However, it is an external tool, and to install it is a bit convoluted. However, it is necessary to download the nicstat tool from http://sourceforge.net/projects/ nicstat/files/. Moreover, it would be nice to download the latest version (with more features), but we will need to compile it.
During this demonstration, I used the version from http://sourceforge.net/projects/ nicstat/files/latest/download?source=files (nicstat-src-1.95.tar.gz).
Once we download the package, we have to open it by running the following command:
root@solaris11-1:~/Downloads# tar zxvf nicstat-src-1.95.tar.gz
Nonetheless, we have a problem this time; it is necessary for a compiler to create the nicstat binary! Go to http://www.oracle.com/technetwork/server-storage/ solarisstudio/downloads/index-jsp-141149.html to get Oracle Solaris Studio 12.3, and click on http://pkg-register.oracle.com to download the version for Oracle Solaris 11 x86. From there, we will be requested to create personal SSL certificates to gain access to restricted repositories with packages such as Oracle Solaris Studio and Oracle Solaris Cluster. Therefore, click on the Request Certificate link. You will be redirected to
a page to download both the key and certificate. It is suggested that you save both in the
/root/Downloads directory.
Chapter 9
Oracle
Solaris Studio installs a very interesting tool named er_kernel
to profile only
the kernel or both the kernel and the load we are running. There is
more information (including examples) about the er_kernel
tool on
http://docs.oracle.com/cd/
E18659_01/html/821-1379/afahw.html.
The following steps are required to install both the key and certificate to include the new publisher (solarisstudio) in the system, to test if we're able to list the Oracle Solaris
Studio files, and then, to install the Oracle Solaris Studio, as shown in the following commands:
root@solaris11-1:~# mkdir -m 0755 -p /var/pkg/ssl
root@solaris11-1:~# cp ~/Downloads/Oracle_Solaris_Studio.key.pem /var/ pkg/ssl
root@solaris11-1:~# cp ~/Downloads/Oracle_Solaris_Studio.certificate.pem
/var/pkg/ssl
root@solaris11-1:~# pkg set-publisher -k /var/pkg/ssl/Oracle_Solaris_ Studio.key.pem -c /var/pkg/ssl/Oracle_Solaris_Studio.certificate.pem -G '*' -g https://pkg.oracle.com/solarisstudio/release solarisstudio
root@solaris11-1:~# pkg list -a pkg://solarisstudio/*
NAME (PUBLISHER) VERSION
IFO
developer/solarisstudio-122 (solarisstudio) 12.2-1.0.0.0
---
developer/solarisstudio-122/analyzer (solarisstudio) 12.2-1.0.0.0
---
(truncated output)
root@solaris11-1:~# pkg install solarisstudio-123
Packages to install: 24 Create boot environment: No
Create backup boot environment: No
(truncated output)
DOWNLOAD PKGS FILES XFER (MB) SPEED
Configuring the Syslog and Monitoring Performance
Completed 24/24 9913/9913 457.1/457.1 301k/s
PHASE ITEMS
Installing new actions 15563/15563 Updating package state database Done
Updating image state Done
Creating fast lookup database Done
This is nice! As the Oracle Solaris Studio is installed out of the system's executable path, we have to include it in the PATH variable by running the following commands:
root@solaris11-1:~/Downloads# cd
root@solaris11-1:~# echo PATH=$PATH:/opt/solarisstudio12.3/bin >> /root/. profile
root@solaris11-1:~# echo export PATH >> /root/.profile
root@solaris11-1:~# . ./.profile
Return to the nicstat directory and compile it by executing the following commands:
root@solaris11-1:~# cd /root/Downloads/nicstat-src-1.95
root@solaris11-1:~/Downloads/nicstat-src-1.95# cp Makefile.Solaris Makefile
root@solaris11-1:~/Downloads/nicstat-src-1.95# make
root@solaris11-1:/tmp/nicstat-src-1.95# cp .nicstat.Solaris_11_i386 nicstat
root@solaris11-1:~/Downloads/nicstat-src-1.95# file nicstat
nicstat: ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, not stripped
Finally, we can use the fantastic nicstat tool! First, list the available interfaces using the
nicstat tool, as shown in the following command:
root@solaris11-1:~/Downloads/nicstat-src-1.95# ./nicstat –l
Int Loopback Mbit/s Duplex State lo0 Yes - unkn up
net0 No 1000 full up vswitch1 No 0 unkn down vnic0 No 40000 unkn up
vnic1 No 40000 unkn up
vnic2 No 40000 unkn up
Chapter 9
The nicstat tool has several options, and they are listed by running the following command:
root@solaris11-1:~/Downloads/nicstat-src-1.95# ./nicstat -h
USAGE: nicstat [-hvnsxpztualMU] [-i int[,int...]] [interval [count]]
-h # help
-v # show version (1.95)
-i interface # track interface only
-n # show non-local interfaces only (exclude
lo0)
eg,
-s # summary output
-x # extended output
-p # parseable output
-z # skip zero value lines
-t # show TCP statistics
-u # show UDP statistics
-a # equivalent to "-x -u -t"
-l # list interface(s)
-M # output in Mbits/sec
-U # separate %rUtil and %wUtil
nicstat # print summary since boot only
nicstat 1 # print every 1 second
nicstat 1 5 # print 5 times only
nicstat -z 1 # print every 1 second, skip zero lines nicstat -i hme0 1 # print hme0 only every 1 second
Based on the available options, the following command brings us an extended output, without zeroed lines and separate %rUtil and %wUtil columns:
RdKB |
WrKB |
RdPkt |
WrPkt |
IErr |
OErr |
Coll |
NoCP |
Defer |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
root@solaris11-1:~/Downloads/nicstat-src-1.95# ./nicstat -zUx 1
14:41:16
%rUtil %wUtil
lo0
0.00 0.00
net0
0.00 0.00
Configuring the Syslog and Monitoring Performance
RdKB |
WrKB |
RdPkt |
WrPkt |
IErr |
OErr |
Coll |
NoCP |
Defer |
0.22 |
0.10 |
3.00 |
1.00 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
RdKB |
WrKB |
RdPkt |
WrPkt |
IErr |
OErr |
Coll |
NoCP |
Defer |
0.28 |
0.09 |
3.96 |
0.99 |
0.00 |
0.00 |
0.00 |
0.00 |
0.00 |
14:41:17
%rUtil %wUtil
net0
0.00 0.00
14:41:18
%rUtil %wUtil
net0
0.00 0.00
^C
The most important columns from the nicstat tool are rAvs (average size of packets received), wAvs (average size of packets transmitted), %Util (maximum utilization of the interface), and Sat (errors per second seen for the interface, and this can be a clue that the interface might be approaching saturation).
At the end, administrators can gather statistics from a specific network interface by running
the following command:
root@solaris11-1:~# dladm show-link -s -i 1 net0
-
LINK OERRORS
IPACKETS
RBYTES IERRORS
OPACKETS
OBYTES
net0
365079
488320923 0
190053
16047649
0
net0
9
1044 0
4
591
0
net0
6
446 0
3
278
0
net0
7
538 0
1
98
0
(truncated output)
In the preceding command, the columns have the following meaning:
f LINK: This refers to the name of the data link
f IPACKETS: Number of packets received on this link
f RBYTES: Number of bytes received on this link
f IERRORS: Number of input errors
f OPACKETS: Number of packets sent on this link
f OBYTES: Number of bytes sent on this link
f OERRORS: Number of output errors
An overview of the recipe
This chapter explained how to configure the syslog framework to record messages and events from the system. Additionally, we gave you a brief introduction to monitoring the performance of the Oracle Solaris 11 system using several commands such as vmstat, sar, prstat, kstat, mdb, iostat, and so on. We also used other tools such as DTrace and DTraceToolkit scripts to measure the performance on the Oracle Solaris 11 system.
Chapter 9