Sysadmin's Shouts!

a blog for sysadmin's rants and raves…

6.- Advanced logrotate for AIX

Leave a comment

The most powerful facilities provided by logrotate are prerotate,postrotate & endscript, and we can make good use of this facilities to employ “complex” log rotation schedules.

We will do a logrotate setup to perform log rotation following IBM recommendations for AIX v7.2, and for that we can resort to the following KBs in IBM Support Knowledgecenter:

/ (root) overflow
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsrootover.htm

Resolving overflows in the /var file system
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsvarover.htm

6.1.- logrotate failedlogin

IBM states in the official documentation, that failedlogin is a binary log, and therefore the utility who must be used to see it’s entries, as follows:

[root@aix72:/]who /etc/security/failedlogin
root vty0 Oct 20 18:03
UNKNOWN_ vty0 Oct 20 18:03
UNKNOWN_ ssh Oct 26 10:22 (10.1.15.12)
root ssh Oct 26 10:22 (10.1.15.12)
root ssh Nov 03 10:38 (10.20.30.129)
root ssh Nov 05 11:49 (srv.dom.myanet)
root ssh Nov 12 11:03 (tsmsrv)
root vty0 Nov 19 07:47
root ssh Nov 21 10:25 (10.20.120.72)
root ssh Jan 25 05:37 (10.20.130.229)
tsminst1 ssh Feb 13 10:03 (loopback)
tsminst1 ssh Feb 13 10:05 (loopback)
root ssh Feb 27 16:45 (10.1.15.214)
root pts/2 Mar 07 15:11 (10.1.165.159)
UNKNOWN_ pts/2 Mar 07 15:11 (10.1.165.159)
...

So we check the file and it’s permissions:

[root@aix72:/]ls -l /etc/security/failedlogin
-rw-rw---- 1 root system 14256 Mar 07 15:15 /etc/security/failedlogin

And we take note of the 660 access rights and user root group system.

Well, we can see that this is an interesting log to keep (and also a log that can grow large in size if there is a problem and we have a lot of terminal users), so using logrotate we can just rotate it by size, say 5 MB each log, keep 3 copies (keep the original log and rotate 2 more versions) and also keep it compressed, so we write the following file: /etc/logrotate.d/failedlogin

# logrotate config for failedlogin which logs failed login sessions in binary form, can be used to detect brute-force attacks. Read with "who".
/etc/security/failedlogin {
  size 5M
  compress
  rotate 2
  create 660 root system
}

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/failedlogin
reading config file /etc/logrotate.d/failedlogin

Handling 1 logs

rotating pattern: /etc/security/failedlogin forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
log does not need rotating

NOTE: If we want to force this rotation, then we could change the size 5M for size 10k, in which case the file will be rotated as it is 14k in size.

6.2.- logrotate wtmp

IBM states in the official documentation, that wtmp is a binary log, and therefore IBM recommends using the utility fwtmp to convert the binary log to an ASCII log, as follows:

Export the wtmp log to an ASCII copy called /tmp/wtmp-delete.me:

[root@aix72:/]/usr/sbin/acct/fwtmp < /var/adm/wtmp > /tmp/wtmp-delete.me

Now we can just read the file /tmp/wtmp-delete.me with vi, emacs, nano, or whichever editor we fancy:

[root@aix72:/]cat /tmp/wtmp-delete.me
clcomd   clcomd                       5 7733518 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
xmdaily  xmdaily                      5 8192296 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
ctrmc    ctrmc                        5 10944850 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
spectrum spectrum                     5 11075926 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
tsmcc    tsmcc                        5 11141464 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
ha_star  ha_star                      5 11010388 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
         pts/1          pts/1         8 8454482 0000 0000 1488245570                                  Mon Feb 27 19:32:50 CST 2017
root     pts/0          pts/0         7 15532358 0000 0000 1488305060 10.1.165.159                    Tue Feb 28 12:04:20 CST 2017
root     pts/1          pts/1         7 15860100 0000 0000 1488305690 10.1.165.159                    Tue Feb 28 12:14:50 CST 2017
root     pts/2          pts/2         7 15663394 0000 0000 1488323898 10.1.165.159                    Tue Feb 28 17:18:18 CST 2017
 ...

This log tells us all valid terminal logins (loging, rlogins and telnet), date and time, which terminal they used and their remote IP. So it is also an interesting log to keep. (and can also be read by who, try ” who /var/adm/wtmp “)

Now to check the file and it’s permissions:

[root@aix72:/]ls -l /var/adm/wtmp
 -rw-rw-r-- 1 adm adm 846288 Mar 07 15:12 /var/adm/wtmp

Aha, this one has different attributes from the one on the step 6.1, we take note of the 664 access rights and user adm group adm.

NOTE: If we wanted to process wtmp log, say to keep the last 1000 lines, and truncating the rest, we could process the ASCII log and convert back to the /var/adm/wtmp binary log using the utility fwtmp like this:

/usr/bin/tail -1000 /tmp/wtmp-delete.me | /usr/sbin/acct/fwtmp -ic > /var/adm/wtmp

Not forgetting to cleanup the temp file afterwards:

rm /tmp/wtmp-delete.me

This used to be a common practice to keep wtmp log under control in AIX, but this is not really needed anymore, as using logrotate we can just rotate wtmp by size, say 5 MB each log, keep 2 copies (keep the original log and rotate 1 more version), so we write the following file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 size 5M
 rotate 1
 create 664 adm adm
 }

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 considering log /var/adm/wtmp
 error: skipping "/var/adm/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Hey!!!, this time we have an error! and this is because the parent directory group is different from root (so it expects to be told the right ones to use on the “su” directive ), so we check this and change our config file accordingly, and we’ll try again:

[root@aix72:/home/admin]ls -la /var/adm
 total 4960
 drwxrwxr-x 15 root adm 4096 Mar 03 19:25 . <-- it's owned by root/adm

Add the su directive to the config file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 su root adm
 size 5M
 rotate 2
 create 664 adm adm
 }
[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 switching euid to 4 and egid to 4
 considering log /var/adm/wtmp
 log does not need rotating
 switching euid to 0 and egid to 0

This time, it has worked fine, and also it tells us that it has switched user to userid 0 / groupid 4 and then back to the original (0/0). So now we know more things about logrotate’s output. Good.

Now we also know that we need to check the owners on the log’s parent directory as well in case we need to add the su directive, and we will from now on.

6.3.- logrotate errlog6.3.- logrotate errlog

Errlog is a binary circular log, and therefore cannot really be rotated. So for this log in particular is not recommended to be managed using logrotate.

By default, root’s crontab comes with the following 2 entries to trim old HW errors:

0 11 * * * /usr/bin/errclear -d S,O 30 <-- delete Software & errlogger msgs (Other) older than 30 days
0 12 * * * /usr/bin/errclear -d H 90   <-- delete Hardware errors older than 90 days

Error messages get overwritten by new ones as they are generated, and there are system admins that do not even use errclear to trim the log, since they prefer to keep an historical of all theHW errors ever generated on one server.

However in our days with virtualized hardware, it’s not that relevant to keep all errors ever registered, and I find a good practice to enable the default errclear entries on crontab, as well as skulker to clean old log & temp system files.

Errlog can however be increased if need be, to see the actual errlog size:

[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes				<-- 1MB default value
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

To increase the size:

[root@aix72:/]/usr/lib/errdemon -s 2097152
[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                2097152 bytes				<-- increased to 2MB
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

NOTE: The errlog daemon ( /usr/lib/errdemon ) can be stopped using the command  ( /usr/lib/errstop ), and one VERY important thing to take into account is that the errlog should not be zeroed (a procedure often used to clear logs), otherwise the daemon would not start. So, let’s try it to see what would happen :o)

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog /var/adm/ras/errlog.bak <-- we do a backup of it first!!!

[root@aix72:/var/adm/ras]> /var/adm/ras/errlog       <-- we zero the file

[root@aix72:/var/adm/ras]/usr/lib/errdemon	     <-- and we try to start the errdemon...
0315-180 logread: UNEXPECTED EOF		     <-- the only entry in the log is the End Of File (it is a zero file)
0315-171 Unable to process the error log file /var/adm/ras/errlog.
errdemon:
0315-001 Failure to open the logfile '/var/adm/ras/errlog' for writing.
Possible causes are:
1. The file exists but the invoking process does not have write
   permission.
2. The file exists but the directory '/var/adm/ras' does not have write
   permission.
3. The file exists but it is not a valid error logfile.  Remove

4. The file does exist and the directory ‘/var/adm/ras’ does not have enough
space available. The minimum logfile size is 8192 bytes.

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog.bak /var/adm/ras/errlog <– restore the backup to be able to start the errdemon again

6.4.- logrotate sulog

Now we will do rotation for sudo’s log ( sulog ):

/var/adm/sulog

We do file, permission & parent directory checks :

[root@aix72:/]ls -l /var/adm/sulog
-rw-------    1 root     system         4693 Mar 03 18:24 /var/adm/sulog
[root@aix72:/]ls -la /var/adm/.
total 5208
drwxrwxr-x   15 root     adm            4096 Mar 20 22:36 .

It has 600 access rights, user root group system, and parent directory has user root group adm, so we will have to use the “su” directive

# logrotate config for sudo's log (sulog).
/var/adm/sulog {    
  su root adm    
  size 5M    
  compress    
  rotate 2    
  create 600 root system
}

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/sulog
reading config file /etc/logrotate.d/sulog
Handling 1 logs
rotating pattern: /var/adm/sulog  forced from command line (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/adm/sulog 
 log needs rotating
rotating log /var/adm/sulog, log->rotateCount is 2
dateext suffix '-20170320'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/adm/sulog.2.gz to /var/adm/sulog.3.gz (rotatecount 2, logstart 1, i 2),
old log /var/adm/sulog.2.gz does not exist
renaming /var/adm/sulog.1.gz to /var/adm/sulog.2.gz (rotatecount 2, logstart 1, i 1),
old log /var/adm/sulog.1.gz does not exist
renaming /var/adm/sulog.0.gz to /var/adm/sulog.1.gz (rotatecount 2, logstart 1, i 0),
old log /var/adm/sulog.0.gz does not exist
log /var/adm/sulog.3.gz doesn't exist -- won't try to dispose of it
renaming /var/adm/sulog to /var/adm/sulog.1
creating new /var/adm/sulog mode = 0600 uid = 0 gid = 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 4
switching euid to 0 and egid to 0

6.5.- logrotate syslog

Syslog is an AIX classic, and the only log churner that got upgraded a while ago and has it’s own “decent” log rotation & compression configuration since at least AIX version 6.1, therefore the recommended log rotation method is to use syslog’s own configuration file and not logrotate’s.

For the official documentation on syslog from AIX v7.2, you can refer to:
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds5/syslogd.htm

To apply a weekly logrotation, with 5 log versions, and compression enabled, just edit the /etc/syslog.conf file and modify the entry *.info for the following:

 *.info /var/log/syslog.log rotate time 1w files 5 compress     #weekly rotation, 5 files, compressed

Just remember to refresh the daemon after making the modifications in the config file:

[root@aix72:/] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.

Well chaps, I think that it is now enough logrotate for AIX now. There are examples of common logrotate configs that I will like to post on the future, but for now, I think that this is a good end of series for Logrotate.

AS always: Thanx for reading!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s