Sysadmin's Shouts!

a blog for sysadmin's rants and raves…


Leave a comment

6.- Advanced logrotate for AIX

The most powerful facilities provided by logrotate are prerotate,postrotate & endscript, and we can make good use of this facilities to employ “complex” log rotation schedules.

We will do a logrotate setup to perform log rotation following IBM recommendations for AIX v7.2, and for that we can resort to the following KBs in IBM Support Knowledgecenter:

/ (root) overflow
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsrootover.htm

Resolving overflows in the /var file system
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsvarover.htm

6.1.- logrotate failedlogin

IBM states in the official documentation, that failedlogin is a binary log, and therefore the utility who must be used to see it’s entries, as follows:

[root@aix72:/]who /etc/security/failedlogin
root vty0 Oct 20 18:03
UNKNOWN_ vty0 Oct 20 18:03
UNKNOWN_ ssh Oct 26 10:22 (10.1.15.12)
root ssh Oct 26 10:22 (10.1.15.12)
root ssh Nov 03 10:38 (10.20.30.129)
root ssh Nov 05 11:49 (srv.dom.myanet)
root ssh Nov 12 11:03 (tsmsrv)
root vty0 Nov 19 07:47
root ssh Nov 21 10:25 (10.20.120.72)
root ssh Jan 25 05:37 (10.20.130.229)
tsminst1 ssh Feb 13 10:03 (loopback)
tsminst1 ssh Feb 13 10:05 (loopback)
root ssh Feb 27 16:45 (10.1.15.214)
root pts/2 Mar 07 15:11 (10.1.165.159)
UNKNOWN_ pts/2 Mar 07 15:11 (10.1.165.159)
...

So we check the file and it’s permissions:

[root@aix72:/]ls -l /etc/security/failedlogin
-rw-rw---- 1 root system 14256 Mar 07 15:15 /etc/security/failedlogin

And we take note of the 660 access rights and user root group system.

Well, we can see that this is an interesting log to keep (and also a log that can grow large in size if there is a problem and we have a lot of terminal users), so using logrotate we can just rotate it by size, say 5 MB each log, keep 3 copies (keep the original log and rotate 2 more versions) and also keep it compressed, so we write the following file: /etc/logrotate.d/failedlogin

# logrotate config for failedlogin which logs failed login sessions in binary form, can be used to detect brute-force attacks. Read with "who".
/etc/security/failedlogin {
  size 5M
  compress
  rotate 2
  create 660 root system
}

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/failedlogin
reading config file /etc/logrotate.d/failedlogin

Handling 1 logs

rotating pattern: /etc/security/failedlogin forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
log does not need rotating

NOTE: If we want to force this rotation, then we could change the size 5M for size 10k, in which case the file will be rotated as it is 14k in size.

6.2.- logrotate wtmp

IBM states in the official documentation, that wtmp is a binary log, and therefore IBM recommends using the utility fwtmp to convert the binary log to an ASCII log, as follows:

Export the wtmp log to an ASCII copy called /tmp/wtmp-delete.me:

[root@aix72:/]/usr/sbin/acct/fwtmp < /var/adm/wtmp > /tmp/wtmp-delete.me

Now we can just read the file /tmp/wtmp-delete.me with vi, emacs, nano, or whichever editor we fancy:

[root@aix72:/]cat /tmp/wtmp-delete.me
clcomd   clcomd                       5 7733518 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
xmdaily  xmdaily                      5 8192296 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
ctrmc    ctrmc                        5 10944850 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
spectrum spectrum                     5 11075926 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
tsmcc    tsmcc                        5 11141464 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
ha_star  ha_star                      5 11010388 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
         pts/1          pts/1         8 8454482 0000 0000 1488245570                                  Mon Feb 27 19:32:50 CST 2017
root     pts/0          pts/0         7 15532358 0000 0000 1488305060 10.1.165.159                    Tue Feb 28 12:04:20 CST 2017
root     pts/1          pts/1         7 15860100 0000 0000 1488305690 10.1.165.159                    Tue Feb 28 12:14:50 CST 2017
root     pts/2          pts/2         7 15663394 0000 0000 1488323898 10.1.165.159                    Tue Feb 28 17:18:18 CST 2017
 ...

This log tells us all valid terminal logins (loging, rlogins and telnet), date and time, which terminal they used and their remote IP. So it is also an interesting log to keep. (and can also be read by who, try ” who /var/adm/wtmp “)

Now to check the file and it’s permissions:

[root@aix72:/]ls -l /var/adm/wtmp
 -rw-rw-r-- 1 adm adm 846288 Mar 07 15:12 /var/adm/wtmp

Aha, this one has different attributes from the one on the step 6.1, we take note of the 664 access rights and user adm group adm.

NOTE: If we wanted to process wtmp log, say to keep the last 1000 lines, and truncating the rest, we could process the ASCII log and convert back to the /var/adm/wtmp binary log using the utility fwtmp like this:

/usr/bin/tail -1000 /tmp/wtmp-delete.me | /usr/sbin/acct/fwtmp -ic > /var/adm/wtmp

Not forgetting to cleanup the temp file afterwards:

rm /tmp/wtmp-delete.me

This used to be a common practice to keep wtmp log under control in AIX, but this is not really needed anymore, as using logrotate we can just rotate wtmp by size, say 5 MB each log, keep 2 copies (keep the original log and rotate 1 more version), so we write the following file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 size 5M
 rotate 1
 create 664 adm adm
 }

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 considering log /var/adm/wtmp
 error: skipping "/var/adm/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Hey!!!, this time we have an error! and this is because the parent directory group is different from root (so it expects to be told the right ones to use on the “su” directive ), so we check this and change our config file accordingly, and we’ll try again:

[root@aix72:/home/admin]ls -la /var/adm
 total 4960
 drwxrwxr-x 15 root adm 4096 Mar 03 19:25 . <-- it's owned by root/adm

Add the su directive to the config file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 su root adm
 size 5M
 rotate 2
 create 664 adm adm
 }
[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 switching euid to 4 and egid to 4
 considering log /var/adm/wtmp
 log does not need rotating
 switching euid to 0 and egid to 0

This time, it has worked fine, and also it tells us that it has switched user to userid 0 / groupid 4 and then back to the original (0/0). So now we know more things about logrotate’s output. Good.

Now we also know that we need to check the owners on the log’s parent directory as well in case we need to add the su directive, and we will from now on.

6.3.- logrotate errlog6.3.- logrotate errlog

Errlog is a binary circular log, and therefore cannot really be rotated. So for this log in particular is not recommended to be managed using logrotate.

By default, root’s crontab comes with the following 2 entries to trim old HW errors:

0 11 * * * /usr/bin/errclear -d S,O 30 <-- delete Software & errlogger msgs (Other) older than 30 days
0 12 * * * /usr/bin/errclear -d H 90   <-- delete Hardware errors older than 90 days

Error messages get overwritten by new ones as they are generated, and there are system admins that do not even use errclear to trim the log, since they prefer to keep an historical of all theHW errors ever generated on one server.

However in our days with virtualized hardware, it’s not that relevant to keep all errors ever registered, and I find a good practice to enable the default errclear entries on crontab, as well as skulker to clean old log & temp system files.

Errlog can however be increased if need be, to see the actual errlog size:

[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes				<-- 1MB default value
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

To increase the size:

[root@aix72:/]/usr/lib/errdemon -s 2097152
[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                2097152 bytes				<-- increased to 2MB
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

NOTE: The errlog daemon ( /usr/lib/errdemon ) can be stopped using the command  ( /usr/lib/errstop ), and one VERY important thing to take into account is that the errlog should not be zeroed (a procedure often used to clear logs), otherwise the daemon would not start. So, let’s try it to see what would happen :o)

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog /var/adm/ras/errlog.bak <-- we do a backup of it first!!!

[root@aix72:/var/adm/ras]> /var/adm/ras/errlog       <-- we zero the file

[root@aix72:/var/adm/ras]/usr/lib/errdemon	     <-- and we try to start the errdemon...
0315-180 logread: UNEXPECTED EOF		     <-- the only entry in the log is the End Of File (it is a zero file)
0315-171 Unable to process the error log file /var/adm/ras/errlog.
errdemon:
0315-001 Failure to open the logfile '/var/adm/ras/errlog' for writing.
Possible causes are:
1. The file exists but the invoking process does not have write
   permission.
2. The file exists but the directory '/var/adm/ras' does not have write
   permission.
3. The file exists but it is not a valid error logfile.  Remove

4. The file does exist and the directory ‘/var/adm/ras’ does not have enough
space available. The minimum logfile size is 8192 bytes.

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog.bak /var/adm/ras/errlog <– restore the backup to be able to start the errdemon again

6.4.- logrotate sulog

Now we will do rotation for sudo’s log ( sulog ):

/var/adm/sulog

We do file, permission & parent directory checks :

[root@aix72:/]ls -l /var/adm/sulog
-rw-------    1 root     system         4693 Mar 03 18:24 /var/adm/sulog
[root@aix72:/]ls -la /var/adm/.
total 5208
drwxrwxr-x   15 root     adm            4096 Mar 20 22:36 .

It has 600 access rights, user root group system, and parent directory has user root group adm, so we will have to use the “su” directive

# logrotate config for sudo's log (sulog).
/var/adm/sulog {    
  su root adm    
  size 5M    
  compress    
  rotate 2    
  create 600 root system
}

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/sulog
reading config file /etc/logrotate.d/sulog
Handling 1 logs
rotating pattern: /var/adm/sulog  forced from command line (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/adm/sulog 
 log needs rotating
rotating log /var/adm/sulog, log->rotateCount is 2
dateext suffix '-20170320'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/adm/sulog.2.gz to /var/adm/sulog.3.gz (rotatecount 2, logstart 1, i 2),
old log /var/adm/sulog.2.gz does not exist
renaming /var/adm/sulog.1.gz to /var/adm/sulog.2.gz (rotatecount 2, logstart 1, i 1),
old log /var/adm/sulog.1.gz does not exist
renaming /var/adm/sulog.0.gz to /var/adm/sulog.1.gz (rotatecount 2, logstart 1, i 0),
old log /var/adm/sulog.0.gz does not exist
log /var/adm/sulog.3.gz doesn't exist -- won't try to dispose of it
renaming /var/adm/sulog to /var/adm/sulog.1
creating new /var/adm/sulog mode = 0600 uid = 0 gid = 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 4
switching euid to 0 and egid to 0

6.5.- logrotate syslog

Syslog is an AIX classic, and the only log churner that got upgraded a while ago and has it’s own “decent” log rotation & compression configuration since at least AIX version 6.1, therefore the recommended log rotation method is to use syslog’s own configuration file and not logrotate’s.

For the official documentation on syslog from AIX v7.2, you can refer to:
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds5/syslogd.htm

To apply a weekly logrotation, with 5 log versions, and compression enabled, just edit the /etc/syslog.conf file and modify the entry *.info for the following:

 *.info /var/log/syslog.log rotate time 1w files 5 compress     #weekly rotation, 5 files, compressed

Just remember to refresh the daemon after making the modifications in the config file:

[root@aix72:/] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.

Well chaps, I think that it is now enough logrotate for AIX now. There are examples of common logrotate configs that I will like to post on the future, but for now, I think that this is a good end of series for Logrotate.

AS always: Thanx for reading!

Advertisements


Leave a comment

Logrotate 4 & 5.- Support & Common Errors

NOTE:  This is a follow-up, from the previous post: Logrotate 3.- Logrotate checks

4.- Logrotate Support

Disclaimer (IBM Unsupported):  IBM stand on opensource utilities is that they are not directly supported by IBM, this is IBM Support’s page for logrotate (dated 06 June 2011):

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012796

So, IBM will not provide any PMR Support on Open Source Software (and this is completely logical, as it’s not an IBM product), but still, you can get community based support at the developerWorks pages, and for this forum-based support, you can go to:

IBMDeveloperWorks: Forum Directory >‎ dW >‎ AIX and UNIX >‎ Forum: AIX Open Source Software

And in that forum, exists an specific YUM topic:

IBMDeveloperWorks: Forum Directory >‎ dW >‎ AIX and UNIX >‎ Forum: AIX Open Source Software >‎ Topic: yum for AIX Toolbox

5.- Fixing logrotate errors

5.1.- config file logrotate.conf errors

[root@aix72:/home/admin]logrotate -vf /etc/logrotate.conf
error: cannot stat /etc/logrotate.conf: A file or directory in the path name does not exist.

Cannot stat means that the config file is NOT FOUND, so revise that /etc/logrotate exists and has the right access rights (if it doesn’t, look for it in /opt/freeware/etc and copy it to /etc)

5.2.- config directory logrotate.d errors

[root@aix72:/home/admin]logrotate -vf /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
error: cannot stat /etc/logrotate.d: A file or directory in the path name does not exist.
removing last 0 log configs

Cannot stat means that the config directory is NOT FOUND, so revise that /etc/logrotate.d exists and has the right access rights (if it doesn’t, look for it in /opt/freeware/etc and copy it to /etc)

5.3.- files in directory logrotate.d errors

[root@aix72:/opt/freeware/etc]logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file yum
error: yum:6 unknown group 'root'
error: found error in /var/log/yum.log , skipping
removing last 1 log configs
error: /etc/logrotate.conf:23 unknown group 'utmp'
error: found error in /var/log/wtmp , skipping
removing last 1 log configs
error: /etc/logrotate.conf:31 unknown group 'utmp'
error: found error in /var/log/btmp , skipping
removing last 1 log configs

Handling 0 logs

This errors are usually caused at installation time of logrotate in AIX, since the config files require some modifications:

error: yum:6 unknown group 'root'
error: found error in /var/log/yum.log , skipping

It complains against the line 6 of /etc/logrotate.d/yum file, since in AIX there isn’t a “root” group, it is “system“, so modify the file:

/var/log/yum.log {  
  missingok 
  notifempty 
  size 30k 
  yearly 
  create 0600 root root 
}

for the file:

/var/log/yum.log {
  missingok
  notifempty
  size 30k
  yearly
  create 0600 root system
}
error: /etc/logrotate.conf:23 unknown group 'utmp' 
error: found error in /var/log/wtmp , skipping

It complains against the line 23 of /etc/logrotate.conf file, since in AIX there isn’t a “utmp” group, and in fact wtmp is not located in /var/log/wtmp, but in /var/adm/wtmp but in any case, we can just refer to the steps in 2.1 to fix it by deleting the wtmp lines in /etc/logrotate.conf.

error: /etc/logrotate.conf:31 unknown group 'utmp'
error: found error in /var/log/btmp , skipping

It complains against the line 31 of /etc/logrotate.conf file, since in AIX there isn’t a “utmp” group, and in fact AIX does not have a btmp, so we can just refer to the steps in 2.1 to fix it by deleting the wtmp lines in /etc/logrotate.conf.

 

That covers the most common Logrotate config errors in AIX. I’m sure that you will find some more obscure ones to entertain yourself with, as it is often the case!

On the next post, it will be time for step 6.- Advanced Logrotate for AIX.  See you then, and thanks for reading!


1 Comment

Logrotate 3.- Logrotate checks

NOTE:  This is a follow-up, from the previous post:  Logrotate 2.- Configure logrotate for AIX

To check that logrotate is configured and working OK, all we need to do is call logrotate from the command line telling it to verbose it’s internal checks ( -v ) and to check the config file ( /etc/logrotate.conf ), like the following:

[root@aix72:/home/admin]/usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d      
reading config file failedlogin 
reading config file sysadmin    
reading config file wtmp        
reading config file yum         

Handling 6 logs

rotating pattern: /etc/security/failedlogin 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
 log does not need rotating        

rotating pattern: /home/admin/log/check_all.log 1048576 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log 
 log does not need rotating         

rotating pattern: /var/adm/wtmp 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 4 and egid to 4
considering log /var/adm/wtmp 
 log does not need rotating   
switching euid to 0 and egid to 0

rotating pattern: /var/log/yum.log yearly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log 
 log does not need rotating      

Once we have checked that the config is OK, we can check the rotation by Forcing rotation with the -f or –force flag:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.conf
 reading config file /etc/logrotate.conf
 including /etc/logrotate.d
 reading config file failedlogin
 reading config file sysadmin
 reading config file wtmp
 reading config file yum
 Handling 6 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/stop_all.log
 log does not need rotating

rotating pattern: /var/log/yum.log forced from command line (4 rotations)
 empty log files are not rotated, old logs are removed
 considering log /var/log/yum.log log needs rotating rotateCount is 4
 dateext suffix '-20170226'
 glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
 glob finding old rotated logs failed
 renaming /var/log/yum.log to /var/log/yum.log-20170226
 creating new /var/log/yum.log mode = 0600 uid = 0 gid = 0

Logrotate is configured OK and it seems to work fine, so if it’s not executing properly, we will have to check it’s schedule on the crontab.

NOTE: Notice that when we configure the rotation to be on size, the –force option cannot force this rotation, so to force rotation on stanzas where size has been used, just lower the size attribute temporarily (size 10k instead of 5M, for example).

3.1- Logrotate individual files/logs check

To check logrotate’s config for a particular file, we will have to identify it first in the /etc/logrotate.d directory, for example to check the config for yum’s logs:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/yum
reading config file /etc/logrotate.d/yum

Handling 1 logs

rotating pattern: /var/log/yum.log forced from command line (no old logs will be kept)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log
 log does not need rotating

To check the config for a specific log, but we don’t see a logrotate file stored by its name in /etc/logrotate.d, we will have to dig it out (for example let’s look for start_all.log):

[root@aix72:/home/admin]grep start_all.log /etc/logrotate.d/*
/etc/logrotate.d/sysadmin:/home/admin/log/start_all.log

OK, so it looks like the logrotate config for start_all.log resides in the /etc/logrotate.d/sysadmin file, so now we can check it:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/sysadmin
reading config file /etc/logrotate.d/sysadmin

Handling 3 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations) 
empty log files are rotated, old logs are removed
considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/stop_all.log
 log does not need rotating

So, as always, an important part of a configuration (the most important, actually) is to check that our new config works just as we expected it.

And now we have seen how to check all the logrotate config, how to force the log rotation, and how to check individual logrotate config files, so with this three checks we should be able to perform config-test-change-retest until our friend logrotate does what we expect it to.

On the step 4, I will talk about logrotate documentation & support, and step 5 will show how to fix common logrotate errors. See you soon.


1 Comment

Logrotate 2.- Configure logrotate for AIX

NOTE:  This is a follow-up, from the previous post: AIX 6L+ , AIX 7DevOps and Logrotate on AIX

Logrotate is a utility from RHEL, and therefore it comes preconfigured for RHEL & fedora, so after installing it using yum, we need to adapt it to work in our AIX system.

2.1- Fix logrotate.conf invalid entries

By default, logrotate’s main config file treats logs of wtmp & btmp, but since we can treat wtmp separately, and btmp is not implemented in AIX, we can just comment out or better still, delete those lines from /etc/logrotate.conf:

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.conf
 # see "man logrotate" for details
 # rotate log files weekly
 weekly

# keep 4 weeks worth of backlogs
 rotate 4

# create new (empty) log files after rotating old ones
 create

# use date as a suffix of the rotated file
 dateext

# uncomment this if you want your log files compressed
 #compress

# RPM packages drop log rotation information into this directory
 include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here      
/var/log/wtmp {
 monthly
 create 0664 root utmp
 minsize 1M
 rotate 1
 }

/var/log/btmp {
 missingok
 monthly
 create 0600 root utmp
 rotate 1
 }

NOTE: There is also a good idea to put a line like the following to the bottom of /etc/logrotate.conf to sepparate the default system config from future additions:

# Installed by Carlos Ijalba, 2017. Put new generic logconfigs below this line: ##########

 

2.2- Fix the log rotation for yum

By default, logrotate comes configured to treat yum logs, but we need to change the owner group of the yum logs in RHEL (root) for AIX default system group (system), so we edit the file /etc/logrotate.d/yum, and change line 6 last root entry for system:

[root@aix72:/etc/logrotate.d]cat /etc/logrotate.d/yum
 /var/log/yum.log {
 missingok
 notifempty
 size 30k
 yearly
 create 0600 root system
 }

2.3- Setup logrotate schedule in crontab

And the last step, will be to configure the contab entry for logrotate execution, by default it is planned daily, but we can configure it more often, and even set up customized logrotates for specific applications, by defining new logrotate config files in different directories and invoking them specifically.

In this example we will just configure daily rotation at day’s change ( 00:00 hours ) so we edit crontab ( crontab -e ) and add the following line after skulker (it makes sense, as skulker does system’s cleanup by deleting old files and logs, so it might save logrotate some extra work):

0 0 * * * /etc/logrotate

Done, so now we can go to the step 3, to check that logrotate works OK.

 

2.4- Add logrotate controls for our logs

Logrotate has loads of options, and even supports mini-scripting previous,during, and post-rotation, etc. Full documentation and examples can be found here:

https://linux.die.net/man/8/logrotate

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-log_rotation.html

If we want to add some simple logrotation configuration, we can add for example the following 3 files to /etc/logrotate.d directory to deal with supposed logs from some of our administration scripts (called check_all.ksh, start_all.ksh & stop_all.ksh):

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/check_all
 # log rotation for check_all.ksh sysadmin script:
 /home/admin/log/check_all.log {
 missingok
 daily
 rotate 2
 size 2M
 }
[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/start_all
 # log rotation for start_all.ksh sysadmin script:
 /home/admin/log/start_all.log {
 missingok
 rotate 1
 size 1M
 }
[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/stop_all
 # log rotation for stop_all.ksh sysadmin script:
 /home/admin/log/stop_all.log {
 rotate 1
 compress
 size 1M
 }

And the options are quite self-explanatory: in this case missingok will not report an error when the log file does not exist, daily rotates the log everyday (can be daily, weekly, monthly, annual), rotate X keeps X additional versions of the log, so rotate 2 will keep the original log, plus a log.1 and a log.2 copies (ex: rotate 2 == keep 2 additional copies), size 1M rotates the log when this one becomes bigger than 1MB (can be 10k, 10M, etc).

But since the above scripts are all part of a set of administration scripts all kept in /home/admin, in this case, it will be a better idea to just add the three stanzas all in the same config file, say sysadmin, as follows :

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/sysadmin
 # log rotation for sysadmin scripts located in /home/admin
 #

# log rotation for check_all.ksh sysadmin script:
 /home/admin/log/check_all.log {
 missingok
 daily
 rotate 2
 size 1M
 }

# log rotation for start_all.ksh sysadmin script:
 /home/admin/log/start_all.log {
 missingok
 rotate 1
 weekly
 size 10k
 }

# log rotation for stop_all.ksh sysadmin script:
 /home/admin/log/stop_all.log {
 rotate 1
 compress
 size 10k
 }

Done, so now we can go to the step 3.1, to check that logrotate works OK with our new config file.

But of course, step 3 and successive, will be food for the next post…


1 Comment

AIX 6L+ , AIX 7DevOps and Logrotate on AIX

When IBM added Linux integration to AIX v5, it called it “AIX 5L” to put special emphasis on the Linux integration part, so today I think that Linux v6 and above should be called AIX v6L+, since now we have at last YUM for AIX, eliminating the painful dependency hell caused by the RPM installations.

Jokes apart, IBM has done a great job porting YUM to AIX v6.1 and above, and now it’s really a breeze to install usual linux programs used in AIX servers, like sudo, gtar, logrotate, python, etc.

This is part of the effort that is currently being done in new technologies applied to the AIX universe, so an automatic package installer was needed, and yum was chosen.

But it doesn’t stop there with yum, as we also have python, perl, ruby, php, Node.JS, chef and other utilities and packages, so now we can build an AIX DevOps capable environment.

IBM announced all this in a december 2016 update, and it is a welcome addition to the AIX ecosystem, to bring it up to date with the latest IT tendencies.

If you want to find more about it, you couldn’t do worse that search on google for “yum on AIX pdf”, to get a couple of entries from IBM site.

The installation is quite simple, I have talked about it previously on my blog, but there’s plenty of info about it in IBM’s FTP site:

https://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/README-yum

And once we have yum installed, then we can move on to more “juicy” things, like Logrotate on AIX, since apart from the syslog facility, which incorporates it’s own advanced log handling, we don’t have a log rotation facility in AIX, and it is something that is really missed in an OS that has been around for a while.

So, here we go with the first step:

1.- Install logrotate with yum

[root@aix72:/home/admin]yum install logrotate
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package logrotate.ppc 0:3.8.5-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================
Package Arch Version Repository Size
============================================================================

Installing:
logrotate ppc 3.8.5-1 AIX_Toolbox 48 k

Transaction Summary
============================================================================

Install 1 Package

Total download size: 48 k
Installed size: 113 k
Is this ok [y/N]: y
Downloading Packages:
logrotate-3.8.5-1.aix6.1.ppc.rpm | 48 kB 00:00:00
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : logrotate-3.8.5-1.ppc 1/1

Installed:
logrotate.ppc 0:3.8.5-1

Complete!

[root@aix72:/home/admin]logrotate
logrotate 3.8.5 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License

Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command] [-s|--state=statefile] [-v|--verbose] [--version]
[-?|--help] [--usage] [OPTION...] <configfile>

OK, so we have installed logrotate, and as it is installed in /usr/bin, it can be invoked from anywhere, like the rest of the system’s external commands.

However, the logrotate package leaves the following files and directories in places where they are not expected to be in our AIX boxes:

[root@aix72:/etc/logrotate.d]ls -l /opt/freeware/etc/
total 8
drwxr-xr-x 2 root system 256 Nov 04 16:39 bash_completion.d
drwxr-xr-x 2 root system 256 Feb 26 16:20 cron.daily
-rw-r--r-- 1 root system 662 Jun 10 2013 logrotate.conf
drwxr-xr-x 2 root system 256 Apr 14 2016 logrotate.d

So, we will have to copy them to where they are supposed to be: that way logrotate will work without modification of the config files, and also sysadmins that know logrotate will be able to use it without having to look all over the system for the config files.

[root@aix72:/home/admin]cd /opt/freeware/etc
#copy the configuration file to /etc:
[root@aix72:/opt/freeware/etc]cp -p logrotate.conf /etc/ 
#copy the configuration directory (and it's files) to /etc:
[root@aix72:/opt/freeware/etc]cp -pr logrotate.d /etc/

[root@aix72:/opt/freeware/etc]cd cron.daily
#copy the logrotate script to be invoked from crontab to /etc:
[root@aix72:/opt/freeware/etc/cron.daily]cp -p logrotate /etc/

#NOTE: Not necessary, but a very good sysadmin practice for cases like this, 
is to update the timestamp on the files related:

#update the modification time of all the logrotate files & dir's:
[root@aix72:/opt/freeware/etc/cron.daily]cd /etc
[root@aix72:/etc]touch logrotate logrotate.conf logrotate.d

Done, so now we can go the next step, to configure logrotate for our AIX system (To be continued…).

 


2 Comments

UNIX/Linux everywhere???

In our days most of the appliances that support a terminal session and provide a CLI or command line interface behave like UNIX/Linux.

This is good, as at least there is a wide standard for basic system commands like:

 ls
 dir
 cat
 df
 uname
 hostname
 ps
 etc...

However the most common implementation of a CLI in appliances that have an embedded OS, is not really a Linux environment or shell, but an implementation of BusyBox.

BusyBox is a multi-call binary that combines many common Unix/Linux commands and utilities into a single executable, therefore creating links or alias to all the individual commands pointing to the same executable, saving space in disk and memory.

The latest version of BusyBox (v1.26.2 as of 10/01/2017) has all the following commands built in:

 [, [[, ar, awk, base64, basename, bunzip2, bzcat, bzip2, cal, cat,
 catv, chat, chgrp, chmod, chown, chpasswd, chpst, chroot, chrt,
 cksum, clear, cmp, comm, cp, cpio, cryptpw, cut, date, dc, dd,
 devmem, df, diff, dirname, dnsd, dnsdomainname, dos2unix, du, echo,
 ed, egrep, env, envdir, envuidgid, expand, expr, fakeidentd, false,
 fgrep, find, fold, fsync, ftpd, ftpget, ftpput, fuser, getopt, grep,
 groups, gunzip, gzip, hd, head, hexdump, hostid, hostname, httpd,
 id, inetd, install, iostat, ipcalc, kill, killall, killall5, less,
 ln, logger, logname, logread, lpd, lpq, lpr, ls, lzcat, lzma, lzop,
 lzopcat, makemime, man, md5sum, mesg, microcom, mkdir, mkfifo,
 mknod, mkpasswd, mktemp, more, mpstat, mt, mv, nc, nice, nmeter,
 nohup, nslookup, od, patch, pgrep, pidof, pipe_progress, pkill,
 popmaildir, printenv, printf, ps, pscan, pstree, pwd, pwdx,
 readlink, realpath, reformime, renice, reset, resize, rm, rmdir,
 rpm, rpm2cpio, run-parts, runsv, runsvdir, sed, sendmail, seq,
 setsid, setuidgid, sha1sum, sha256sum, sha512sum, sleep, smemcap,
 softlimit, sort, split, start-stop-daemon, stat, strings, stty, sum,
 sv, svlogd, sync, sysctl, syslogd, tac, tail, tar, tcpsvd, tee,
 telnet, telnetd, test, tftp, tftpd, time, timeout, touch, tr, true,
 tty, ttysize, udpsvd, uname, uncompress, unexpand, uniq, unix2dos,
 unlzma, unlzop, unxz, unzip, usleep, uudecode, uuencode, vlock,
 volname, watch, wc, which, whoami, whois, xargs, xz, xzcat, yes,
 zcat

If you want to try busybox on real time, you can go to this page, where there is a JavaScript PC emulator with BusyBox v1.20.0:

http://www.busybox.net/live_bbox/live_bbox.html

If you want to know more about this excellent and widely-used Open-Source project, visit:

http://www.busybox.net/about.html

Traditionally, to try a UNIX like environment on a windows PC, you had a couple of options: install microsoft’s own windows unix support, or install cygwin.

Well it’s good to know that now we have another option, and that it’s easier to install.
But in any case, my all-time favourite until today, is the use of MobaXterm.

This is a product to handle all sort of sessions & connections, and the best of it is that it lets you setup connections with terminals with SSH and Telnet, but also handles RDP, HTTP, etc. It even imports settings from PuTTY.

The best of all, is that it lets you open a local terminal session in windows, and that local session uses our new friend busybox, so you can also type windows commands (not built-ins, only externals, but hey!) or unix-like commands, at your leisure.

It’s specially good to develop unix or linux scripts in your PC, instead of doing so in a real server (really handy when you don’t have a development server in your environment, or if you are developing or fixing scripts in a laptop while you travel, on a hotel, etc.).

So there you go, if you want to have the best of linux in windows you can use busybox, cygwin, or MobaXterm. In Windows 10 mind, we also have the ubuntu bash shell, which is also very nice, and supported by Microsoft, excellent.


Leave a comment

Instalar YUM en AIX v7.2+

Una de las novedades de AIX v7.2 es que por fín IBM ha sacado un bundle para poder instalar yum bajo AIX. Y viene preconfigurado para usar el repositorio de IBM AIX Toolbox, BONUS!

Para instalar YUM en AIX, primero tenemos que actualizar el rpm a la v4.9 o superior, la última siempre se puede bajar de:

ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/INSTALLP/ppc/rpm.rte

O directamente desde el servidor AIX en donde lo vamos a instalar (mas cómodo si disponemos de conexión a internet en dicho servidor):

[root@tsm_srv:/software]ftp ftp.software.ibm.com
Connected to dispsd-40-www3.boulder.ibm.com.
220-**********************************************************************
220-* *
220-* IBM's internal systems must only be used for conducting IBM's *
220-* business or for purposes authorized by IBM management. *
220-* *
220-* Use is subject to audit at any time by IBM management. *
220-* *
220-* Important Please read *
220-* *
220-* Machine Code updates provided through this site are available *
220-* only for IBM machines that are under warranty or an IBM hardware *
220-* maintenance service agreement Code for operating systems or other *
220-* software products is available only where entitled under the *
220-* applicable software warranty or IBM software maintenance *
220-* agreement. All code (including Machine Code updates, samples, *
220-* fixes or other software downloads)provided through this site *
220-* is subject to the terms of the license agreements which *
220-* govern the use of the associated code. Some exceptions may *
220-* apply.IBM reserves the right to change, modify or withdraw its *
220-* offerings,policies and practices at any time. *
220-**********************************************************************
220-
220 service.boulder.ibm.com FTP server (Version wu-2.6.2.1(5) Custom Tue Aug 17 13:28:23 MDT 2010) ready.
Name (ftp.software.ibm.com:root): ftp
331 Guest login ok, send any password.
Password: aaaa@bbbb.com
230 Guest login ok, access restrictions apply.
ftp> cd /aix/freeSoftware/aixtoolbox/INSTALLP/ppc
ftp> get rpm.rte
200 PORT command successful.
150 Opening ASCII mode data connection for rpm.rte (354266 bytes).
226 Transfer complete.
355464 bytes received in 5.399 seconds (64.3 Kbytes/s)
local: rpm.rte remote: rpm.rte
ftp> bye

Una vez bajado, lo instalamos:

[root@tsm_srv:/software]installp -aXYgd . rpm.rte
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
Filesets listed in this section passed pre-installation verification
and will be installed.

Selected Filesets
-----------------
rpm.rte 4.9.1.3 # RPM Package Manager

<< End of Success Section >>

Una vez instalado el rpm actualizado, ya podemos bajarnos el bundle especial de YUM:

ftp://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/yum_bundle_v1.tar

Lo desempaquetamos:

[root@tsm_srv:/software]tar -xvf yum_bundle_v1.tar
x curl-7.44.0-1.aix6.1.ppc.rpm, 584323 bytes, 1142 media blocks.
x db-4.8.24-3.aix6.1.ppc.rpm, 2897799 bytes, 5660 media blocks.
x gdbm-1.8.3-5.aix5.2.ppc.rpm, 56991 bytes, 112 media blocks.
...

Instalamos todos los RPMs que vienen en el bundle:

[root@tsm_srv:/software]rpm -Uvh *.rpm
# Preparing... ########################################### [100%]
1:python ########################################### [ 9%]
2:pysqlite ########################################### [ 18%]
3:python-iniparse ########################################### [ 27%]
...

Y ya tenemos yum:

[root@tsm_srv:/software]<span style="color: #0000ff;">yum --version</span>
3.4.3
Installed: yum-3.4.3-3.noarch at 2016-11-04 21:39
Built : None at 2016-08-18 11:06
Committed: Sangamesh Mallayya <sangamesh.swamy@in.ibm.com> at 2016-08-19

[root@tsm_srv:/software]<span style="color: #0000ff;">yum repolist</span>
repo id repo name status
AIX_Toolbox AIX generic repository 259
AIX_Toolbox_72 AIX 7.2 specific repository 10
AIX_Toolbox_noarch AIX noarch repository 26
repolist: 295

Gracias IBM, y gracias Sangamesh!

Por fin ya podemos decir que rpm en AIX está “deprecated“.

;o)