Sysadmin's Shouts!

a blog for sysadmin's rants and raves…


1 Comment

Logrotate 3.- Logrotate checks

NOTE:  This is a follow-up, from the previous post:  Logrotate 2.- Configure logrotate for AIX

To check that logrotate is configured and working OK, all we need to do is call logrotate from the command line telling it to verbose it’s internal checks ( -v ) and to check the config file ( /etc/logrotate.conf ), like the following:

[root@aix72:/home/admin]/usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d      
reading config file failedlogin 
reading config file sysadmin    
reading config file wtmp        
reading config file yum         

Handling 6 logs

rotating pattern: /etc/security/failedlogin 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
 log does not need rotating        

rotating pattern: /home/admin/log/check_all.log 1048576 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log 
 log does not need rotating         

rotating pattern: /var/adm/wtmp 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 4 and egid to 4
considering log /var/adm/wtmp 
 log does not need rotating   
switching euid to 0 and egid to 0

rotating pattern: /var/log/yum.log yearly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log 
 log does not need rotating      

Once we have checked that the config is OK, we can check the rotation by Forcing rotation with the -f or –force flag:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.conf
 reading config file /etc/logrotate.conf
 including /etc/logrotate.d
 reading config file failedlogin
 reading config file sysadmin
 reading config file wtmp
 reading config file yum
 Handling 6 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/stop_all.log
 log does not need rotating

rotating pattern: /var/log/yum.log forced from command line (4 rotations)
 empty log files are not rotated, old logs are removed
 considering log /var/log/yum.log log needs rotating rotateCount is 4
 dateext suffix '-20170226'
 glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
 glob finding old rotated logs failed
 renaming /var/log/yum.log to /var/log/yum.log-20170226
 creating new /var/log/yum.log mode = 0600 uid = 0 gid = 0

Logrotate is configured OK and it seems to work fine, so if it’s not executing properly, we will have to check it’s schedule on the crontab.

NOTE: Notice that when we configure the rotation to be on size, the –force option cannot force this rotation, so to force rotation on stanzas where size has been used, just lower the size attribute temporarily (size 10k instead of 5M, for example).

3.1- Logrotate individual files/logs check

To check logrotate’s config for a particular file, we will have to identify it first in the /etc/logrotate.d directory, for example to check the config for yum’s logs:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/yum
reading config file /etc/logrotate.d/yum

Handling 1 logs

rotating pattern: /var/log/yum.log forced from command line (no old logs will be kept)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log
 log does not need rotating

To check the config for a specific log, but we don’t see a logrotate file stored by its name in /etc/logrotate.d, we will have to dig it out (for example let’s look for start_all.log):

[root@aix72:/home/admin]grep start_all.log /etc/logrotate.d/*
/etc/logrotate.d/sysadmin:/home/admin/log/start_all.log

OK, so it looks like the logrotate config for start_all.log resides in the /etc/logrotate.d/sysadmin file, so now we can check it:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/sysadmin
reading config file /etc/logrotate.d/sysadmin

Handling 3 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations) 
empty log files are rotated, old logs are removed
considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/stop_all.log
 log does not need rotating

So, as always, an important part of a configuration (the most important, actually) is to check that our new config works just as we expected it.

And now we have seen how to check all the logrotate config, how to force the log rotation, and how to check individual logrotate config files, so with this three checks we should be able to perform config-test-change-retest until our friend logrotate does what we expect it to.

On the step 4, I will talk about logrotate documentation & support, and step 5 will show how to fix common logrotate errors. See you soon.


1 Comment

Logrotate 2.- Configure logrotate for AIX

NOTE:  This is a follow-up, from the previous post: AIX 6L+ , AIX 7DevOps and Logrotate on AIX

Logrotate is a utility from RHEL, and therefore it comes preconfigured for RHEL & fedora, so after installing it using yum, we need to adapt it to work in our AIX system.

2.1- Fix logrotate.conf invalid entries

By default, logrotate’s main config file treats logs of wtmp & btmp, but since we can treat wtmp separately, and btmp is not implemented in AIX, we can just comment out or better still, delete those lines from /etc/logrotate.conf:

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.conf
 # see "man logrotate" for details
 # rotate log files weekly
 weekly

# keep 4 weeks worth of backlogs
 rotate 4

# create new (empty) log files after rotating old ones
 create

# use date as a suffix of the rotated file
 dateext

# uncomment this if you want your log files compressed
 #compress

# RPM packages drop log rotation information into this directory
 include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here      
/var/log/wtmp {
 monthly
 create 0664 root utmp
 minsize 1M
 rotate 1
 }

/var/log/btmp {
 missingok
 monthly
 create 0600 root utmp
 rotate 1
 }

NOTE: There is also a good idea to put a line like the following to the bottom of /etc/logrotate.conf to sepparate the default system config from future additions:

# Installed by Carlos Ijalba, 2017. Put new generic logconfigs below this line: ##########

 

2.2- Fix the log rotation for yum

By default, logrotate comes configured to treat yum logs, but we need to change the owner group of the yum logs in RHEL (root) for AIX default system group (system), so we edit the file /etc/logrotate.d/yum, and change line 6 last root entry for system:

[root@aix72:/etc/logrotate.d]cat /etc/logrotate.d/yum
 /var/log/yum.log {
 missingok
 notifempty
 size 30k
 yearly
 create 0600 root system
 }

2.3- Setup logrotate schedule in crontab

And the last step, will be to configure the contab entry for logrotate execution, by default it is planned daily, but we can configure it more often, and even set up customized logrotates for specific applications, by defining new logrotate config files in different directories and invoking them specifically.

In this example we will just configure daily rotation at day’s change ( 00:00 hours ) so we edit crontab ( crontab -e ) and add the following line after skulker (it makes sense, as skulker does system’s cleanup by deleting old files and logs, so it might save logrotate some extra work):

0 0 * * * /etc/logrotate

Done, so now we can go to the step 3, to check that logrotate works OK.

 

2.4- Add logrotate controls for our logs

Logrotate has loads of options, and even supports mini-scripting previous,during, and post-rotation, etc. Full documentation and examples can be found here:

https://linux.die.net/man/8/logrotate

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-log_rotation.html

If we want to add some simple logrotation configuration, we can add for example the following 3 files to /etc/logrotate.d directory to deal with supposed logs from some of our administration scripts (called check_all.ksh, start_all.ksh & stop_all.ksh):

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/check_all
 # log rotation for check_all.ksh sysadmin script:
 /home/admin/log/check_all.log {
 missingok
 daily
 rotate 2
 size 2M
 }
[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/start_all
 # log rotation for start_all.ksh sysadmin script:
 /home/admin/log/start_all.log {
 missingok
 rotate 1
 size 1M
 }
[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/stop_all
 # log rotation for stop_all.ksh sysadmin script:
 /home/admin/log/stop_all.log {
 rotate 1
 compress
 size 1M
 }

And the options are quite self-explanatory: in this case missingok will not report an error when the log file does not exist, daily rotates the log everyday (can be daily, weekly, monthly, annual), rotate X keeps X additional versions of the log, so rotate 2 will keep the original log, plus a log.1 and a log.2 copies (ex: rotate 2 == keep 2 additional copies), size 1M rotates the log when this one becomes bigger than 1MB (can be 10k, 10M, etc).

But since the above scripts are all part of a set of administration scripts all kept in /home/admin, in this case, it will be a better idea to just add the three stanzas all in the same config file, say sysadmin, as follows :

[root@aix72:/etc/logrotate.d]vi /etc/logrotate.d/sysadmin
 # log rotation for sysadmin scripts located in /home/admin
 #

# log rotation for check_all.ksh sysadmin script:
 /home/admin/log/check_all.log {
 missingok
 daily
 rotate 2
 size 1M
 }

# log rotation for start_all.ksh sysadmin script:
 /home/admin/log/start_all.log {
 missingok
 rotate 1
 weekly
 size 10k
 }

# log rotation for stop_all.ksh sysadmin script:
 /home/admin/log/stop_all.log {
 rotate 1
 compress
 size 10k
 }

Done, so now we can go to the step 3.1, to check that logrotate works OK with our new config file.

But of course, step 3 and successive, will be food for the next post…


Leave a comment

AIX 6L+ , AIX 7DevOps and Logrotate on AIX

When IBM added Linux integration to AIX v5, it called it “AIX 5L” to put special emphasis on the Linux integration part, so today I think that Linux v6 and above should be called AIX v6L+, since now we have at last YUM for AIX, eliminating the painful dependency hell caused by the RPM installations.

Jokes apart, IBM has done a great job porting YUM to AIX v6.1 and above, and now it’s really a breeze to install usual linux programs used in AIX servers, like sudo, gtar, logrotate, python, etc.

This is part of the effort that is currently being done in new technologies applied to the AIX universe, so an automatic package installer was needed, and yum was chosen.

But it doesn’t stop there with yum, as we also have python, perl, ruby, php, Node.JS, chef and other utilities and packages, so now we can build an AIX DevOps capable environment.

IBM announced all this in a december 2016 update, and it is a welcome addition to the AIX ecosystem, to bring it up to date with the latest IT tendencies.

If you want to find more about it, you couldn’t do worse that search on google for “yum on AIX pdf”, to get a couple of entries from IBM site.

The installation is quite simple, I have talked about it previously on my blog, but there’s plenty of info about it in IBM’s FTP site:

https://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/README-yum

And once we have yum installed, then we can move on to more “juicy” things, like Logrotate on AIX, since apart from the syslog facility, which incorporates it’s own advanced log handling, we don’t have a log rotation facility in AIX, and it is something that is really missed in an OS that has been around for a while.

So, here we go with the first step:

1.- Install logrotate with yum

[root@aix72:/home/admin]yum install logrotate
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package logrotate.ppc 0:3.8.5-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================
Package Arch Version Repository Size
============================================================================

Installing:
logrotate ppc 3.8.5-1 AIX_Toolbox 48 k

Transaction Summary
============================================================================

Install 1 Package

Total download size: 48 k
Installed size: 113 k
Is this ok [y/N]: y
Downloading Packages:
logrotate-3.8.5-1.aix6.1.ppc.rpm | 48 kB 00:00:00
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : logrotate-3.8.5-1.ppc 1/1

Installed:
logrotate.ppc 0:3.8.5-1

Complete!

[root@aix72:/home/admin]logrotate
logrotate 3.8.5 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License

Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command] [-s|--state=statefile] [-v|--verbose] [--version]
[-?|--help] [--usage] [OPTION...] <configfile>

OK, so we have installed logrotate, and as it is installed in /usr/bin, it can be invoked from anywhere, like the rest of the system’s external commands.

However, the logrotate package leaves the following files and directories in places where they are not expected to be in our AIX boxes:

[root@aix72:/etc/logrotate.d]ls -l /opt/freeware/etc/
total 8
drwxr-xr-x 2 root system 256 Nov 04 16:39 bash_completion.d
drwxr-xr-x 2 root system 256 Feb 26 16:20 cron.daily
-rw-r--r-- 1 root system 662 Jun 10 2013 logrotate.conf
drwxr-xr-x 2 root system 256 Apr 14 2016 logrotate.d

So, we will have to copy them to where they are supposed to be: that way logrotate will work without modification of the config files, and also sysadmins that know logrotate will be able to use it without having to look all over the system for the config files.

[root@aix72:/home/admin]cd /opt/freeware/etc
#copy the configuration file to /etc:
[root@aix72:/opt/freeware/etc]cp -p logrotate.conf /etc/ 
#copy the configuration directory (and it's files) to /etc:
[root@aix72:/opt/freeware/etc]cp -pr logrotate.d /etc/

[root@aix72:/opt/freeware/etc]cd cron.daily
#copy the logrotate script to be invoked from crontab to /etc:
[root@aix72:/opt/freeware/etc/cron.daily]cp -p logrotate /etc/

#NOTE: Not necessary, but a very good sysadmin practice for cases like this, 
is to update the timestamp on the files related:

#update the modification time of all the logrotate files & dir's:
[root@aix72:/opt/freeware/etc/cron.daily]cd /etc
[root@aix72:/etc]touch logrotate logrotate.conf logrotate.d

Done, so now we can go the next step, to configure logrotate for our AIX system (To be continued…).