Sysadmin's Shouts!

a blog for sysadmin's rants and raves…


Leave a comment

Revisando Vulnerabilidades en AIX

IBM tiene una herramienta para reportar vulnerabilidades en sus productos, llamada Fix Level Recommendation Tool o FLRT (herramienta de recomendación de parches).

https://www-304.ibm.com/support/customercare/flrt/

Para AIX en particular, disponemos del  Security APAR Information, o el Security Bulletin information for AIX 7.2, 7.1, 6.1, 5.3, and VIOS

https://www-304.ibm.com/webapp/set2/flrt/doc?page=security

Para facilitar la comprobación de nuestros sistemas, tenemos el script en korn shell flrtvc.ksh, el cual nos obsequia con informes en varios tipos de formatos (CSV para importar en excel u otros, personalizados, compacto, detallado, etc).

Los prerequisitos de este script son los siguientes:

1.- access to internet to retrieve the latest vulnerability CSV listing (aparCSV)
2.- wget
3.- curl

Los puntos 2 y 3 son fácilmente obtenibles, si hemos configurado yum en nuestro AIX (yum install wget curl).

Algunos ejemplos de ejecución del script flrtvc:

[root@aixtest:/home/admin]./flrtvc.ksh | cut -c 1-110
Fileset|Current Version|Type|EFix Installed|Abstract|Unsafe Versions|APARs|Bulletin URL|Download URL
bos.acct|7.2.1.0|sec||NOT FIXED - (caccelstat) Vulnerabilities in bellmail / caccelstat / iostat / l
bos.acct|7.2.1.0|sec||NOT FIXED - (iostat) Vulnerabilities in bellmail / caccelstat / iostat / lquer
bos.acct|7.2.1.0|sec||NOT FIXED - (vmstat) Vulnerabilities in bellmail / caccelstat / iostat / lquer
bos.cluster.rte|7.2.1.0|hiper||NOT FIXED - CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLU
bos.mp64|7.2.1.1|hiper||NOT FIXED - getsockname() returns incorrect NameLength|7.2.1.0-7.2.1.1|IV914
bos.mp64|7.2.1.1|hiper||NOT FIXED - PROBLEMS CAN OCCUR WITH THREAD_CPUTIME AND THREAD_CPUTIME_FAST|7
bos.mp64|7.2.1.1|hiper||NOT FIXED - CRASH OR POTENTIAL DATA LOSS AFTER REMOVING LARGE JFS2 FILES ON
bos.mp64|7.2.1.1|hiper||NOT FIXED - SYSTEM CRASH WHEN USING PROCFS FOR PROCESSES CLOSING MANY FILES|
bos.mp64|7.2.1.1|sec||NOT FIXED - IBM has released AIX and VIOS iFixes in response to the vulnerabil
bos.net.tcp.bind_utils|7.2.1.1|sec||NOT FIXED - There is a vulnerability in BIND that impacts AIX.|7
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - There is a vulnerability in bellmail that impacts A
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - Vulnerabilities in BIND impact AIX|7.2.1.0|CVE-2016
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - There are two vulnerabilities in BIND that impact A
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - Vulnerability in bellmail affects AIX|7.2.1.0-7.2.1
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - (bellmail) Vulnerabilities in bellmail / caccelstat
bos.net.tcp.ntp|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in NTPv3 and NTPv4 that
bos.net.tcp.ntpd|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in NTPv3 and NTPv4 that
bos.net.tcp.tcpdump|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in tcpdump that impa
bos.rte.archive|7.2.1.0|sec||NOT FIXED - (restbyinode) Vulnerabilities in bellmail / caccelstat / io
bos.rte.lvm|7.2.1.0|sec||NOT FIXED - (lquerypv) Vulnerabilities in bellmail / caccelstat / iostat /
devices.fcp.disk.rte|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERRORS WITH CERTA
devices.pci.77102224.com|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERRORS WITH C
devices.pciex.df1060e214103404.com|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERR
devices.vdevice.ibm.l-lan.rte|7.2.1.0|hiper||NOT FIXED - CRASH IN VIOENT_INIT_LS_TIMER WHEN POLL_UPL
devices.vdevice.ibm.vfc-client.rte|7.2.1.0|hiper||NOT FIXED - Potential data loss using Virtual FC w
java7_64.jre|7.0.0.370|sec||NOT FIXED - There are multiple vulnerabilities in IBM SDK Java Technolog
java7_64.sdk|7.0.0.370|sec||NOT FIXED - Multiple vulnerabilities in IBM Java SDK affect AIX|<7.0.0.4
java7_64.sdk|7.0.0.370|sec||NOT FIXED - Multiple vulnerabilities in IBM Java SDK affect AIX|<7.0.0.5
java7_64.sdk|7.0.0.370|sec||NOT FIXED - There are multiple vulnerabilities in IBM SDK Java Technolog
openssh.base.client|6.0.0.6201|sec||NOT FIXED - AIX OpenSSH Vulnerability|4.0.0.5200-6.0.0.6201|CVE-
openssh.base.client|6.0.0.6201|sec||NOT FIXED - Vulnerabilities in OpenSSH affect AIX|4.0.0.5200-6.0
openssl.base|1.0.2.800|sec||NOT FIXED - There is a vulnerability in OpenSSL used by AIX|1.0.2.500-1.
openssl.base|1.0.2.800|sec||NOT FIXED - Vulnerability in OpenSSL affects AIX|1.0.2.500-1.0.2.1100|CV
...
[root@aixtest:/home/admin]./flrtvc.ksh -v | pg
////////////////////////////////////////////////////////////
// IBM FLRTVC (v0.7.3) Report
// Server: aixtest
// Date: Fri Feb 9 10// Report by: root
// Vulnerable Filesets: 22
// Total Vulnerabilities: 54
// Total Fixes (not shown): 22
////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------
bos.acct - 7.2.1.0 - Vulnerabilities (3)
--------------------------------------------------------------------------------

(1) NOT FIXED - (caccelstat) Vulnerabilities in bellmail / caccelstat / iostat / lquerypv / restbyinode / vmstat affect AIX (CVE-2017-1692)

Type: sec
Score: CVE-2017-1692:8.4
Versions: 7.2.1.0-7.2.1.0
APARs/CVEs: IV97811
Last Update: 02/05/2018
Bulletin: http://aix.software.ibm.com/aix/efixes/security/suid_advisory.asc
Download: ftp://aix.software.ibm.com/aix/efixes/security/suid_fix.tar
Fixed In: 7200-01-04

(2) NOT FIXED - (iostat) Vulnerabilities in bellmail / caccelstat / iostat / lquerypv / restbyinode / vmstat affect AIX (CVE-2017-1692)
Type: sec
Score: CVE-2017-1692:8.4
Versions: 7.2.1.0-7.2.1.1
APARs/CVEs: IV97898
Last Update: 02/05/2018
Bulletin: http://aix.software.ibm.com/aix/efixes/security/suid_advisory.asc
Download: ftp://aix.software.ibm.com/aix/efixes/security/suid_fix.tar
Fixed In: 7200-01-04
...

Francamente, yo la encuentro una herramienta fantástica, que puede ahorrarnos un montón de tiempo cuando necesitamos efectuar un control de vulnerabilidades en alguno de nuestros sistemas.

El listado de parámetros completo de la herramienta es el siguiente:

Usage flrtvc: Change delimiter for compact reporting
 ./flrtvc.ksh -d '||'

Usage flrtvc: Generate full reporting (verbose mode)
 ./flrtvc.ksh -v

Usage flrtvc: Choose custom apar.csv file to use
 ./flrtvc.ksh -f myfile.csv

Usage flrtvc: Only show specific filesets in verbose mode
 ./flrtvc.ksh -vg printers

Usage flrtvc: Show only hiper results
 ./flrtvc.ksh -t hiper

Usage flrtvc: Custom lslpp and emgr outputs
 ./flrtvc.ksh -l lslpp.txt -e emgr.txt

Flags:

-d = Change delimiter for compact reporting
-f = Enter a custom aparCSV file in local filesystem
-q = Quiet mode, hide compact reporting header
-s = Skip download and locate 'apar.csv' filename in current directory
-v = Verbose, full report (for piping to email)
-g = Filter filesets for specific phrase, useful for verbose mode
-t = Type of APAR [hiper | sec]
-l = Enter a custom LSLPP output file, must match lslpp -Lqc
-e = Enter a custom EMGR output file, must match emgr -lv3
-x = Skip EFix processing
-a = Show all fixed and non-fixed HIPER/Security vulnerabilities.
Advertisements


Leave a comment

Update Cisco Servers CIMC’s Firmware

Cisco’s CIMC updates are done using the HUU – Cisco Host Upgrade Utility, which comes in ISO form so can be burned on physical media or mounted remotely via the KVM on the CICM interface. Quoting Cisco’s manual:

<<Upgrading BIOS and Cisco IMC Firmware, Cisco provides the Cisco Host Upgrade Utility to assist you in upgrading the BIOS, Cisco IMC, CMC LOM, LSI storage controller, and Cisco UCS Virtual Interface Cards firmware to compatible levels.

Always upgrade the BIOS, the Cisco IMC and CMC from the HUU ISO. Do not upgrade individual components (only BIOS or only Cisco IMC or CMC), since this could lead to unexpected behavior. If you choose to upgrade BIOS, the Cisco IMC and the CMC individually and not from the HUU ISO, make sure to upgrade both Cisco IMC, BIOS and CMC to the same container release. If the BIOS, CMC and the Cisco IMC versions are from different container releases, it could result in unexpected behavior. Cisco recommends that you use the Update All option from the Host Upgrade Utility to update the firmware versions of Cisco IMC, BIOS, CMC and all other server components (VIC, RAID Controllers, PCI devices, and LOM) together.>>

CIMC_01

1.- Connect to Cisco Download Software website & download the ISO update.

https://software.cisco.com/download/navigator.html

Example of direct dowload HUU 3.03e

https://software.cisco.com/download/release.html?mdfid=286281356&flowid=71443&softwareid=283850974&release=3.0(3e)&relind=AVAILABLE&rellifecycle=&reltype=latest

Easiest way to find the latest software for your Server if you have updated it before:

From Download Software > My Download History > UCS C240 Rack Server Software > Latest Release > Download File

2.- Perform a Backup

Backup of the FW.

NOTE: in CIMC this step is performed automatically. Backup FW versions can be perused in CIMC > Admin > Firmware Management

Backup of the BIOS.

Navigation Panel (Top Left Icon) > Compute > BIOS > Configure BIOS Profile > Take Backup

3.- Revise NTP & TimeZone

Set NTP:

Navigation Panel (Top Left Icon) > Admin > Networking > NTP Setting > NTP Properties
NTP Enabled x
Server 1: 0.pool.ntp.org
Server 2: 1.pool.ntp.org
Server 3: 2.pool.ntp.org
Server 4: 3.pool.ntp.org
> Save Changes

Set TimeZone:

Navigation Panel (Top Left Icon) > Chassis > Summary >Cisco Integrated Management Controller (Cisco IMC) Information > Timezone: Europe/Madrid

Check Time:

Navigation Panel (Top Left Icon) > Chassis > Summary >Cisco Integrated Management Controller (Cisco IMC) Information > Local Time

Check NTP (Note: leave at least 15 minutes between configuration and verification –to avoid getting a stratum 16):

Navigation Panel (Top Left Icon) > Admin > Networking > NTP Setting > NTP Properties > Status: Values 1-15

4.- Start Console

Navigation Panel (Top Left Icon) > Admin > Firmware Management > Firmware Management Review the versions stored.

Open a KVM (Java or HTML), map the ISO downloaded to the server as Virtual Media, CD.
NOTE: if we choose the Java KVM, we can record the whole installation in a video (for documentation purposes), the HTML KVM option can only take snaps, not videos.

If the UCS is being managed from a vCenter or a OVMmanager, move the VMs off the server first and put into maintenance or stopped mode.

5.- Reboot from HUU

On Boot, press <F6>, to choose the Boot Menu, and once there, select boot device Cisco vKVM-Mapped vDVD1.22. Now it will start from the HUU software.

ISOLINUX 4.04… image will start booting, followed by a GUI named Cisco UCS Host Upgrade Utility, with an animated loading circle, which does a few full turns.

6.- Update FW Packages

Once HUU has finished copying the FW to the CIMC, it will display the Agreement page, which we will confirm, and after that we can review the FW levels to be updated.

We will select the Update All option, and Are you sure? Yes

It will mark all the packages in the inventory as SCHEDULED, and it will go through each one as IN PROGRESS, PASS.
NOTE: The BIOS is updated last, as after doing that step, the system reboots.

7.- Verify Upgrade & System Status

Once all the packages have a green PASS update status, we can press the Exit button, wait 5 minutes and login into the new CIMC.

Review that the CIMC is correct via HOME > Chassis > Summary, note that the CIMC starts pretty quick, but the server takes longer to start, and it will show as powered off from the CIMC until it has boot up, so it’s best to wait for a while to see if the server comes up OK, before issuing a host power on from the CIMC.

8.- Now, go and grab a coffee… ‘coz we’ve finished!

 


Leave a comment

SuSe VM Migrated from VMware

SuSe VMs migrated from VMware to other platforms (OVM, Xen, KVM, Ravello) check for compliance on startup, and give the following message (or simmilar):

The profile does not allow you to run the products on this system.

Hypervisor used current value: 'xen' must be one of: 'vmware'.

Forcing you to go into the console and press any key to accept the warning disclaimer that the actual setup is not supported by Suse.

OVM - Suse VMware to OVM migrations

While this might be good to know the first time you migrate a VM, it’s a disguised “Nagging Screen“, which limits the use of this VMs in a production setup, as reboots require manual intervention. And as we know, sometimes is not that easy to recreate the VM from fresh in the supported setup.

To bypass this compliance check, we should find out the script which is giving us the error, so we can decide how to bypass it:

1.- First, we check the release version of our troubled system, so we can write down the solution for this particular version (in case different versions have diff. solutions).

SUSETEST:~ # cat /etc/*release*
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4
LSB_VERSION="core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64"
cat: /etc/lsb-release.d: Is a directory
NAME="SLES"
VERSION="11.4"
VERSION_ID="11.4"
PRETTY_NAME="SUSE Linux Enterprise Server 11 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:11:4"

2.- Second, we will try to find the script that gives us the nasty message, by launching a brute-force grep on /usr/bin:

SUSETEST:~ # grep "profile does not allow you" /usr/bin/*
isCompliant: print __("The profile does not allow you to run the products on this system.\n");

3.- Third, we have found one file called isCompliant on /usr/bin, so we will find more about it:

SUSETEST:~# file /usr/bin/isCompliant
/usr/bin/isCompliant: a /usr/bin/perl -w script text

It is a perl script, we can check to see if it has manual page:

SUSETEST:~# man isCompliant

ISCOMPLIANT(1) ISCOMPLIANT(1)

NAME
isCompliant

SYNOPSIS
isCompliant [options]

DESCRIPTION
"isCompliant" check a profile with the system where products gets installed.

OPTIONS
--quiet -q
Do not print messages on stdout or stderr. Only the exit code indicate if the compliance check was successfull or
not.

--debug -d
Turn on the debug mode.

AUTHORS and CONTRIBUTORS
Duncan Mac-Vicar Prett, Michael Calmer

LICENSE
Copyright (c) 2011 SUSE LINUX Products GmbH, Nuernberg, Germany.

Or Help info:

SUSETEST:~# /usr/bin/isCompliant -h
usage: /usr/bin/isCompliant [--quiet|-q]
/usr/bin/isCompliant [--debug|-d]
/usr/bin/isCompliant [--help|-h|-?]
Options:
-h -? [--help] show this help
-q [--quiet] print no messages
-d [--debug] print debug messages
isCompliant check a profile with the system where products gets installed.
isCompliant exit with 0 if the system is compliant to the product profile.
Otherwise it exist with 1

Looks like it’s just a script that checks certain compliance requirements and returns our infamous message, and on the help it says that it returns a RC=0 when a system is compliant, and a RTC=1 when it’s not.

We also see, that running it with the -q flag, it will only run and return a RC, without displaying messages on screen.

OK, knowing this info, let’s invoke it, and check it out:

SUSETEST:~# /usr/bin/isCompliant
The profile does not allow you to run the products on this system.

Proceeding to run this installation will leave you in an
unsupported state and might impact your compliance requirements.

The following requirements are not fulfilled on this system:
* Hypervisor used (current value: 'xen'); must be one of: 'vmware'

SUSETEST:~# echo $?
1

SUSETEST:~# /usr/bin/isCompliant -q
SUSETEST:~# echo $?
1

Bingo! This is definitely our boot pausing friend. It gives the error message, and returns a RC=1. But it’s not the one that forces us to press a key if the system is not
compliant, so it must be invoked by another script at boot time.

Now we could do two things, one is replacing this script for our own “compliance checker” and the other is to find the isCompliant invoking script and change it there, just
in that case replacing the original isCompliant breaks something else further down the line.

My recommended method is Solution 2, but the fastest to implement is Solution 1.

Solution 1.- Replace isCompliant with our own brew.

Before replacing the original script, we will keep the original, just in case it’s needed further on, and to have a rollback option.

SUSETEST:~# cp -p /usr/bin/isCompliant /usr/bin/isCompliant.orig

Now we will just replace the script with a simple “exit 0”, that will allways return OK in any case.

SUSETEST:~# echo "exit 0" > /usr/bin/isCompliant

 

Solution 2.- Modify isCompliant boot invoking script.

First we need to find out the name of the isCompliant invoking scrpits, so we launch another brute-force grep, this time at /etc/init.d folders:

SUSETEST:~# grep "/usr/bin/isCompliant" /etc/init.d/*
/etc/init.d/boot.compliance: MSG=`/usr/bin/isCompliant`

And we find it, it’s called boot.compliance (so it really was pretty well self-documented).

We check the startup script, and we find the start sequence:

case "$1" in
start|restart|force-reload)
# Check if we're running in inst-sys or stage 2 of the installation - if so,
# the check must not be performed as there's already a check in YaST
if test -f /var/lib/YaST2/runme_at_boot ; then
exit
fi
echo -n "Check if the profiles matches the system"

MSG=`/usr/bin/isCompliant`
CODE=$?
if [ "$CODE" != "0" ]; then
splash=""
# Switch bootsplash to verbose mode to make text messages visible.
if test -f /proc/splash ; then
read splash < /proc/splash echo "verbose" > /proc/splash
fi

clear

echo -e "\n"
echo -e "=========================================================================="
echo -e "\n"
echo -e "$MSG"
echo -e "\n"
echo -e "Press any key to proceed with booting."
echo -e "\n"
echo -e "=========================================================================="

read -n 1

clear
[[ "$splash" =~ silent ]] && echo silent > /proc/splash
fi

As we can see marked in red, the variable MSG executes and captures the isCompliant output, and the variable CODE collects it’s return code.
Then an “if” sentence checks the compliance, and if its different from 0 (Not Compliant), shows the screen with the MSG and waits for any key to be pressed before continuing.

As usual, we will make a backup of the original config file before making any changes into the working configuration:

SUSETEST:~# cp -p /etc/init.d/boot.compliance /etc/init.d/boot.compliance.orig

So, our change it’s going to be replacing the “read -n 1” with a “sleep 6“, because we are going to leave the “Not Compliance” message on screen during 6 seconds at boot time, to remind us that this system is not supported by SuSe in it’s actual configuration.

We will also comment the “Press any key…” message, to allow for automatic rebooting of the system.

SUSETEST:~# vi /etc/init.d/boot.compliance

It will end up as follows:

case "$1" in
start|restart|force-reload)
# Check if we're running in inst-sys or stage 2 of the installation - if so,
# the check must not be performed as there's already a check in YaST
if test -f /var/lib/YaST2/runme_at_boot ; then
exit
fi
echo -n "Check if the profiles matches the system"

MSG=`/usr/bin/isCompliant`
CODE=$?
if [ "$CODE" != "0" ]; then
splash=""
# Switch bootsplash to verbose mode to make text messages visible.
if test -f /proc/splash ; then
read splash < /proc/splash echo "verbose" > /proc/splash
fi

clear

echo -e "\n"
echo -e "=========================================================================="
echo -e "\n"
echo -e "$MSG"
echo -e "\n"
#echo -e "Press any key to proceed with booting."
echo -e "\n"
echo -e "=========================================================================="

#read -n 1
sleep 6

clear
[[ "$splash" =~ silent ]] && echo silent > /proc/splash
fi

We can save the modified file and reboot the system to check that we have the 6 seconds warning and then just carries on booting as usual.

So, with a little digging around the system we have found how to bypass the nagging screen and convert back our migrated VM into a workable system.

 


Leave a comment

AIX Vulnerabilities

IBM has a tool to track and report vulnerabilites in it’s products, called the Fix Level Recommendation Tool (FLRT).

https://www-304.ibm.com/support/customercare/flrt/

Particularly for AIX, it has the Security APAR Information, or Security Bulletin information for AIX 7.2, 7.1, 6.1, 5.3, and VIOS

https://www-304.ibm.com/webapp/set2/flrt/doc?page=security

And to check our systems, IBM provides the flrtvc.ksh script, which produces an awesome output, in different formats.
As prerequisites, it needs:

1.- access to internet to retrieve the latest vulnerability CSV listing (aparCSV)
2.- wget
3.- curl

Points 2 & 3 are easily done if we have setup yum in our AIX system (yum install wget curl).

Some examples of flrtvc script execution:

[root@aixtest:/home/admin]./flrtvc.ksh | cut -c 1-110
Fileset|Current Version|Type|EFix Installed|Abstract|Unsafe Versions|APARs|Bulletin URL|Download URL
bos.acct|7.2.1.0|sec||NOT FIXED - (caccelstat) Vulnerabilities in bellmail / caccelstat / iostat / l
bos.acct|7.2.1.0|sec||NOT FIXED - (iostat) Vulnerabilities in bellmail / caccelstat / iostat / lquer
bos.acct|7.2.1.0|sec||NOT FIXED - (vmstat) Vulnerabilities in bellmail / caccelstat / iostat / lquer
bos.cluster.rte|7.2.1.0|hiper||NOT FIXED - CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLU
bos.mp64|7.2.1.1|hiper||NOT FIXED - getsockname() returns incorrect NameLength|7.2.1.0-7.2.1.1|IV914
bos.mp64|7.2.1.1|hiper||NOT FIXED - PROBLEMS CAN OCCUR WITH THREAD_CPUTIME AND THREAD_CPUTIME_FAST|7
bos.mp64|7.2.1.1|hiper||NOT FIXED - CRASH OR POTENTIAL DATA LOSS AFTER REMOVING LARGE JFS2 FILES ON
bos.mp64|7.2.1.1|hiper||NOT FIXED - SYSTEM CRASH WHEN USING PROCFS FOR PROCESSES CLOSING MANY FILES|
bos.mp64|7.2.1.1|sec||NOT FIXED - IBM has released AIX and VIOS iFixes in response to the vulnerabil
bos.net.tcp.bind_utils|7.2.1.1|sec||NOT FIXED - There is a vulnerability in BIND that impacts AIX.|7
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - There is a vulnerability in bellmail that impacts A
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - Vulnerabilities in BIND impact AIX|7.2.1.0|CVE-2016
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - There are two vulnerabilities in BIND that impact A
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - Vulnerability in bellmail affects AIX|7.2.1.0-7.2.1
bos.net.tcp.client_core|7.2.1.0|sec||NOT FIXED - (bellmail) Vulnerabilities in bellmail / caccelstat
bos.net.tcp.ntp|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in NTPv3 and NTPv4 that
bos.net.tcp.ntpd|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in NTPv3 and NTPv4 that
bos.net.tcp.tcpdump|7.2.1.0|sec||NOT FIXED - There are multiple vulnerabilities in tcpdump that impa
bos.rte.archive|7.2.1.0|sec||NOT FIXED - (restbyinode) Vulnerabilities in bellmail / caccelstat / io
bos.rte.lvm|7.2.1.0|sec||NOT FIXED - (lquerypv) Vulnerabilities in bellmail / caccelstat / iostat /
devices.fcp.disk.rte|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERRORS WITH CERTA
devices.pci.77102224.com|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERRORS WITH C
devices.pciex.df1060e214103404.com|7.2.1.0|hiper||NOT FIXED - UNDETECTED DATA LOSS AFTER STORAGE ERR
devices.vdevice.ibm.l-lan.rte|7.2.1.0|hiper||NOT FIXED - CRASH IN VIOENT_INIT_LS_TIMER WHEN POLL_UPL
devices.vdevice.ibm.vfc-client.rte|7.2.1.0|hiper||NOT FIXED - Potential data loss using Virtual FC w
java7_64.jre|7.0.0.370|sec||NOT FIXED - There are multiple vulnerabilities in IBM SDK Java Technolog
java7_64.sdk|7.0.0.370|sec||NOT FIXED - Multiple vulnerabilities in IBM Java SDK affect AIX|<7.0.0.4
java7_64.sdk|7.0.0.370|sec||NOT FIXED - Multiple vulnerabilities in IBM Java SDK affect AIX|<7.0.0.5
java7_64.sdk|7.0.0.370|sec||NOT FIXED - There are multiple vulnerabilities in IBM SDK Java Technolog
openssh.base.client|6.0.0.6201|sec||NOT FIXED - AIX OpenSSH Vulnerability|4.0.0.5200-6.0.0.6201|CVE-
openssh.base.client|6.0.0.6201|sec||NOT FIXED - Vulnerabilities in OpenSSH affect AIX|4.0.0.5200-6.0
openssl.base|1.0.2.800|sec||NOT FIXED - There is a vulnerability in OpenSSL used by AIX|1.0.2.500-1.
openssl.base|1.0.2.800|sec||NOT FIXED - Vulnerability in OpenSSL affects AIX|1.0.2.500-1.0.2.1100|CV
...
[root@aixtest:/home/admin]./flrtvc.ksh -v | pg
////////////////////////////////////////////////////////////
// IBM FLRTVC (v0.7.3) Report
// Server: aixtest
// Date: Fri Feb 9 10// Report by: root
// Vulnerable Filesets: 22
// Total Vulnerabilities: 54
// Total Fixes (not shown): 22
////////////////////////////////////////////////////////////

--------------------------------------------------------------------------------
bos.acct - 7.2.1.0 - Vulnerabilities (3)
--------------------------------------------------------------------------------

(1) NOT FIXED - (caccelstat) Vulnerabilities in bellmail / caccelstat / iostat / lquerypv / restbyinode / vmstat affect AIX (CVE-2017-1692)

Type: sec
Score: CVE-2017-1692:8.4
Versions: 7.2.1.0-7.2.1.0
APARs/CVEs: IV97811
Last Update: 02/05/2018
Bulletin: http://aix.software.ibm.com/aix/efixes/security/suid_advisory.asc
Download: ftp://aix.software.ibm.com/aix/efixes/security/suid_fix.tar
Fixed In: 7200-01-04

(2) NOT FIXED - (iostat) Vulnerabilities in bellmail / caccelstat / iostat / lquerypv / restbyinode / vmstat affect AIX (CVE-2017-1692)
Type: sec
Score: CVE-2017-1692:8.4
Versions: 7.2.1.0-7.2.1.1
APARs/CVEs: IV97898
Last Update: 02/05/2018
Bulletin: http://aix.software.ibm.com/aix/efixes/security/suid_advisory.asc
Download: ftp://aix.software.ibm.com/aix/efixes/security/suid_fix.tar
Fixed In: 7200-01-04
...

It really is a great tool, that can save us a lot of time when a vulnerability check is needed in our systems.

For a full usage of the tool:

Usage flrtvc: Change delimiter for compact reporting
 ./flrtvc.ksh -d '||'

Usage flrtvc: Generate full reporting (verbose mode)
 ./flrtvc.ksh -v

Usage flrtvc: Choose custom apar.csv file to use
 ./flrtvc.ksh -f myfile.csv

Usage flrtvc: Only show specific filesets in verbose mode
 ./flrtvc.ksh -vg printers

Usage flrtvc: Show only hiper results
 ./flrtvc.ksh -t hiper

Usage flrtvc: Custom lslpp and emgr outputs
 ./flrtvc.ksh -l lslpp.txt -e emgr.txt

Flags:

-d = Change delimiter for compact reporting
-f = Enter a custom aparCSV file in local filesystem
-q = Quiet mode, hide compact reporting header
-s = Skip download and locate 'apar.csv' filename in current directory
-v = Verbose, full report (for piping to email)
-g = Filter filesets for specific phrase, useful for verbose mode
-t = Type of APAR [hiper | sec]
-l = Enter a custom LSLPP output file, must match lslpp -Lqc
-e = Enter a custom EMGR output file, must match emgr -lv3
-x = Skip EFix processing
-a = Show all fixed and non-fixed HIPER/Security vulnerabilities.


Leave a comment

6.- Advanced logrotate for AIX

The most powerful facilities provided by logrotate are prerotate,postrotate & endscript, and we can make good use of this facilities to employ “complex” log rotation schedules.

We will do a logrotate setup to perform log rotation following IBM recommendations for AIX v7.2, and for that we can resort to the following KBs in IBM Support Knowledgecenter:

/ (root) overflow
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsrootover.htm

Resolving overflows in the /var file system
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.osdevice/fsvarover.htm

6.1.- logrotate failedlogin

IBM states in the official documentation, that failedlogin is a binary log, and therefore the utility who must be used to see it’s entries, as follows:

[root@aix72:/]who /etc/security/failedlogin
root vty0 Oct 20 18:03
UNKNOWN_ vty0 Oct 20 18:03
UNKNOWN_ ssh Oct 26 10:22 (10.1.15.12)
root ssh Oct 26 10:22 (10.1.15.12)
root ssh Nov 03 10:38 (10.20.30.129)
root ssh Nov 05 11:49 (srv.dom.myanet)
root ssh Nov 12 11:03 (tsmsrv)
root vty0 Nov 19 07:47
root ssh Nov 21 10:25 (10.20.120.72)
root ssh Jan 25 05:37 (10.20.130.229)
tsminst1 ssh Feb 13 10:03 (loopback)
tsminst1 ssh Feb 13 10:05 (loopback)
root ssh Feb 27 16:45 (10.1.15.214)
root pts/2 Mar 07 15:11 (10.1.165.159)
UNKNOWN_ pts/2 Mar 07 15:11 (10.1.165.159)
...

So we check the file and it’s permissions:

[root@aix72:/]ls -l /etc/security/failedlogin
-rw-rw---- 1 root system 14256 Mar 07 15:15 /etc/security/failedlogin

And we take note of the 660 access rights and user root group system.

Well, we can see that this is an interesting log to keep (and also a log that can grow large in size if there is a problem and we have a lot of terminal users), so using logrotate we can just rotate it by size, say 5 MB each log, keep 3 copies (keep the original log and rotate 2 more versions) and also keep it compressed, so we write the following file: /etc/logrotate.d/failedlogin

# logrotate config for failedlogin which logs failed login sessions in binary form, can be used to detect brute-force attacks. Read with "who".
/etc/security/failedlogin {
  size 5M
  compress
  rotate 2
  create 660 root system
}

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/failedlogin
reading config file /etc/logrotate.d/failedlogin

Handling 1 logs

rotating pattern: /etc/security/failedlogin forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
log does not need rotating

NOTE: If we want to force this rotation, then we could change the size 5M for size 10k, in which case the file will be rotated as it is 14k in size.

6.2.- logrotate wtmp

IBM states in the official documentation, that wtmp is a binary log, and therefore IBM recommends using the utility fwtmp to convert the binary log to an ASCII log, as follows:

Export the wtmp log to an ASCII copy called /tmp/wtmp-delete.me:

[root@aix72:/]/usr/sbin/acct/fwtmp < /var/adm/wtmp > /tmp/wtmp-delete.me

Now we can just read the file /tmp/wtmp-delete.me with vi, emacs, nano, or whichever editor we fancy:

[root@aix72:/]cat /tmp/wtmp-delete.me
clcomd   clcomd                       5 7733518 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
xmdaily  xmdaily                      5 8192296 0000 0000 1488218371                                  Mon Feb 27 11:59:31 CST 2017
ctrmc    ctrmc                        5 10944850 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
spectrum spectrum                     5 11075926 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
tsmcc    tsmcc                        5 11141464 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
ha_star  ha_star                      5 11010388 0000 0000 1488218371                                 Mon Feb 27 11:59:31 CST 2017
         pts/1          pts/1         8 8454482 0000 0000 1488245570                                  Mon Feb 27 19:32:50 CST 2017
root     pts/0          pts/0         7 15532358 0000 0000 1488305060 10.1.165.159                    Tue Feb 28 12:04:20 CST 2017
root     pts/1          pts/1         7 15860100 0000 0000 1488305690 10.1.165.159                    Tue Feb 28 12:14:50 CST 2017
root     pts/2          pts/2         7 15663394 0000 0000 1488323898 10.1.165.159                    Tue Feb 28 17:18:18 CST 2017
 ...

This log tells us all valid terminal logins (loging, rlogins and telnet), date and time, which terminal they used and their remote IP. So it is also an interesting log to keep. (and can also be read by who, try ” who /var/adm/wtmp “)

Now to check the file and it’s permissions:

[root@aix72:/]ls -l /var/adm/wtmp
 -rw-rw-r-- 1 adm adm 846288 Mar 07 15:12 /var/adm/wtmp

Aha, this one has different attributes from the one on the step 6.1, we take note of the 664 access rights and user adm group adm.

NOTE: If we wanted to process wtmp log, say to keep the last 1000 lines, and truncating the rest, we could process the ASCII log and convert back to the /var/adm/wtmp binary log using the utility fwtmp like this:

/usr/bin/tail -1000 /tmp/wtmp-delete.me | /usr/sbin/acct/fwtmp -ic > /var/adm/wtmp

Not forgetting to cleanup the temp file afterwards:

rm /tmp/wtmp-delete.me

This used to be a common practice to keep wtmp log under control in AIX, but this is not really needed anymore, as using logrotate we can just rotate wtmp by size, say 5 MB each log, keep 2 copies (keep the original log and rotate 1 more version), so we write the following file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 size 5M
 rotate 1
 create 664 adm adm
 }

And now we check that we don’t have any typos or problems on the logrotate config of the file just created:

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 considering log /var/adm/wtmp
 error: skipping "/var/adm/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Hey!!!, this time we have an error! and this is because the parent directory group is different from root (so it expects to be told the right ones to use on the “su” directive ), so we check this and change our config file accordingly, and we’ll try again:

[root@aix72:/home/admin]ls -la /var/adm
 total 4960
 drwxrwxr-x 15 root adm 4096 Mar 03 19:25 . <-- it's owned by root/adm

Add the su directive to the config file:

# logrotate config for wtmp which logs all logins, rlogins and telnet sessions in binary form
 /var/adm/wtmp {
 su root adm
 size 5M
 rotate 2
 create 664 adm adm
 }
[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/wtmp
 reading config file /etc/logrotate.d/wtmp

Handling 1 logs

rotating pattern: /var/adm/wtmp forced from command line (1 rotation)
 empty log files are rotated, old logs are removed
 switching euid to 4 and egid to 4
 considering log /var/adm/wtmp
 log does not need rotating
 switching euid to 0 and egid to 0

This time, it has worked fine, and also it tells us that it has switched user to userid 0 / groupid 4 and then back to the original (0/0). So now we know more things about logrotate’s output. Good.

Now we also know that we need to check the owners on the log’s parent directory as well in case we need to add the su directive, and we will from now on.

6.3.- logrotate errlog6.3.- logrotate errlog

Errlog is a binary circular log, and therefore cannot really be rotated. So for this log in particular is not recommended to be managed using logrotate.

By default, root’s crontab comes with the following 2 entries to trim old HW errors:

0 11 * * * /usr/bin/errclear -d S,O 30 <-- delete Software & errlogger msgs (Other) older than 30 days
0 12 * * * /usr/bin/errclear -d H 90   <-- delete Hardware errors older than 90 days

Error messages get overwritten by new ones as they are generated, and there are system admins that do not even use errclear to trim the log, since they prefer to keep an historical of all theHW errors ever generated on one server.

However in our days with virtualized hardware, it’s not that relevant to keep all errors ever registered, and I find a good practice to enable the default errclear entries on crontab, as well as skulker to clean old log & temp system files.

Errlog can however be increased if need be, to see the actual errlog size:

[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes				<-- 1MB default value
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

To increase the size:

[root@aix72:/]/usr/lib/errdemon -s 2097152
[root@aix72:/]/usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                2097152 bytes				<-- increased to 2MB
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds
Duplicate Error Maximum 1000
PureScale Logging       off
PureScale Logstream     CentralizedRAS/Errlog

NOTE: The errlog daemon ( /usr/lib/errdemon ) can be stopped using the command  ( /usr/lib/errstop ), and one VERY important thing to take into account is that the errlog should not be zeroed (a procedure often used to clear logs), otherwise the daemon would not start. So, let’s try it to see what would happen :o)

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog /var/adm/ras/errlog.bak <-- we do a backup of it first!!!

[root@aix72:/var/adm/ras]> /var/adm/ras/errlog       <-- we zero the file

[root@aix72:/var/adm/ras]/usr/lib/errdemon	     <-- and we try to start the errdemon...
0315-180 logread: UNEXPECTED EOF		     <-- the only entry in the log is the End Of File (it is a zero file)
0315-171 Unable to process the error log file /var/adm/ras/errlog.
errdemon:
0315-001 Failure to open the logfile '/var/adm/ras/errlog' for writing.
Possible causes are:
1. The file exists but the invoking process does not have write
   permission.
2. The file exists but the directory '/var/adm/ras' does not have write
   permission.
3. The file exists but it is not a valid error logfile.  Remove

4. The file does exist and the directory ‘/var/adm/ras’ does not have enough
space available. The minimum logfile size is 8192 bytes.

[root@aix72:/var/adm/ras]cp -p /var/adm/ras/errlog.bak /var/adm/ras/errlog <– restore the backup to be able to start the errdemon again

6.4.- logrotate sulog

Now we will do rotation for sudo’s log ( sulog ):

/var/adm/sulog

We do file, permission & parent directory checks :

[root@aix72:/]ls -l /var/adm/sulog
-rw-------    1 root     system         4693 Mar 03 18:24 /var/adm/sulog
[root@aix72:/]ls -la /var/adm/.
total 5208
drwxrwxr-x   15 root     adm            4096 Mar 20 22:36 .

It has 600 access rights, user root group system, and parent directory has user root group adm, so we will have to use the “su” directive

# logrotate config for sudo's log (sulog).
/var/adm/sulog {    
  su root adm    
  size 5M    
  compress    
  rotate 2    
  create 600 root system
}

[root@aix72:/]/usr/sbin/logrotate -vf /etc/logrotate.d/sulog
reading config file /etc/logrotate.d/sulog
Handling 1 logs
rotating pattern: /var/adm/sulog  forced from command line (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/adm/sulog 
 log needs rotating
rotating log /var/adm/sulog, log->rotateCount is 2
dateext suffix '-20170320'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/adm/sulog.2.gz to /var/adm/sulog.3.gz (rotatecount 2, logstart 1, i 2),
old log /var/adm/sulog.2.gz does not exist
renaming /var/adm/sulog.1.gz to /var/adm/sulog.2.gz (rotatecount 2, logstart 1, i 1),
old log /var/adm/sulog.1.gz does not exist
renaming /var/adm/sulog.0.gz to /var/adm/sulog.1.gz (rotatecount 2, logstart 1, i 0),
old log /var/adm/sulog.0.gz does not exist
log /var/adm/sulog.3.gz doesn't exist -- won't try to dispose of it
renaming /var/adm/sulog to /var/adm/sulog.1
creating new /var/adm/sulog mode = 0600 uid = 0 gid = 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 4
switching euid to 0 and egid to 0

6.5.- logrotate syslog

Syslog is an AIX classic, and the only log churner that got upgraded a while ago and has it’s own “decent” log rotation & compression configuration since at least AIX version 6.1, therefore the recommended log rotation method is to use syslog’s own configuration file and not logrotate’s.

For the official documentation on syslog from AIX v7.2, you can refer to:
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds5/syslogd.htm

To apply a weekly logrotation, with 5 log versions, and compression enabled, just edit the /etc/syslog.conf file and modify the entry *.info for the following:

 *.info /var/log/syslog.log rotate time 1w files 5 compress     #weekly rotation, 5 files, compressed

Just remember to refresh the daemon after making the modifications in the config file:

[root@aix72:/] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.

Well chaps, I think that it is now enough logrotate for AIX now. There are examples of common logrotate configs that I will like to post on the future, but for now, I think that this is a good end of series for Logrotate.

AS always: Thanx for reading!


2 Comments

Logrotate 4 & 5.- Support & Common Errors

NOTE:  This is a follow-up, from the previous post: Logrotate 3.- Logrotate checks

4.- Logrotate Support

Disclaimer (IBM Unsupported):  IBM stand on opensource utilities is that they are not directly supported by IBM, this is IBM Support’s page for logrotate (dated 06 June 2011):

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012796

So, IBM will not provide any PMR Support on Open Source Software (and this is completely logical, as it’s not an IBM product), but still, you can get community based support at the developerWorks pages, and for this forum-based support, you can go to:

IBMDeveloperWorks: Forum Directory >‎ dW >‎ AIX and UNIX >‎ Forum: AIX Open Source Software

And in that forum, exists an specific YUM topic:

IBMDeveloperWorks: Forum Directory >‎ dW >‎ AIX and UNIX >‎ Forum: AIX Open Source Software >‎ Topic: yum for AIX Toolbox

5.- Fixing logrotate errors

5.1.- config file logrotate.conf errors

[root@aix72:/home/admin]logrotate -vf /etc/logrotate.conf
error: cannot stat /etc/logrotate.conf: A file or directory in the path name does not exist.

Cannot stat means that the config file is NOT FOUND, so revise that /etc/logrotate exists and has the right access rights (if it doesn’t, look for it in /opt/freeware/etc and copy it to /etc)

5.2.- config directory logrotate.d errors

[root@aix72:/home/admin]logrotate -vf /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
error: cannot stat /etc/logrotate.d: A file or directory in the path name does not exist.
removing last 0 log configs

Cannot stat means that the config directory is NOT FOUND, so revise that /etc/logrotate.d exists and has the right access rights (if it doesn’t, look for it in /opt/freeware/etc and copy it to /etc)

5.3.- files in directory logrotate.d errors

[root@aix72:/opt/freeware/etc]logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file yum
error: yum:6 unknown group 'root'
error: found error in /var/log/yum.log , skipping
removing last 1 log configs
error: /etc/logrotate.conf:23 unknown group 'utmp'
error: found error in /var/log/wtmp , skipping
removing last 1 log configs
error: /etc/logrotate.conf:31 unknown group 'utmp'
error: found error in /var/log/btmp , skipping
removing last 1 log configs

Handling 0 logs

This errors are usually caused at installation time of logrotate in AIX, since the config files require some modifications:

error: yum:6 unknown group 'root'
error: found error in /var/log/yum.log , skipping

It complains against the line 6 of /etc/logrotate.d/yum file, since in AIX there isn’t a “root” group, it is “system“, so modify the file:

/var/log/yum.log {  
  missingok 
  notifempty 
  size 30k 
  yearly 
  create 0600 root root 
}

for the file:

/var/log/yum.log {
  missingok
  notifempty
  size 30k
  yearly
  create 0600 root system
}
error: /etc/logrotate.conf:23 unknown group 'utmp' 
error: found error in /var/log/wtmp , skipping

It complains against the line 23 of /etc/logrotate.conf file, since in AIX there isn’t a “utmp” group, and in fact wtmp is not located in /var/log/wtmp, but in /var/adm/wtmp but in any case, we can just refer to the steps in 2.1 to fix it by deleting the wtmp lines in /etc/logrotate.conf.

error: /etc/logrotate.conf:31 unknown group 'utmp'
error: found error in /var/log/btmp , skipping

It complains against the line 31 of /etc/logrotate.conf file, since in AIX there isn’t a “utmp” group, and in fact AIX does not have a btmp, so we can just refer to the steps in 2.1 to fix it by deleting the wtmp lines in /etc/logrotate.conf.

 

That covers the most common Logrotate config errors in AIX. I’m sure that you will find some more obscure ones to entertain yourself with, as it is often the case!

On the next post, it will be time for step 6.- Advanced Logrotate for AIX.  See you then, and thanks for reading!


1 Comment

Logrotate 3.- Logrotate checks

NOTE:  This is a follow-up, from the previous post:  Logrotate 2.- Configure logrotate for AIX

To check that logrotate is configured and working OK, all we need to do is call logrotate from the command line telling it to verbose it’s internal checks ( -v ) and to check the config file ( /etc/logrotate.conf ), like the following:

[root@aix72:/home/admin]/usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d      
reading config file failedlogin 
reading config file sysadmin    
reading config file wtmp        
reading config file yum         

Handling 6 logs

rotating pattern: /etc/security/failedlogin 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /etc/security/failedlogin
 log does not need rotating        

rotating pattern: /home/admin/log/check_all.log 1048576 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log 
 log does not need rotating         

rotating pattern: /var/adm/wtmp 5242880 bytes (2 rotations)
empty log files are rotated, old logs are removed
switching euid to 4 and egid to 4
considering log /var/adm/wtmp 
 log does not need rotating   
switching euid to 0 and egid to 0

rotating pattern: /var/log/yum.log yearly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log 
 log does not need rotating      

Once we have checked that the config is OK, we can check the rotation by Forcing rotation with the -f or –force flag:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.conf
 reading config file /etc/logrotate.conf
 including /etc/logrotate.d
 reading config file failedlogin
 reading config file sysadmin
 reading config file wtmp
 reading config file yum
 Handling 6 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
 empty log files are rotated, old logs are removed
 considering log /home/admin/log/stop_all.log
 log does not need rotating

rotating pattern: /var/log/yum.log forced from command line (4 rotations)
 empty log files are not rotated, old logs are removed
 considering log /var/log/yum.log log needs rotating rotateCount is 4
 dateext suffix '-20170226'
 glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
 glob finding old rotated logs failed
 renaming /var/log/yum.log to /var/log/yum.log-20170226
 creating new /var/log/yum.log mode = 0600 uid = 0 gid = 0

Logrotate is configured OK and it seems to work fine, so if it’s not executing properly, we will have to check it’s schedule on the crontab.

NOTE: Notice that when we configure the rotation to be on size, the –force option cannot force this rotation, so to force rotation on stanzas where size has been used, just lower the size attribute temporarily (size 10k instead of 5M, for example).

3.1- Logrotate individual files/logs check

To check logrotate’s config for a particular file, we will have to identify it first in the /etc/logrotate.d directory, for example to check the config for yum’s logs:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/yum
reading config file /etc/logrotate.d/yum

Handling 1 logs

rotating pattern: /var/log/yum.log forced from command line (no old logs will be kept)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log
 log does not need rotating

To check the config for a specific log, but we don’t see a logrotate file stored by its name in /etc/logrotate.d, we will have to dig it out (for example let’s look for start_all.log):

[root@aix72:/home/admin]grep start_all.log /etc/logrotate.d/*
/etc/logrotate.d/sysadmin:/home/admin/log/start_all.log

OK, so it looks like the logrotate config for start_all.log resides in the /etc/logrotate.d/sysadmin file, so now we can check it:

[root@aix72:/etc/logrotate.d]logrotate -vf /etc/logrotate.d/sysadmin
reading config file /etc/logrotate.d/sysadmin

Handling 3 logs

rotating pattern: /home/admin/log/check_all.log forced from command line (2 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/check_all.log
 log does not need rotating

rotating pattern: /home/admin/log/start_all.log forced from command line (1 rotations) 
empty log files are rotated, old logs are removed
considering log /home/admin/log/start_all.log
 log does not need rotating

rotating pattern: /home/admin/log/stop_all.log forced from command line (1 rotations)
empty log files are rotated, old logs are removed
considering log /home/admin/log/stop_all.log
 log does not need rotating

So, as always, an important part of a configuration (the most important, actually) is to check that our new config works just as we expected it.

And now we have seen how to check all the logrotate config, how to force the log rotation, and how to check individual logrotate config files, so with this three checks we should be able to perform config-test-change-retest until our friend logrotate does what we expect it to.

On the step 4, I will talk about logrotate documentation & support, and step 5 will show how to fix common logrotate errors. See you soon.