Comparison of Unix log file management tools by Dusan Baljevic, December 2008

1. Unity and Disunity of
Unix Log File
Management Tools
Dusan Baljevic
Sydney, Australia
© 2008 Dusan Baljevic The information contained herein is subject to change without notice

2. Unix Log Files and Their Management
Tools - Present
• Most of the time, admins rely on Shell or Perl scripts
• As well, find command is commonly used for cleanups
March 1, 2014
Webinar - Dusan Baljevic
2

3. Unix Log Files and Their Management
Tools
Solaris logadm
Linux
logrotate
AIX
(built-in log file rotation and compression)
HP-UX (various tools)
March 1, 2014
Webinar - Dusan Baljevic
3

4. AIX syslog
• AIX has built-in log file rotation and compression. They are optional
fields
• Format
msg_src_list destination [rotate [size sizek|m] [files files] [time timeh|d|w|
m|y] [compress] [archive archive]]
msg_src_list is a semicolon separated list of facility.priority
facility all (except mark)
mark - time marks kern,user,mail,daemon, auth,...
priority is one of (from high to low):
emerg/panic,alert,crit,err(or),warn(ing),notice,info,debug
(meaning all messages of this priority or higher)
destination is:
/filename - log to this file
username[,username2...] - write to user(s)
@hostname - send to syslogd on this machine
March 1, 2014
Webinar - Dusan Baljevic
* - send to all logged in users
4

5. AIX syslog Features (part 1/2)
• If destination is a regular file and the word rotate is specified, then the
destination is limited by either size or time, or both. The backup filenames
are created by appending a period and a number to destination, starting
with .0. The time value causes the destination to be rotated after time. If
both time and size are specified, then logfiles will be rotated once the
logfile size exceeds size or the after time, whichever is earlier
• If the compress option is specified then the logfile names will be
generated with a .Z extension. The files keyword will be applicable to the
logfiles which are currently under rotation. For example, if we specify the
compress option, then only file with .Z extension will be under rotation and
the number of such files will be limited byfiles files. Any logfiles with an
extension other than .Z will not be under the rotation scheme and thus will
not be under the restriction of files files. Similarly if the compress option is
removed then the files which have been generated with .Z extension will
no longer be the part of rotation scheme and will not be limited by the files
files 1, 2014
March
Webinar - Dusan Baljevic
5

6. AIX syslog Features (part 2/2)
• The minimum size that can be specified is 10k. The minimum number of
files that can be specified is 2. The default size is 1MB and the default for
files is unlimited. Therefore, if only rotate is specified, the log will be
rotated with size = 1m. The compress option means that rotated log files
that are not in use will be compressed. The archive option will save
rotated log files that are not in use to archive. The default is not to rotate
log files
• The letter indicating the unit must immediately follow the number in the
syntax. For example, to specify the log rotation of every two days, the
phrase time “2d” is correct, but “2 d” is not
March 1, 2014
Webinar - Dusan Baljevic
6

7. AIX /etc/syslog.conf
• Example /e tc /s y s lo g . c o nf
*.info /var/adm/syslog/syslog.log
*.alert /var/adm/syslog/syslog.log
*.notice /var/adm/syslog/syslog.log
*.warning /var/adm/syslog/syslog.log
*.err /var/adm/syslog/syslog.log
*.crit /var/adm/syslog/syslog.log rotate time 1d files 9
daemon.debug /var/adm/ftpd.log rotate size 1024k files 5
March 1, 2014
Webinar - Dusan Baljevic
7

8. Linux logrotate
• It allows automatic rotation, compression, removal, and mailing of log
files. Each log file may be handled daily, weekly, monthly, or when it
grows too large.
Normally, lo g ro ta te is run as a daily cron job. It will not modify a log
multiple times in one day unless the formula for that log is based on the
logs size and lo g ro ta te is being run multiple times each day, or unless
the “-f” or “-fo rc e ” option is us e d .
• Cro n jo b /e tc /c ro n. d a ily /lo g ro ta te
#!/bin/sh /usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with
[$EXITVALUE]"
fi
March 1,
Webinar - Dusan Baljevic
exit 0 2014
8

9. Linux /etc/logrotate.conf
weekly
rotate 4
create
dateext
include /etc/logrotate.d
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}
March 1, 2014
Webinar - Dusan Baljevic
9

10. Linux logrotate Command Usage
Usage: logrotate [OPTION...] <configfile>
-d, --debug
Don't do anything, just test (implies -v)
-f, --force
Force file rotation
-m, --mail=command Command to send mail (instead of `/bin/mail')
-s, --state=statefile
Path of state file
-v, --verbose
Display messages during rotation
Help options:
-?, --help
--usage
March 1, 2014
Show this help message
Display brief usage message
Webinar - Dusan Baljevic
10

11. Linux /etc/logrotate.d Directory
# ls /e tc /lo g ro ta te . d
bittorrent
fail2ban
squid
yum
nagios
samba
collectl
mgetty
syslog
cups
setroubleshoot tux
psacct
snmpd
March 1, 2014
munin-node
rpm
clamav-update httpd
squidGuard
zabbix
named
sa-update
mimedefang
ppp
dirmngr
munin
wpa_supplicant
Webinar - Dusan Baljevic
11

12. Linux /etc/logrotate.d Example
# c a t /e tc /lo g ro ta te . d /http d
/var/log/httpd/*log {
missingok
notifempty
sharedscripts
postrotate
kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null
|| true
endscript
}
March 1, 2014
Webinar - Dusan Baljevic
12

13. Solaris logadm
• Starting from Solaris 9, there is a standard tool, called lo g a d m , to rotate
logs
• lo g a d m is an independent utility (unlike integrated in syslog daemon
capability that can be found in AIX). The lo g a d m command is a
preconfigured entry in the default crontab file supplied starting with
Solaris 9
•/us r/lib/ne ws y s lo g script is no longer used
• Before Solaris 9 there was FreeBSD-style tool ne ws y s lo g located in
/usr/lib and Perl script ro ta te lo g . It was run from cron. For Solaris 8 and
earlier download the tar.gz file, untar it, go in the new directory, and
execute m a ke ins ta ll. It will install /us r/lo c a l/s bin/ro ta te lo g and
/us r/lo c a l/e tc /ro ta te lo g . c o nf
March 1, 2014
Webinar - Dusan Baljevic
13

14. Solaris /etc/logadm.conf (part 1/2)
lo g a d m . c o nf specifies the schedule for log rotation and
options with which rotation will be performed. The default
configuration:
/var/log/syslog -C 8 -P 'Sun Sep 14 17:10:00 2008' -a 'kill
-HUP `cat /var/run/syslog.pid`‘
/var/adm/messages -C 4 -P 'Fri Sep 12 17:10:00 2008' -a 'kill
-HUP `cat /var/run/syslog.pid`'
/var/cron/log -P 'Fri Aug 22 17:10:00 2008' -c -s 512k -t
/var/cron/olog
/var/lp/logs/lpsched -C 2 -N -t '$file.$N'
March 1, 2014
Webinar - Dusan Baljevic
14

15. Solaris /etc/logadm.conf (part 2/2)
/var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv
/var/fm/fmd/errlog.0- $nfile' -N -s 2m
smf_logs -C 8 -s 1m /var/svc/log/*.log
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm
-m 664 -o adm -p never
/var/log/pool/poold -N -a 'pkill -HUP poold; true' -s 512k
/var/fm/fmd/fltlog -A 6m -M '/usr/sbin/fmadm -q rotate fltlog
&& mv /var/fm/fmd/fltlog.0- $nfile' -N -s 10m
March 1, 2014
Webinar - Dusan Baljevic
15

16. Solaris logadm Command Usage (part
1/3)
Usage: logadm [options]
(processes all entries in /etc/logadm.conf or conffile given by -f)
or: logadm [options] logname... (processes the given lognames)
General options:
-e mailaddr mail errors to given address
-f conffile
use conffile instead of /etc/logadm.conf
-h
display help
-N
not an error if log file nonexistent
-n
show actions, don't perform them
-r
remove logname entry from conffile
-V
ensure conffile entries exist, correct
-v
print info about actions happening
-w entryname write entry to config file
March 1, 2014
Webinar - Dusan Baljevic
16

17. Solaris logadm Command Usage (part
2/3)
Options which control when a logfile is rotated:
(default is: -s1b -p1w if no -s or -p)
-p period
only rotate if period passed since last rotate
-P timestamp used to store rotation date in conffile
-s size
only rotate if given size or greater
Options which control how a logfile is rotated:
(default is: -t '$file.$n', owner/group/mode taken from log file)
-a cmd
execute cmd after taking actions
-b cmd
execute cmd before taking actions
-c
copy & truncate logfile, don't rename
-g group
new empty log file group
-l
rotate log file with local time rather than UTC
-m mode
new empty log file mode
-M cmd
execute cmd to rotate the log file
-o owner
new empty log file owner
-R cmd
run cmd on file after rotate
-t template template for naming old logs
-z count
gzip old logs except most recent count
March 1, 2014
Webinar - Dusan Baljevic
17

18. Solaris logadm Command Usage (part
3/3)
Options which control the expiration of old logfiles:
(default is: -C10 if no -A, -C, or -S)
-A age
expire logs older than age
-C count
expire old logs until count remain
-E cmd
run cmd on file to expire
-S size
expire until space used is below size
-T pattern pattern for finding old logs
March 1, 2014
Webinar - Dusan Baljevic
18

19. Solaris logadm and Timezone
• By default, lo g a d m works in GMT. All entries written to
the /e tc /lo g a d m . c o nf file will have a GMT timestamp
• Use the “-l” option to set lo g a d m to local time
March 1, 2014
Webinar - Dusan Baljevic
19

20. Solaris logadm – Example for wtmpx
• Add into /e tc /lo g a d m . c o nf
/var/adm/utmpx -C 12 -P ‘Mon Oct 13 17:00:00 2008' -s 100m -z 0
"-C 12" means it will preserve 12 versions of the log file
"-P .." means when to first start processing the log file
"-s 100m" defines the maximum size of the log file before it is rotated
"-z 0" sets the gzip compression
• Run command:
# lo g a d m
• Check it:
# lo g a d m -V
March 1, 2014
Webinar - Dusan Baljevic
20

21. HP-UX 11i syslogd
• HP-UX
11i v1 and earlier do not have log file automation
• HP-UX 11.23 and later s y s lo g d logs messages into a set of
files. Once the size of a log file reaches 2 GB, syslogd stops
logging to that file. Configure the maximum size of syslogd
log files by setting the variable LOG_SIZE in
/e tc /d e fa ult/s y s lo g d
The value of LOG_SIZE can be any positive integer greater
than 2, representing the maximum size of the file in GB.
When LOG_SIZE=NOLIMIT, syslogd uses the limit imposed
by the file system on file size
March 1, 2014
Webinar - Dusan Baljevic
21

22. HP-UX 11i syslogd Simple Rotation
# /s bin/init. d /s y s lo g d s to p
# /sbin/init.d/syslogd start
It will rename s y s lo g . lo g to O LDs y s lo g . lo g in
/v a r/a d m /s y s lo g directory.
March 1, 2014
Webinar - Dusan Baljevic
22

23. HP-UX 11i Other RC Cleanups
• /e tc /rc . c o nfig . d /c le a n_ tm p for /tm p cleanup at boot
CLEAR_TMP=1
• /e tc /rc . c o nfig . d /c le a n
CLEAN_ADM=1
CLEAN_UUCP=1
/va r/a d m /s ulo g /va r/a d m /d ia g lo g /va r/a d m /m e s s a g e s
renamed to OLD*
• /e tc /rc . c o nfig . d /c le a n_ uuc p
CLEAN_UUCP=1
March 1, 2014
uuclean(1m) at boot
Webinar - Dusan Baljevic
23

24. HP-UX 11i Examples of Log Directories
and Files
• Examples
of log files that can grow out of bounds:
/va r/s p o o l/lp
/va r/a d m /lp
/va r/o p t/p e rf/d a ta file s
lo s t+ fo und directories in top-level of each file system
/va r/a d m /d ia g
/va r/o p t/ig nite
/va r/s tm /lo g s /s y s *
/var/adm/wtmp
/var/adm/wtmps
/var/adm/btmp
/var/adm/sw
(others truncated for the sake of brevity)
March 1, 2014
Webinar - Dusan Baljevic
24

25. HP-UX 11i auto_parms.log
• /e tc /a uto _ p a rm s . lo g is updated by auto_parms(1m)
command that handles first-boot configuration
(setting of unique system “initial identity
parameters”), and ongoing management of DHCP
leases
• auto_parms(1m) saves old copy into
/e tc /a uto _ p a rm s . lo g . o ld
March 1, 2014
Webinar - Dusan Baljevic
25

26. HP-UX 11i rc.log
• Run Command (RC) scripts update /e tc /rc . lo g at
boot time
• At reboot, previous version of /e tc /rc . lo g is
renamed to /e tc /rc . lo g . o ld
March 1, 2014
Webinar - Dusan Baljevic
26

27. HP-UX 11i EMS Logs
• The EMS log files in /e tc /o p t/re s m o n/lo g are limited to 500
KB in size and are then moved to <logfile>.old. The previous
*.old gets lost
• The limit of 500 KB per logfile can be removed by creating
the file /e tc /o p t/re s m o n/unlim ite d _ lo g
• Be careful with creating the unlim ite d _ lo g . Growing EMS log
files can easily fill up root file system
March 1, 2014
Webinar - Dusan Baljevic
27

28. HP-UX 11i Glance and MeasureWare
Logs
• /va r/o p t/p e rf/p a rm is read by both the GlancePlus product
and the MeasureWare products. Glance uses only the
Application definitions
size global=10, application=10, process=20, device=10, transaction=10
The sizes are in MB
• The logfiles are stored in /va r/o p t/p e rf/d a ta file s directory
March 1, 2014
Webinar - Dusan Baljevic
28

29. HP-UX 11i Integrity VM Driver Log File
• /va r/o p t/hp vm /c o m m o n/hp vm _ m o n_ lo g is limited to 1024
KB by default. When the log file grows larger than this, it is
copied to a new file (hp vm _ m o n_ lo g . $ tim e ) and an empty
one is created for the new log
• To allow this log file to grow larger than 1024 KB, include
the following line in /e tc /rc . c o nfig . d /hp vm c o nf
VMMLOGSIZE=10420
# In KB
Then, restart the daemon:
# kill – HUP ` c a t /va r/run/hp vm m o nlo g d . p id `
March 1, 2014
Webinar - Dusan Baljevic
29

30. HP-UX 11i Integrity VM Guest Log File
• /va r/o p t/hp vm /g ue s ts /g ue s t_ na m e /lo g file records
guest start and stop information. These log files can
grown very large
To close the current log file, rename it, and open a
new one:
# hp vm c o ns o le re c -ro ta te
March 1, 2014
Webinar - Dusan Baljevic
30

31. HP-UX 11i SMH *
# c a t /o p t/hp s m h/c o nf. c o m m o n/s m hp d . x m l
<?xml version="1.0" encoding="UTF-8"?>
<system-management-homepage>
<admin-group></admin-group>
<operator-group></operator-group>
<user-group></user-group>
<allow-default-os-admin>True</allow-default-os-admin>
<anonymous-access>False</anonymous-access>
<localaccess-enabled>False</localaccess-enabled>
<localaccess-type>Anonymous</localaccess-type>
<trustmode>TrustByCert</trustmode>
<xenamelist></xenamelist>
<ip-restricted-logins>False</ip-restricted-logins>
<ip-restricted-include></ip-restricted-include>
<ip-restricted-exclude></ip-restricted-exclude>
<ip-binding>False</ip-binding>
<ip-binding-list></ip-binding-list>
<rotate-logs-size>N</
rotate-logs-size>
</system-management-homepage>
March 1, 2014
Webinar - Dusan Baljevic
31

32. HP-UX 11i TCB Auditing
• It records instances of access by subjects to objects and allows detection of any
(repeated) attempts to bypass the protection mechanism and any misuses of
privileges
• a ud s y s allows the user to start or halt the auditing system, to specify the auditing
system "current" and "next" audit files (and their switch sizes), or to display
auditing system status information. The "current" audit file is the file to which the
auditing system writes audit records. When the "current" file grows to either its
Audit File Switch (AFS) size or its File Space Switch (FSS) size (see a ud o m o n),
the auditing system switches to write to the "next“ audit file
# audsys
Auditing system is currently on
current file: /var/adm/audit/audfile1
next file: /var/adm/audit/audfile2
statisticsafs Kb used Kb avail % fs Kb
used Kb avail %
current file: 10000
0
100 4825088 963704
80
next file: 10000
0
100 4825088 963704
80
March 1, 2014
Webinar - Dusan Baljevic
32

33. HP-UX 11i HIDS
HIDS log files increase rapidly. However, the Configuration Change Console
agent keeps log files truncated to save disk space. To ensure that the log files do
not increase in file size while the agent is not running, run a script to periodically
truncate the HIDS log files.
A sample script to manage HIDS log files is provided. This script should be run
from the crontab:
#!/bin/s h
file s iz e = ` /bin/ls -l /va r/o p t/id s /a le rt. lo g | /bin/a wk '{p rint $ 5 }'`
if [ " $ file s iz e " -g t " 5 0 0 0 0 0 0 " ]
the n
m v /v a r/o p t/id s /a le rt. lo g /va r/o p t/id s /a le rt. lo g _ De c _ 2 0 0 8
fi
rm /va r/o p t/id s /id s _ 1 *
Sample entry to configure the crontab to run every hour where the bold letters are
replaced by the actual path of the trunclog.sh file:
0 * * * * /<location of script>/trunclog.sh 2>/dev/null 2>&1
March 1, 2014
Webinar - Dusan Baljevic
33

34. HP-UX 11i ServiceGuard Package Log
File
SCRIPT_LOG_FILE (SG 11.17+) A new package attribute
that allows a name to be assigned to a package log file
Necessary for support of multiple packages sharing a
common package control script
Legacy Package Configuration
SCRIPT_LOG_FILE /e tc /c m c lus te r/p kg a /p kg a . lo g
Modular Package Configuration
script_log_file
$ SG RUN g /$ SG _ PA
/lo
CKA E. lo g
G
March 1, 2014
Webinar - Dusan Baljevic
34

35. HP-UX 11i ulimit
# ulim it -a
time(seconds)
unlimited
file(blocks)
unlimited
data(kbytes)
1048576
stack(kbytes)
8192
memory(kbytes)
unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048
March 1, 2014
Webinar - Dusan Baljevic
35

36. HP-UX 11i v3 coreadm *
# c o re a d m
global core file pattern:
init(1M) core file pattern:
global core dumps:
disabled
per-process core dumps:
enabled
global setid core dumps:
disabled
per-process setid core dumps: disabled
March 1, 2014
Webinar - Dusan Baljevic
36

37. HP-UX cleanup - HP-UX patch cleanup
utility
# c le a nup – c 1
The cleanup command provides functions useful when
dealing with HP-UX patches.
The cleanup command logs all information to
/var/adm/cleanup.log.
March 1, 2014
Webinar - Dusan Baljevic
37

38. HP-UX savecrash utility
/etc/rc.config.d/savecrash
CHUNK_SIZE Size of single crash image file (how big you want each of
image.n.x, image.n.x+1, etc. to be).
If not specified, savecrash will choose one based on the
physical memory size of the system. Can be specified in
bytes (b), kilobytes (k), megabytes (m), or gigabytes (g).
The default unit is KB.
See savecrash(1M) “–s” option for size constraints.
COMPRESS:
March 1, 2014
Whether you want the kernel and crash image files to be
compressed.
Webinar - Dusan Baljevic
38

39. HP-UX Alternative Log File Tools
(part 1/3)
• Old
but maybe still applicable bundle (needs to be tested):
http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/logrotate-2.5/
• Shell script logrotate:
http://iain.cx/src/logrotate/
• Scripts based on Perl modules like Logfile-Rotate
•Perl-Logrotate:
http://freshmeat.net/projects/perl-logrotate
March 1, 2014
Webinar - Dusan Baljevic
39

40. HP-UX Alternative Log File Tools
(part 2/3)
• newsyslog project (old and possibly obsolete):
http://www.weird.com/~woods/projects/newsyslog.html
• logtrim by Bill Hassell (released in HP ITRC forum several years ago):
http://forums11.itrc.hp.com/service/forums/questionanswer.d
o?
threadId=1053445&admit=109447626+1221799837763+283
53475
• Replace standard syslog daemon with Syslog-NG and SQL database:
http://www.balabit.com/network-security/syslogng/features/detailed
March 1, 2014
Webinar - Dusan Baljevic
40

41. HP-UX Alternative Log File Tools
(part 3/3)
• smartlog (very old bundles for HP-UX 10.20 and 11.00 only):
http://gatekeep.cs.utah.edu/hppd/hpux/Sysadmin/smartlog3.5/
• Many other Shell scripts, for example:
http://www.zazzybob.com/bin/logrevolver.sh.html
• LogWatch:
http://www2.logwatch.org:81/
March 1, 2014
Webinar - Dusan Baljevic
41

42. HP-UX Syslog-NG
• Syslog-NG and SQL database (MySQL, Microsoft SQL
(MSSQL), Oracle, PostgreSQL, SQLite)
• Log
rotation based on output filenames - Log output
filenames can be based on templates names which support
macro expansion. For example, if the output filename
template contains the month macro, a new filename will
created each month
• Often, s y s lo g -ng is used for log file consolidation
(centralized management)
March 1, 2014
Webinar - Dusan Baljevic
42

43. Conclusion
• Log file management is mostly managed reactively
• Majority of Unix admins I meet are not aware of OS-native
tools that are designed for log file administration
March 1, 2014
Webinar - Dusan Baljevic
43

44. Thank You!
Dusan Baljevic
Sydney, Australia
© 2008 Dusan Baljevic The information contained herein is subject to change without notice