F5 error message “Log disk usage still higher than 80% after logrotate” is issued and “df” shows that the var/log partition is nearly full but “du” finds no files to delete. This article describes how to find the “invisible” log files and regain the disk space.
Description of the Issue
The following error can be seen logged to the console:
011d0004:3: Disk partition var_log has only 7% free Feb 25 09:25:03 f5-01 emerg alertd[7775]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion
“df” shows that the /var/log partition is nearing its limit:
[admin@f5-01:Active:In Sync] ~ # df -h Filesystem Size Used Avail Use% Mounted on <snip> /dev/mapper/vg--db--sda-dat.log.1 485M 427M 33M 93% /var/log <snip>
“du” however shows that there are only a few MBs in the /var/log directory:
[admin@f5-01:Active:In Sync] log # du -h ./ –max-depth=1
<snip>
26M ./
Cause
Monitor logging has been enabled for e.g. a node which is ongoing. The file however has been deleted but as it is still being written to, the disk space is still in use.
Fix
We simply need to identify fo which node the monitoring has been enabled and then disable it. We can see a list of the files which have been deleted from the log directory with the following command:
[admin@f5edin:Active:In Sync] monitors # lsof | grep deleted | grep log bigd 6775 root 7w REG 253,2 407867086 109730 /var/log/monitors/Common_icmp-Common_panel1-0.log.1 (deleted)
Now go to Local Traffic -> Nodes -> Select the node and uncheck the monitor logging box:
Running “df” should now show plenty of recovered disk space:
[admin@f5-01:Active:In Sync] log # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg--db--sda-dat.log.1
485M 36M 424M 8% /var/log