• Home
  • Free Stuff
  • About
  • Contact

Zimbra : No space left on device & Inode Usage

December 9, 2015

Yesterday, I got a phone call from my customer regarding their Zimbra Mail Server. I’ve helped them setting up and configuring Zimbra, 2-3 years ago. Their Zimbra mail server not working properly since yesterday morning. Dig on log details show me error message : “Zimbra No space left on device”, but it should be wrong, because they have plenty free space both on system and data disk. I’m remembering that I’ve  experience  similar error on one of my hosting server when compromised account broadcast too many small files and fill in all Inode allocation.

vavai-df-h

The first command that I did was stopping Zimbra services. Deeper investigation with df -ih prove me that the problem came from similar cause. All Inode file system has been full and no free Inode available. Checking /tmp folder seems to make the server hang for a while, until then displaying too many files. Looking at the result, there were about 1 million files on /tmp/tmpXXX, eating all Inode stock.

vavai-df-i

I was lucky to find out that /tmp has too many files on it, but you can also using following command to looking for suspected folder :
[code lang=”bash”]sudo find . -xdev -type f | cut -d “/” -f 2 | sort | uniq -c | sort -n
[/code]
or
[code lang=”bash”]for i in /*; do echo $i; find $i |wc -l; done[/code]
or for specific folder :
[code lang=”bash”]for i in /tmp/*; do echo $i; find $i |wc -l; done[/code]
I’m trying to remove all /tmp file starting with tmp-XXX on file name but server responded that removal command was not successful  because there were too many files on the search criteria (I forgot an error details 🙂 ). I ended up manually delete the file with : rm /tmp/tmp1*, rm /tm/tmp2* and so on, increment with number and then alphabet on it’s file name.

After a while, I’m too tired to delete temporary file manually. In short calculate, I must execute 10 (for number from 1-10)+26 (for lowercase alphabet)+26 (for uppercase alphabet), totally 62 times execution. Not a smart ways to do stupid works 🙂

Searching for a bash script, I got this nice and simpler script :
[code lang=”bash”]
for alphabet in {A..Z}
do
echo $alphabet
rm /tmp/tmp_$alphabet*
done
[/code]
The script could be nicer if could also include number and lowercase alphabet on top of script but never mind, I was quite satisfied to run above script as much as 3 times only by changing the parameter.

In less than 5 minutes, all temporary files was deleted successfully and server have pretty much Inode available now. I can now starting Zimbra services and monitor log for any possible cause. The following log entry on /var/log/zimbra.log caught my attention :
>Dec  7 13:14:36 mail postfix/proxymap[19610]: warning: ldap:/opt/zimbra/conf/ldap-slm.cf is unavailable. open /opt/zimbra/conf/ldap-slm.cf: Permission denied
Checking on my other running Zimbra instance give me following permission :
[code lang=”bash”]# ls -ltrh  /opt/zimbra/conf/ldap-slm.cf[/code]
>-rw-r—– 1 zimbra postfix 603 Dec  3 02:20 /opt/zimbra/conf/ldap-slm.cf
I’m stopping Zimbra services again, then execute fix the permission :
[code lang=”bash”]/opt/zimbra/libexec/zmfixperms –extended –verbose[/code]
and then looking at file permission again before restarting the services. However, my eyes stumble upon with other error logs :
>Dec  7 06:27:41 mail postfix/postdrop[4233]: warning: mail_queue_enter: create file maildrop/952460.4233: Permission denied
>Dec  7 06:27:42 mail postfix/postdrop[6057]: warning: mail_queue_enter: create file maildrop/43485.6057: Permission denied
>Dec  7 06:27:42 mail postfix/postdrop[6084]: warning: mail_queue_enter: create file maildrop/127795.6084: Permission denied
>Dec  7 06:27:51 mail postfix/postdrop[19190]: warning: mail_queue_enter: create file maildrop/739237.19190: Permission denied
It’s seems some postdrop services are still exist and running and was not affected by stopping Zimbra services. I’m trying to stop Zimbra services again and then kill all Zimbra and postdrop process with root permission :
[code lang=”bash”]
killall zimbra
killall postdrop
ps aux | grep zimbra
ps aux | grep postdrop
[/code]
and then restarting Zimbra services. Finally, Zimbra working properly now and no error entries on Zimbra logs. Mission accomplised 😉

SLESZimbra
Share

Linux

Masim "Vavai" Sugianto
Traveller, Open Source Enthusiast & Book Lover. Works as Independent Worker & Self-Employer.

2 Comments


Karl Ove Hufthammer
December 9, 2015 at 04:08
Reply

I think the tmpwatch tool could be useful here. Just execute ‘tmpwatch 7d /tmp’ to delete all files which haven’t been access in the last week.



Masim "Vavai" Sugianto
December 9, 2015 at 06:57
Reply

Hi Karl Ove Hufthammer,

Thank you and noted for tmpwatch



Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Recent Posts

    • Januari 2021 Portfolio Update
    • December 2020 Portfolio Update
    • After a While
    • Life goes On : Farming
    • Pursuing FIRE in Indonesia : Are You Prepared For A Recession?
    • VirtualBox Error Kernel Driver not Installed (rc=-1908) on Zorin OS 15.2
    • ZorinOS on Intel NUC Hades Canyon Series NUC817HVK
    • Banana Farm
    • Zorin OS 15.2 Ultimate
    • Haraka and LDAP Authentication with Zimbra



© Copyright LetsBlog Theme Demo - Theme by ThemeGoods