Zimbra : No space left on device & Inode Usage

Yesterday, I got a phone call from my customer regarding their Zimbra Mail Server. I’ve helped them setting up and configuring Zimbra, 2-3 years ago. Their Zimbra mail server not working properly since yesterday morning. Dig on log details show me error message : “Zimbra No space left on device”, but it should be wrong, because they have plenty free space both on system and data disk. I’m remembering that I’ve  experience  similar error on one of my hosting server when compromised account broadcast too many small files and fill in all Inode allocation.

The first command that I did was stopping Zimbra services. Deeper investigation with df -ih prove me that the problem came from similar cause. All Inode file system has been full and no free Inode available. Checking /tmp folder seems to make the server hang for a while, until then displaying too many files. Looking at the result, there were about 1 million files on /tmp/tmpXXX, eating all Inode stock.

I was lucky to find out that /tmp has too many files on it, but you can also using following command to looking for suspected folder :

sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

or

for i in /*; do echo $i; find $i |wc -l; done

or for specific folder :

for i in /tmp/*; do echo $i; find $i |wc -l; done

I’m trying to remove all /tmp file starting with tmp-XXX on file name but server responded that removal command was not successful  because there were too many files on the search criteria (I forgot an error details 🙂 ). I ended up manually delete the file with : rm /tmp/tmp1*, rm /tm/tmp2* and so on, increment with number and then alphabet on it’s file name.

After a while, I’m too tired to delete temporary file manually. In short calculate, I must execute 10 (for number from 1-10)+26 (for lowercase alphabet)+26 (for uppercase alphabet), totally 62 times execution. Not a smart ways to do stupid works 🙂

Searching for a bash script, I got this nice and simpler script :

for alphabet in {A..Z}
do
echo $alphabet
rm /tmp/tmp_$alphabet*
done

The script could be nicer if could also include number and lowercase alphabet on top of script but never mind, I was quite satisfied to run above script as much as 3 times only by changing the parameter.

In less than 5 minutes, all temporary files was deleted successfully and server have pretty much Inode available now. I can now starting Zimbra services and monitor log for any possible cause. The following log entry on /var/log/zimbra.log caught my attention :
>Dec  7 13:14:36 mail postfix/proxymap[19610]: warning: ldap:/opt/zimbra/conf/ldap-slm.cf is unavailable. open /opt/zimbra/conf/ldap-slm.cf: Permission denied
Checking on my other running Zimbra instance give me following permission :

# ls -ltrh  /opt/zimbra/conf/ldap-slm.cf

>-rw-r—– 1 zimbra postfix 603 Dec  3 02:20 /opt/zimbra/conf/ldap-slm.cf
I’m stopping Zimbra services again, then execute fix the permission :

/opt/zimbra/libexec/zmfixperms --extended --verbose

and then looking at file permission again before restarting the services. However, my eyes stumble upon with other error logs :
>Dec  7 06:27:41 mail postfix/postdrop[4233]: warning: mail_queue_enter: create file maildrop/952460.4233: Permission denied
>Dec  7 06:27:42 mail postfix/postdrop[6057]: warning: mail_queue_enter: create file maildrop/43485.6057: Permission denied
>Dec  7 06:27:42 mail postfix/postdrop[6084]: warning: mail_queue_enter: create file maildrop/127795.6084: Permission denied
>Dec  7 06:27:51 mail postfix/postdrop[19190]: warning: mail_queue_enter: create file maildrop/739237.19190: Permission denied
It’s seems some postdrop services are still exist and running and was not affected by stopping Zimbra services. I’m trying to stop Zimbra services again and then kill all Zimbra and postdrop process with root permission :

killall zimbra
killall postdrop
ps aux | grep zimbra
ps aux | grep postdrop

and then restarting Zimbra services. Finally, Zimbra working properly now and no error entries on Zimbra logs. Mission accomplised 😉