Mango not starting ! Disk is full.

[stensma]

I have described my problems in the past. Now it is happening again.
The Mango ES is not starting up the web service, or it is impossible to access.
This time there is no huge log files on the disk, and no way for me to find where or what is causing this overloaded disk. There is a brief error code in the Webmin interface when I try to get the system status that imply that the disk is full and the temporary file could not be created. The result is that I have deleted all obvious files as logs and other stuff to get some free space on the disk.
But no I can not access the Mango ES at all from the browser.
Please advis how I could find my Mango ES and clean the disk, and find why the disk gets full.
The system is very small so it should not be a problem.
My raspeberry is running a free version collecting all data into a modbus publishing then collected by the Mango ES and this is where I keep my status screen as a graphical view and some other data sources collected.
What do I do?

[stensma]

Please look at my error in the past with unaccessable Mango

[phildunlap]

Hi stensma,

Whenever a disk is full there's only one course of action: free up some space. I would use the command line, and SSH into the device. If you are using a Mac or a Linux OS, you can type 'ssh -p 2222 mango@ip' into a terminal to begin the SSH connection, or if you are on Windows you may need to download Putty as the SSH client. The login credentials will probably be the same over SSH as they were through Webmin

I would start by checking the size of Mango's files, specifically:

du -sh /opt/mango/backup
du -sh /opt/mango/logs
du -sh /opt/mango

Emptying the logs directory and the backup directory (or copying them elsewhere) is a good first step.

Here are some helpful commands, where %d is a file or directory name:

#Size of a file or directory:
du -sh %d

#List 20 largest files:
sudo du -ahx / | sort -rh | head -20

#Find all files over 100 MB:
sudo find / -size +100MB

#Remove a file
rm %d

#Remove a directory
rm -r %d

Please do share where the space has gone when you find some large files, if you can.

[mihairosu]

I have had an issue with full disk size with the publisher logs (Mine was 4.6 GB):

Example location: /opt/mango/logs/processLog.PersistentPublisher-1.log

Make sure Log Level is set to None for your publisher:

Hopefully that's what it is.

[phildunlap]

What Mihai says is true! Or, was true. That problem was among those fixed in 2.8.x and there are no more logs which can easily grow indefinitely anymore.

[stensma]

Hi all !
Mango still has some issues.
There was not room for the temp files needed in the commands.
What is wrong ? Where do I find space?

mango@stensmango:/$ sudo du -ahx / | sort -rh | head -20
[sudo] password for mango:
Sorry, try again.
[sudo] password for mango:
sort: cannot create temporary file in ‘/tmp’: No space left on device
ot local

Filesystem from Webmin:

[phildunlap]

Weird... why does / report as only being 47% full if the disk is full? I would restart the device if I saw that.

You could see if there's an abundance of logging from something else

sudo du -sh /var/log/*

[JoelHaggar]

I would check your /opt/mango/backup folder. You may have a lot of NoSQL database backups which you can delete to free up space. You can turn off incremental backups on the system settings page under the NoSQL settings.

[stensma]

Hi this pictures was taken right after a cold boot.
And ALL backup are moved to an external disk.

[stensma]

@phildunlap

[stensma]

Result with Find size

mango@stensmango:/opt/mango/logs$ sudo find / -size +100M
find: /proc/26770/task/26770/fd/5': No such file or directory find: /proc/26770/task/26770/fdinfo/5': No such file or directory
find: /proc/26770/fd/5': No such file or directory find: /proc/26770/fdinfo/5': No such file or directory

[stensma]

SSH picture

[phildunlap]

Can you check the output of df -i as it may be there are no more inodes. If that is the case that they're 100% utilized, you'll have to find some files (and directories) to delete. They can be any files of any size, but you should of course exercise care.

A useful command may be:

#count the number of files and directories in /path/to/dir
find /path/to/dir | wc -l

[stensma]

After :

Then I am able to run basic Webmin functions, but what is creating the problem?
I have tried no to run apt-get purge and autoremove but there is nothing to clean out.

Now I work with following command.
$ for i in /*; do echo $i; find $i |wc -l; done
This lists the folders and inode number.
and I fount that /var had over 300k nodes
I will dig deeper on where exactly

What is creating the files in /var/spool/exim4/input ??

[stensma]

I have now removed the files in the /var/spool/exim4/input.
Now the resources is back on track, but still what is creating the files?

[stensma]

New files are created all the time.
There two different contents one is:

The other has following content:

I have now created an empty log.txt file in the missing mango-backups folder,
lets see if this does the trick...

[stensma]

I found the bad guy !
A Cron Job create all the files in /var/spool/exim4/input.
When I disable this task, the folder is quiet .
But why is this job there ? and what is it suppose to do?

[phildunlap]

It was supposed to be constructing a file in /opt/systeminfo/ (I believe, might be /opt/info/ or something like that) which would then be read by a preconfigured ASCII File Data Source for system information like remaining disk space. You can certainly deactivate it without much loss, and you may as well disable the ASCII data source that was preloaded. I will definitely investigate if other MangoES's are having a problem with it creating files in /var/spool/

[mihairosu]

I used this command straight from the terminal to list inodes and found it easier:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

[mihairosu]

This issue has happened again, and the same folder caused the problem.

ASCII File Reader has been off for a long time, so that did not prevent the issue from re-occurring.

edit: This is happening on Mango v3, but not on Mango v1 or on an Ubuntu install.