Mango not starting ! Disk is full.

[stensma]

Please look at my error in the past with unaccessable Mango

[phildunlap]

Hi stensma,

Whenever a disk is full there's only one course of action: free up some space. I would use the command line, and SSH into the device. If you are using a Mac or a Linux OS, you can type 'ssh -p 2222 mango@ip' into a terminal to begin the SSH connection, or if you are on Windows you may need to download Putty as the SSH client. The login credentials will probably be the same over SSH as they were through Webmin

I would start by checking the size of Mango's files, specifically:

du -sh /opt/mango/backup
du -sh /opt/mango/logs
du -sh /opt/mango

Emptying the logs directory and the backup directory (or copying them elsewhere) is a good first step.

Here are some helpful commands, where %d is a file or directory name:

#Size of a file or directory:
du -sh %d

#List 20 largest files:
sudo du -ahx / | sort -rh | head -20

#Find all files over 100 MB:
sudo find / -size +100MB

#Remove a file
rm %d

#Remove a directory
rm -r %d

Please do share where the space has gone when you find some large files, if you can.

[mihairosu]

I have had an issue with full disk size with the publisher logs (Mine was 4.6 GB):

Example location: /opt/mango/logs/processLog.PersistentPublisher-1.log

Make sure Log Level is set to None for your publisher:

Hopefully that's what it is.

[phildunlap]

What Mihai says is true! Or, was true. That problem was among those fixed in 2.8.x and there are no more logs which can easily grow indefinitely anymore.

[stensma]

Hi all !
Mango still has some issues.
There was not room for the temp files needed in the commands.
What is wrong ? Where do I find space?

mango@stensmango:/$ sudo du -ahx / | sort -rh | head -20
[sudo] password for mango:
Sorry, try again.
[sudo] password for mango:
sort: cannot create temporary file in ‘/tmp’: No space left on device
ot local

Filesystem from Webmin:

[phildunlap]

Weird... why does / report as only being 47% full if the disk is full? I would restart the device if I saw that.

You could see if there's an abundance of logging from something else

sudo du -sh /var/log/*

[JoelHaggar]

I would check your /opt/mango/backup folder. You may have a lot of NoSQL database backups which you can delete to free up space. You can turn off incremental backups on the system settings page under the NoSQL settings.

[stensma]

Hi this pictures was taken right after a cold boot.
And ALL backup are moved to an external disk.

[stensma]

@phildunlap

[stensma]

Result with Find size

mango@stensmango:/opt/mango/logs$ sudo find / -size +100M
find: /proc/26770/task/26770/fd/5': No such file or directory find: /proc/26770/task/26770/fdinfo/5': No such file or directory
find: /proc/26770/fd/5': No such file or directory find: /proc/26770/fdinfo/5': No such file or directory

[stensma]

SSH picture

[phildunlap]

Can you check the output of df -i as it may be there are no more inodes. If that is the case that they're 100% utilized, you'll have to find some files (and directories) to delete. They can be any files of any size, but you should of course exercise care.

A useful command may be:

#count the number of files and directories in /path/to/dir
find /path/to/dir | wc -l

[stensma]

After :

Then I am able to run basic Webmin functions, but what is creating the problem?
I have tried no to run apt-get purge and autoremove but there is nothing to clean out.

Now I work with following command.
$ for i in /*; do echo $i; find $i |wc -l; done
This lists the folders and inode number.
and I fount that /var had over 300k nodes
I will dig deeper on where exactly

What is creating the files in /var/spool/exim4/input ??

[stensma]

I have now removed the files in the /var/spool/exim4/input.
Now the resources is back on track, but still what is creating the files?

[stensma]

New files are created all the time.
There two different contents one is:

The other has following content:

I have now created an empty log.txt file in the missing mango-backups folder,
lets see if this does the trick...

[stensma]

I found the bad guy !
A Cron Job create all the files in /var/spool/exim4/input.
When I disable this task, the folder is quiet .
But why is this job there ? and what is it suppose to do?

[phildunlap]

It was supposed to be constructing a file in /opt/systeminfo/ (I believe, might be /opt/info/ or something like that) which would then be read by a preconfigured ASCII File Data Source for system information like remaining disk space. You can certainly deactivate it without much loss, and you may as well disable the ASCII data source that was preloaded. I will definitely investigate if other MangoES's are having a problem with it creating files in /var/spool/

[mihairosu]

I used this command straight from the terminal to list inodes and found it easier:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

[mihairosu]

This issue has happened again, and the same folder caused the problem.

ASCII File Reader has been off for a long time, so that did not prevent the issue from re-occurring.

edit: This is happening on Mango v3, but not on Mango v1 or on an Ubuntu install.