Hi Terry,
Thank you for the quick response. I have just tried this on another PC and I don't get the error so it must be something cached.

Thank you for your help!
Matt.

Hi all,
I have just updated one of our MangoES units to 3.3.0 and now get the following error when trying to create (or modify) a persistent TCP publisher. "Missing method or missing parameter converters"

Everything is filled out correctly. Saving an existing publisher results in the same message as well. I have tried re-installing the Persistent module with the same outcome.

Help!

Cheers,
Matt.

Hi Phil,
Thanks for the tips.
I upgraded JAVA about a month ago on all systems so that shouldn't be the issue. Yes, 4 persistent sources (from 4 MangoES publishers) and one publisher (to a MangoES). I'll adjust the thread count and overlap values and see what happens.
Yes, the 6GB option has been enabled in the ext-enabled folder. Mango reports 5.3GB with 5.1GB free. The server has 8 GB total.

I did notice this while checking the memory usage.

I guess this could be due to the large thread count from the persistent publishers.

I will do some more probing in the console to see if I get a bit more info. At least I can trigger the fault easily by running a backup.

Will keep you posted.

Matt.

Hi Phil,
Thanks for your response.

The virtual machine has been upgraded recently and Mango now has access to up to 6GB of RAM (up from 1.5) so memory shouldn't be the issue.
We have plenty of hdd space with 17% used of 70GB.

I've turned off backups for now as they cause the system to crash. Errors will begin populating within the first minute of the backup starting.

The persistent data sync is running with the default settings so the thread count is set to 10. We currently have 4 persistent data sources on the main server and one persistent data publisher.

I'm not sure what you mean by the minimum window size. I cannot find this option in the publisher, source or system settings.

We're running core 3.2.1+ enterprise with NoSQL and all modules are up to date.

To me it feels like there is a bad data point somewhere in the system and the backup and persistent processes force Mango to probe it, causing an exception. If this is the case, how would I go about finding the problematic point?

Cheers,
Matt.

Hi all,
We're having an issue with our Mango instance crashing randomly. Data points are received from sources but not committed to the database. The issue can be re-created by running a NoSQL backup. The relevant logs are attached below with the 'Timer already cancelled.' error repeating until the process is rebooted. Rebooting the process gets things going again for another couple of days.

INFO  2017-09-25T09:03:36,695 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute:109) - Starting database backup WorkItem. 
INFO  2017-09-25T09:03:36,697 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.createLogOutputStream:224) - Writing upgrade log to /opt/mango/logs/com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.log 
FATAL 2017-09-25T09:03:51,645 (com.serotonin.timer.TimerThread.run:41) - TimerThread failed 
java.lang.NullPointerException: null
	at com.serotonin.timer.OrderedThreadPoolExecutor$TimePriorityLimitedTaskQueue.add(OrderedThreadPoolExecutor.java:442) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.OrderedThreadPoolExecutor.execute(OrderedThreadPoolExecutor.java:204) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.OrderedTimerThread.executeTask(OrderedTimerThread.java:26) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.TimerThread.mainLoop(TimerThread.java:133) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.TimerThread.run(TimerThread.java:38) [mango-3.2.1.jar:?]
ERROR 2017-09-25T09:03:53,506 (com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run:559) - Error in work item 
java.lang.IllegalStateException: Timer already cancelled.
	at com.serotonin.timer.RealTimeTimer.scheduleImpl(RealTimeTimer.java:110) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.AbstractTimer.schedule(AbstractTimer.java:26) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing.schedule(BackgroundProcessing.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.util.timeout.TimeoutTask.<init>(TimeoutTask.java:41) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.util.timeout.TimeoutTask.<init>(TimeoutTask.java:35) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.TimeoutDetectorRT.scheduleJob(TimeoutDetectorRT.java:103) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.TimeDelayedEventDetectorRT.scheduleJob(TimeDelayedEventDetectorRT.java:27) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.DifferenceDetectorRT.pointData(DifferenceDetectorRT.java:42) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.NoUpdateDetectorRT.pointUpdated(NoUpdateDetectorRT.java:22) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:672) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run(BackgroundProcessing.java:556) [mango-3.2.1.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.1.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.1.jar:?]
	at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
ERROR 2017-09-25T09:03:55,954 (com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run:559) - Error in work item 
java.lang.IllegalStateException: Timer already cancelled.
	at com.serotonin.timer.RealTimeTimer.scheduleImpl(RealTimeTimer.java:110) ~[mango-3.2.1.jar:?]
	at com.serotonin.timer.AbstractTimer.schedule(AbstractTimer.java:26) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing.schedule(BackgroundProcessing.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.util.timeout.TimeoutTask.<init>(TimeoutTask.java:41) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.util.timeout.TimeoutTask.<init>(TimeoutTask.java:35) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.TimeoutDetectorRT.scheduleJob(TimeoutDetectorRT.java:103) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.TimeDelayedEventDetectorRT.scheduleJob(TimeDelayedEventDetectorRT.java:27) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.DifferenceDetectorRT.pointData(DifferenceDetectorRT.java:42) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.event.detectors.NoUpdateDetectorRT.pointUpdated(NoUpdateDetectorRT.java:22) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:94) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:672) ~[mango-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run(BackgroundProcessing.java:556) [mango-3.2.1.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.1.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.1.jar:?]
	at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

I've also noticed that the persistent data historical sync also gets held up when trying to fill the holes where data wasn't logged to the main server during these outages (we are using MangoES units on site for redundancy).

Any ideas on where to start looking? We need to get this sorted as we are going live to our clients in the next few days.

Thanks in advance,
Matt.

@phildunlap Thanks Phil. I'll do the update and let you know how I get on.
Interestingly I seem to be getting the following error repeating over and over, sometimes only 10 or so seconds apart.

High priority task: com.serotonin.m2m2.persistent.ds.PersistentDataSourceRT$StatusProvider was rejected because it is already running.

This is on the data source end or the persistent data source publisher. Could this be an issue of corruption?

Hi Phil,
I think you're right about the error being a symptom of something else.
Looking at the data I noticed that one of the persistent data points showed data up until the date the Mango server started to crash. Data on the MangoES was still being recorded. I'm guessing something had caused corruption in the database and when a historical sync was started it would get hung up in this data point as clearing all data point data and doing a complete re-sync with the MangoES on site seems to have solved things.

I'm guessing the file limit issues us due to too many hung up historical sync threads running. I'll look into the user limits though.

I'd still like a way of being able to work out if this happens again and how to fix it, especially if Mango is unable to start at all.

Many thanks for your support.
Matt.

Hi all,
I am having issues with the latest update.
It seems like after a random amount of time (usually around 30mins) Mango gets stuck in some sort of loop, consuming 100% of the CPU and not allowing HTTP access.
Attached is a screenshot of the Thread Monitoring page showing the runaway thread.

This corresponds to what I see in top

I'm getting a number of these errors in ma.log which may be the cause of the lockup or may be a side effect of it.

ERROR 2017-01-06 14:17:39,989 (com.infiniteautomation.datafilesource.rt.DataFileDataSourceRT.doPoll:269) -
java.lang.NullPointerException
at com.infiniteautomation.datafilesource.rt.DataFileDataSourceRT.loadNewFiles(DataFileDataSourceRT.java:182)
at com.infiniteautomation.datafilesource.rt.DataFileDataSourceRT.doPoll(DataFileDataSourceRT.java:259)
at com.infiniteautomation.datafilesource.rt.DataFileDataSourceRT.doPollNoSync(DataFileDataSourceRT.java:250)
at com.serotonin.m2m2.rt.dataSource.PollingDataSource.scheduleTimeout(PollingDataSource.java:134)
at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:69)
at com.serotonin.timer.TimerTask.runTask(TimerTask.java:148)
at com.serotonin.timer.OrderedTimerTaskWorker.run(OrderedTimerTaskWorker.java:29)
at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTask.run(OrderedThreadPoolExecutor.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Logs attached...0_1483665559469_ma.log

I've performed a clean install leaving only the databases and db folders behind and am still having issues.
I am running a Persistent TCP data source to sync with a remote MangoES.

Rebooting Mango seems to sort things out for 30 mins or so.

Cheers,
Matt.

I have done some more testing and I definitely think there is an issue with the file data source.
Every time I load the CSV I get a different number of data points cached and displayed correctly on the Data points page chart, depending on how full the cache buffer is at the time of the import but only a few points are actually committed to the database.

This screenshot for example shows data logged for 21 April, 04 April, 10 Mar, etc. This is daily data from the 14 Oct to the 21 April. The CSV file is of a constant format. You can see from the graph that daily data is being cached from roughly the 1st April to the 21st April but everything else is in bits. This data is missing from the database data.

Running the import again, this time with display cached data set to 150 I get the complete dataset but nothing more is recorded to the database (see history table).

The datasource is set to log all data so I would assume that if data is being cached correctly, unless there is a double up it should be logged to the database.

Regards,
Matt.

The points are set to log all data. My timestamps seem fine when looking at the cached data so I don't think that's the issue. The CSV that I am importing has a timestamp of yyyyMMdd so I am setting HHmm to 00:00 which seems to work OK looking at the cached data.

Hi Joel,
Did you manage to look into when the database time stamping occurs? It is strange that the import works fine for cahced data but when checking the database points are missing. I'm thinking that data points are being written so quickly that they get the same EPROC time, even though they have different imported time stamps.
Cheers,
Matt.

I've just realized the screen shots were the same.
Cached:

Database

Is the EPROC timestamp when data is imported or when the data is timestamped in the CSV? When looking at the data I import the cached data is correct with daily points logged but when I look at the non cached (database) data it is missing most of the points.
Here is an example:
Cached data:

Database data

This is the same as the downloaded CSV. This is 21 days worth of data but only 2 days have actually been stored in the database.

Matt.

Excellent. I'll try a couple of imports and export the CSV. It would be awesome if it worked that way as I could set up a script to automatically grab the CSV from the email we get and place it in a folder on the server for Mango to process.

Matt.

I am also looking at doing the same. We get emailed daily logs of data from the last month and if I was to import this data every day I would get duplicates. I am using the NoSQL database but still seem to get duplicate data. My data is a single daily value so the time portion of the timestamp is set manually to 00:00. Unless of course I am missing something and duplicate data is just kept in memory, not necessarily saved to the database. It appears under point history anyway.

Apart from that the file data source is my new favorite thing in Mango. It allows us to start importing data from other companies data sources and comparing it against the readings that we are recording on our hardware.

Matt.

That was it. I had completely misinterpreted what the check boxes were for and had them selected. That resolves the issue of clients causing havoc but it would be nice to remove access to all modules except DGLux.
I'm looking forward to the next version. Development seems to be happening in leaps and bounds lately. I especially like the new module upgrade feature.

Matt.

Can I disable access to all modules except DGLux for non admin users?
If I log in to Mango as a non admin user currently I am able to modify data sources and cause all sorts of havoc.

Yes, my dglux.properties file looks the same with the addition of a logout redirect.
Is it possible to run DGLux on a different port or web server service, essentially isolating it from Mango?
Obviously I will still need access to Mango for administration purposes.

Hi Joel,
I have used the dglux.properties file to modify the login page and redirection.
The problem is how do I disable access to Mango?
If a client logs in to DGLux they get a URL of <hostname>/dglux/view
If they changed that to <hostname>/blah they would get the Mango page not found page with access to many parts of Mango.

Cheers,
Matt.

Hi All,
We have our Mango installation set up so that when you access the login page it directs you to the DGLux login page.
When logging out of DGLux the user is redirected back to the DGLux login page.
The problem is if you enter an invalid URL or a Mango URL you get sent through to Mango.

How can I disable non administrator access to Mango, only allowing DGLux access to clients? I don't want our clients getting into Mango and re-configuring data points, etc.

Cheers,
Matt.