Mango Won't Start Stuck at Watchslists Loading

phildunlap

Pedro, we reworked some aspects of the polling system, and you are correct that message replaced the old poll aborted.

Does your instance start in SAFE mode?

phildunlap

Is this a new installation or recent upgrade? It sounds like something may have gone awry in the download of the module or some such, if you cannot get past loading that module. I would try removing watchlist from your module folder, or downloading a new copy of it.

Pedro

The system started in SAFE mode, but I had to start a lot of datasources. I don't know what will happen when I restart with the datasources enabled. I increased "High priority maximum pool size" from 1000 to 5000, but am still unable to increase "Medium priority core pool size."

phildunlap

Hmm. Definitely worth an experiment. Can you check that "runtime.datasource.logStartupMetrics=true" in your env.properties? This will hopefully get more information during the startup of the data sources, provided com.serotonin.m2m2.rt or a parent (such as com.serotonin.m2m2) has logging enabled in your log4j configuration at either debug or info levels. It is at info level, by default.

phildunlap

Also, very troubling you cannot increase your medium priority pool. Do you have overridden jars or classes? Do you get a stack trace when it fails? Did you attempt to change that value before or after enabling your data sources? I am certainly able to edit that.

Pedro

I just added

runtime.datasource.logStartupMetrics=true

To env.properties. However, I avoid startups because if they don't work then I'll be offline. Also, since restarts often don't work because they get stuck during a shutdown, I usually have to kill mango, which leaves tsdb dirty and adds at least another 10 minutes to the startup process, which further increases the downtime.

I will start a new thread to address the thread pool issue. I have no override jar or class files.

phildunlap

Hi Pedro,

It is fairly likely your shutdown will be much faster than it was on 2.7.2. We reworked the polling in part because this was related to some situations causing very long shutdowns and frequent aborted poll behavior.

Pedro

That change was made only to the modbus module, correct?

phildunlap

No, it would affect everything. I believe the long shutdown was only in systems using the NoSQL database.

phildunlap

Pedro, what are you NoSQL batch write behind settings on your system settings page?

Pedro

The current settings below should be the same as the defaults, since I've never changed them:

Batch write behind spawn threshold 	100000
Max batch write behind tasks 	10
Batch write behind inserts per task     100000

For what it's worth, there are a number of points (perhaps more than a dozen) that are configured to log every second, with no cache. Is that an issue?

phildunlap

It shouldn't be. Those are the defaults, so they're probably fine.

The only way I've been able to replicate your situation (stalled at 75%) involved having far too many write behind threads and far too little data per thread (inverting those numbers, for instance). Still looking.

Pedro

Does the medium priority pool size affect this? Recall that in SAFE mode, my metadata sources were also disabled. I have a substantial number of metadata points, and for years I have frequently have problems where they stop updating for no known reason. My medium priority pool is set to the default of "3," and I get a DWR error whenever I try to save a larger number.

phildunlap

I wouldn't think so, since by definition high priority tasks (like starting) would supersede other tasks. It's possible priority is inverting somewhere, but that's not an immediately fruitful line of inquiry.

I don't know why your medium pool doesn't save. That's why I keep saying check overrides classes (I just discovered overrides/properties is also on the classpath and shouldn't have any extra .jar or .class files hanging around. Perhaps it's your browser cache? Perhaps you could export your system settings and see if something bizarre is in there...

If you start a fresh Mango in a new folder, I sincerely doubt you will have that issue, which leads me to believe there is an override in overrides/classes (or properties or lib), duplicate jars in Mango/lib/ or your browser has cached the client side of the DWR code.

Pedro

I just emailed you my system settings. I deleted the work directory:
$ sudo rm -r work/jsp
I used Chromium in case the Firefox cache is an issue, and there are no jar or class files in overrides:
$ find /opt/mango/overrides -type f
/opt/mango/overrides/web/favicon.ico
/opt/mango/overrides/web/MangoDailyScheduled.sh
/opt/mango/overrides/web/images/favicon.ico
/opt/mango/overrides/web/images/logo.png
/opt/mango/overrides/web/images/Tides_icon.png
/opt/mango/overrides/web/WEB-INF/tags/page.tag
/opt/mango/overrides/web/WEB-INF/tags/html5/footer.tag
/opt/mango/overrides/web/WEB-INF/tags/html5/footer.tag-orig
/opt/mango/overrides/web/WEB-INF/tags/mobile-page.tag
/opt/mango/overrides/properties/keystore.jks
/opt/mango/overrides/properties/i18n.properties
/opt/mango/overrides/properties/env.properties-derby~
/opt/mango/overrides/properties/env.properties-mysql~
/opt/mango/overrides/properties/env.properties
/opt/mango/overrides/properties/env.properties.13Jan2015
/opt/mango/overrides/__MACOSX/classes/..DS_Store
/opt/mango/overrides/__MACOSX/classes/com/..DS_Store
/opt/mango/overrides/__MACOSX/classes/com/serotonin/m2m2/..DS_Store
/opt/mango/overrides/__MACOSX/classes/com/serotonin/m2m2/rt/..DS_Store
/opt/mango/overrides/__MACOSX/classes/com/serotonin/._.DS_Store
/opt/mango/overrides/classes/log4j.xml

vanguard

Thanks for the replys.
I had to reinstall everthing, clear browser cache and run in admin mode. Now it works, i will let anyone know if the problem exists after creating some watchlists and graphics.