Server Time out

CraigWeb

I get this often when working on a cloud server. Haven't seen it when working on a localhost like you are experiencing.

phildunlap

Hi gquiroz,

I am not able to figure out which part is relevant.

There are many exceptions in the file, all could suggest things to improve.

Starting at the bottom and working backwards,

ERROR 2018-11-07 11:19:41,446 (com.serotonin.m2m2.Main.main:119) - Error during initialization 
java.lang.RuntimeException: java.net.BindException: Address already in use: bind
	at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:810)
	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:223)
	at com.serotonin.m2m2.Main.main(Main.java:115)
Caused by: java.net.BindException: Address already in use: bind
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Unknown Source)
	at sun.nio.ch.Net.bind(Unknown Source)
	at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
	at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.server.Server.doStart(Server.java:366)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:807)
	... 2 more

Suggests the previous launch of Mango is still running. Usually if you close the command prompt Window that it was launched in, Mango will have stopped, but, you may need to kill Java in your Task Manager.

FATAL 2018-11-07 10:18:30,555 (com.serotonin.m2m2.util.timeout.RejectedRunnableEventGenerator.rejectedExecution:51) - Task com.serotonin.ma.bacnet.BACnetDataSourceRT$1@19e4bfd rejected from com.serotonin.timer.OrderedThreadPoolExecutor@997126[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 996] 
java.util.concurrent.RejectedExecutionException: Task com.serotonin.ma.bacnet.BACnetDataSourceRT$1@19e4bfd rejected from com.serotonin.timer.OrderedThreadPoolExecutor@997126[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 996]
	at com.serotonin.m2m2.util.timeout.RejectedRunnableEventGenerator.rejectedExecution(RejectedRunnableEventGenerator.java:50)
	at com.serotonin.timer.OrderedThreadPoolExecutor.rejectedExecution(OrderedThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
	at com.serotonin.timer.TimerThread.execute(TimerThread.java:53)
	at com.serotonin.timer.RealTimeTimer.execute(RealTimeTimer.java:86)
	at com.serotonin.ma.bacnet.BACnetDataSourceRT.iAmReceived(BACnetDataSourceRT.java:531)
	at com.serotonin.bacnet4j.event.DeviceEventHandler.fireIAmReceived(DeviceEventHandler.java:99)
	at com.serotonin.bacnet4j.service.unconfirmed.IAmRequest.handle(IAmRequest.java:95)
	at com.serotonin.bacnet4j.transport.DefaultTransport.receiveAPDU(DefaultTransport.java:539)
	at com.serotonin.bacnet4j.transport.DefaultTransport.receiveImpl(DefaultTransport.java:463)
	at com.serotonin.bacnet4j.transport.DefaultTransport.run(DefaultTransport.java:404)
	at java.lang.Thread.run(Unknown Source)

In 2.x BACnet could get pretty thread hungry. It was best practice to combine BACnet data sources together whenever possible. You could also increase your maximum high priority thread pool limit.

FATAL 2018-11-07 10:15:20,063 (com.serotonin.timer.TimerThread.run:41) - TimerThread failed 
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Out of memory. [90108-181]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:772)
	at com.serotonin.db.DaoUtils.query(DaoUtils.java:308)
	at com.serotonin.m2m2.db.dao.EventHandlerDao.getEventHandlers(EventHandlerDao.java:155)
	at com.serotonin.m2m2.db.dao.EventHandlerDao.getEventHandlers(EventHandlerDao.java:125)
	at com.serotonin.m2m2.rt.EventManager.setHandlers(EventManager.java:804)
	at com.serotonin.m2m2.rt.EventManager.raiseEvent(EventManager.java:111)
	at com.serotonin.m2m2.rt.event.type.SystemEventType.raiseEvent(SystemEventType.java:99)
	at com.serotonin.m2m2.util.timeout.RejectedRunnableEventGenerator.rejectedExecution(RejectedRunnableEventGenerator.java:58)
	at com.serotonin.timer.OrderedThreadPoolExecutor.rejectedExecution(OrderedThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
	at com.serotonin.timer.OrderedThreadPoolExecutor.execute(OrderedThreadPoolExecutor.java:179)
	at com.serotonin.timer.OrderedTimerThread.executeTask(OrderedTimerThread.java:28)
	at com.serotonin.timer.TimerThread.mainLoop(TimerThread.java:127)
	at com.serotonin.timer.TimerThread.run(TimerThread.java:38)
Caused by: org.h2.jdbc.JdbcSQLException: Out of memory. [90108-181]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:168)
	at org.h2.message.DbException.convert(DbException.java:289)
	at org.h2.engine.Database.openDatabase(Database.java:297)
	at org.h2.engine.Database.<init>(Database.java:260)
	at org.h2.engine.Engine.openSession(Engine.java:60)
	at org.h2.engine.Engine.openSession(Engine.java:167)
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
	at org.h2.engine.Engine.createSession(Engine.java:128)
	at org.h2.engine.Engine.createSession(Engine.java:26)
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353)
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385)
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227)
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	... 19 more

That's the killer for sure. Once the TimerThread is cancelled, you need to restart Mango. I wonder if your Mango/databases/mah2.h2.db file has grown, and that is why you're seeing memory issues now where you did not before. You can find many threads with instructions on shrinking the H2 database, but in short it's a SQL backup / restore. You may also need to purge your events, userEvents, and audit tables, as well as the pointValues table since from your logs I can see you are not using the NoSQL module.

ERROR 2018-11-07 10:14:06,918 (org.apache.taglibs.log.LoggerTag.doStartTag:97) - org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: The database has been closed [90098-181]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63)
	at com.serotonin.db.DaoUtils.queryForObject(DaoUtils.java:336)
	at com.serotonin.m2m2.db.dao.SystemSettingsDao.getValue(SystemSettingsDao.java:158)
	at com.serotonin.m2m2.db.dao.SystemSettingsDao.getValue(SystemSettingsDao.java:151)
	at com.serotonin.m2m2.db.dao.SystemSettingsDao.getColour(SystemSettingsDao.java:283)
	at com.serotonin.m2m2.util.chart.ImageChartUtils.writeChart(ImageChartUtils.java:74)
	at com.serotonin.m2m2.util.chart.ImageChartUtils.getChartData(ImageChartUtils.java:63)
	at com.serotonin.m2m2.util.chart.ImageChartUtils.getChartData(ImageChartUtils.java:55)
	at com.serotonin.m2m2.web.servlet.ImageChartServlet.getImageData(ImageChartServlet.java:233)
	at com.serotonin.m2m2.web.servlet.ImageChartServlet.doGet(ImageChartServlet.java:73)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:87)
	at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:47)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed [90098-181]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:179)
	at org.h2.message.DbException.get(DbException.java:155)
	at org.h2.message.DbException.get(DbException.java:144)
	at org.h2.engine.Database.checkPowerOff(Database.java:496)
	at org.h2.command.Command.executeUpdate(Command.java:274)
	at org.h2.jdbc.JdbcConnection.rollbackInternal(JdbcConnection.java:1501)
	at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:489)
	at org.h2.jdbcx.JdbcXAConnection.getConnection(JdbcXAConnection.java:107)
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:229)
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	... 44 more

Likely caused by a memory error that occurred before this log starts. After all, there's a lot of spam in there from the BACnet data sources consuming all the threads.

JAST

Hello Phil I tried all the things except the sql backup and restore thing because when I logged in mysql there was no sql database.So what can I do in this situation?

phildunlap

Did you get Mango restarted? It's all from the memory error. With SQL the database expansion can be managed just by keeping the tables mentioned in check (which you could do at the command line). It's not necessary to do a dump/restore - I assumed you were using the H2 database.

JAST

Yes I restarted the mango but it is same issue.I think there is something wrong with database. Beacuse when I removed the database file and started mango again it was working fine and it also created new h2database file.

phildunlap

It would not switch database types....

Do you have a Mango/databases/mah2.h2.db file and how big is it?

If you are using H2 then the MySQL command line program won't be of use to us.

JAST

It's size is 1.58Gb

phildunlap

You can use the /system_settings.shtm page to create a SQL backup (SQL Backup section), then stop Mango, move the existing database so that Mango starts on a new database, start Mango, and then restore the SQL backup in the same section on the /system_settings.shtm page. There's other information here: https://forum.infiniteautomation.com/topic/2748/how-to-restore-a-database-backup Having a smaller H2 database may help some, generally.

Yes I restarted the mango but it is same issue.

Which issue? The memory is instantly running out or there is a bind exception because something is already running on the port you are trying to launch Mango on?

JAST

memory is running out .
org.h2.jdbc.JdbcSQLException: Out of memory. [90108-181]

phildunlap

You can

Use Mango/bin/h2-web-console script (bat or sh) to connect to the H2 database without running Mango. This would allow you to delete from the events, userEvents, audit, or pointValues tables. I would guess you need to purge events and userEvents at the very least, like DELETE FROM events; DELETE FROM userEvents; Mango needs to be stopped.
Start in SAFE mode. This will start will all data sources disabled, if for some reason they're the source of the memory loss.
Start on a clean database and restore an existing backup using the SQL backup configuration section of the /system_settings.shtm page.
Ensure a proper amount of memory has been allocated to Mango in a Mango/bin/ext-enabled script.

JAST

I tried connecting to h2 databse via console after stopping mango.i got this error.
"IO Exception: "C:/Users/DCU/test outside C:/Mango" [90028-181] 90028/90028 (Help)
org.h2.jdbc.JdbcSQLException: IO Exception: "C:/Users/DCU/test outside C:/Mango" [90028-181]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.engine.ConnectionInfo.setBaseDir(ConnectionInfo.java:182)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:104)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
at org.h2.Driver.connect(Driver.java:72)
at org.h2.server.web.WebServer.getConnection(WebServer.java:750)
at org.h2.server.web.WebApp.login(WebApp.java:957)
at org.h2.server.web.WebApp.process(WebApp.java:211)
at org.h2.server.web.WebApp.processRequest(WebApp.java:170)
at org.h2.server.web.WebThread.process(WebThread.java:137)
at org.h2.server.web.WebThread.run(WebThread.java:93)
at java.lang.Thread.run(Unknown Source) "

phildunlap

C:/Users/DCU/test outside C:/Mango

The webserver cannot try to open a database URL outside its Mango directory. You're probably looking for a URL of jdbc:h2:C:/Mango/databases/mah2