Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
NoSQL Task Queue Full
-
@phildunlap said in NoSQL Task Queue Full:
The task queue message is probably incidental to the crash. If you change your spawn threshold / max instances you can see this (force it to have max instances all the time with a low spawn threshold, the messages will go away, but it will probably still crash).
Phil would you elaborate a little on the above comment, what is considered a low threshold?
Our situation is similar to the one discussed in this thread in that our systems have a limited number of points < 300 and very lengthy periods 2-5 years on every minute so about 100 million of values a year. I noticed your remark about streaming as opposed to loading the entire period's values in memory, Are there any plans to move in this direction? I have tried your tuning suggestions above and I believe you correctly suggested shortening the time period is the most obvious solution so I modified the run loop script you offered in the other thread so the the period gets divided-up and points processed repediavely and linearly keeping things under control with a RuntimeManager.sleep(5000) thrown in for good luck if the pointsTBW gets out of control. Our system was consistently running out of memory before I read this thread and now that I understand why a little better the result is crashes have been eliminated completely on metapoint regenerations. Combined with your tuning suggestions in this thread, the system works much better and we get hassle-free regeneration over a lengthy periods of a year or more.I also remember something discussed in another thread about automatically regenerating the missing metapoints whenever the source data sync runs; Is there any motivation within your collective talents to create this feature?
Thanks in advance. -
Phil would you elaborate a little on the above comment, what is considered a low threshold?
In the context of that post, anything that would cause the maximum number of batch write behind instances to exist in regular operating conditions.
I noticed your remark about streaming as opposed to loading the entire period's values in memory, Are there any plans to move in this direction?
I can't say if anyone will want to change anything about it, but I began I REST controller that did streaming values (or optionally load them all), generating multiple points' histories in one request, and provided the ability to cancel a history generation task. There was some stuff left there to be finished / fixed up, but I'm sure some more feature-ful REST controller is on its way.
I also remember something discussed in another thread about automatically regenerating the missing metapoints whenever the source data sync runs; Is there any motivation within your collective talents to create this feature?
Possible, but unlikely. I don't think you should have meta points on the receiver end unless you really cannot have them on the publisher's side. You could already take that history generation loop I gave you (invokes the meta edit dwr from a loop of points to generate iirc) and place that into a set point event handler for the persistent receiver's sync completed event. So, in that sense it is possible and not my favorite idea, so I'm unlikely to encourage it further without good cause.
There some discussion of this in this thread: https://forum.infiniteautomation.com/topic/3189/using-persistent-tcp-points-in-script-calculations
