Hi @mattfox ,

You can indeed! Tags will definitely help in this instance.
Like I said, we do not need to have your exact code. But! If we can mimic it, you will be in a better position to realise a solution.

Alright, excellent! Let me first find the time to try and do it on my own, and if something fails I'll let you know how far I got here.

With your help the emergency is gone for now, so we can take some time to improve it.

Thank you, have a great day!

Hey, @mattfox !

Okaaaay I think I get it now, looks like I should really take some time to program and test following your suggestions. From what I'm seeing, the use of "this" might be the key.

For the RQL query, how can I identify the points if the XID and variable name aren't an option? Can data point tags help with this?

By the way, it was very dumb from me to use "eval" as a function name example, but that's not my actual function name and I'm not using the javascript eval, was just too lazy to write something like "evaluationFunction" hahaha. I think my function is called something like "calculateResults" in spanish.

Many thanks once again!
Greetings.

@mattfox wow okay thank you!

Just a doubt with this part:

//Going out on a limb here...
pt.set(  generateCmdString("id", "LocId") );

This is I think my main problem, what I'm probably not getting is, how can "pt" refer to the different points that I have?

My lines are mostly:

location1_deviceId_value1.set(result1);
location1_deviceId_value2.set(result2);
...
locationN_deviceId_valueN.set(resultN);

So from your ideas (which are great) what I'm still not getting is how to call the different set methods for the different points based on the generated strings? This is probably not possible but if it is, it probably looks kinda like:

//Generating strings that are the variable names
var variableNames=generateCmdString(id, locId);

//Calling the read function
var valuesToEval=readValues(variableNames);

//Reading script variables
//**my doubt is here
function readValues(variableNames){
var values=[];
for(var k=0; i<variableNames.length; k++){
values.push(${variableNames[k]}.value); //how to do this?
}
return values;
}

//Analizing
var results=eval(values);

//Setting
//**my doubt is also here
for(var j=0; j<variableNames.length; j++){
${variableNames[j]}.set(results[j]); //how to do this?
}

I'm not sure if this would even be necessary if I followed your approach, but the

pt.set(  generateCmdString("id", "LocId") );

line confused me a bit. If I'm wrong and this isn't necessary, the part that I'm probably not getting is the first one, where the locations object is defined. Would there be functions inside that object?

Greetings.

Hey @mattfox ,

Sorry, I am not allowed to share code here. Will try to explain what's going on though, hopefully you'll be able to understand. Not sure if this is the best approach but this is working atm.

Yes, it would be awesome to be able to reuse code, probably if I knew more javascript I would already know the way to do so, but for now what makes the lines grow is that the same script sets 440 datapoints and checks values for them too. Those 440 datapoints correspond to 20 locations (same 22 datapoints for each location).

Right now what happens is that I get a string, this is my context variable. This string first gets split into ID and data, then that ID is looked up in an object array, with an object for each location. According to where it's found it will determine which location corresponds to that ID, enters a switch, and fires a specific function for that location.

In that function, which is the same for each location and this is why there are so many lines, what happens first is that the specific values for that location are read from the variable names for that location, then stored in an object, then that object is sent to an evaluation function (this function is perfectly reused by all 20 locations) that analyzes them and returns more values, then those values are used to set the location's values using the specific variable names for that location.

I do have a scructure for variable names, so the only things I would need is the location's id and the id from the data, and from those two IDs I could construct the strings that would represent the variable names to read and set. I mean, I could already generate those strings, but what I don't know how to do is how to pass those strings to a function that reads and sets the variables (datapoints) named as the strings that I generated with those two IDs.

I hope I am being somewhat clear with my explanation.
Many thanks in advance!

@mattfox Yes, this helped, had to move around 200 data points to the new source and divide my device group into 2, transfer the devices etc, a bit of a hassle, but now it works better. Some odd behaviour I started to notice is now gone.

Lesson learned:
Will keep Scripting source data points below 500 and script lines below 1600.

Thanks a lot!

@mattfox Will try that, thanks.

Hello,

I have a scripting datasource on a MangoES that sets lots of different points on change of an HTTP receiver data point.

Today I tried growing the script with more places I had to add but the Data Source is throwing an error I had never seen before:

Error saving data source: Internal Server Error — com.serotonin.ShouldNeverHappenException: java.io.UTFDataFormatException

It seems to talk about the encoding, but I didn't change it from one script version to the next one.

Now, it looks like the data source is disabled even though it's supposed to be enabled.

The script is about 1,900 lines long.

Any idea of why this could be happening? I really need to be able to update the data source soon, I'm losing data :(

@mattfox got it! Thanks a lot, man!

@mattfox I tried removing the plastic case but I haven't figured out the trick to release the corner with the red clip that grabs onto the DIN rail. Any advice so I can avoid breaking it?

Oh well, I'm sorry but I take it back, seems like we can't stop the loop, even with the Log4j approach we are still getting thousands of events. More than 24,000 in a couple of hours. Most of them are like this:

ERROR 2019-10-29T11:44:19,621 (com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.postMessage:145) - Slack message failed to send java.net.ConnectException: ratelimited at com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connectImpl(SlackWebSocketSessionImpl.java:338) ~[?:?] at com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connect(SlackWebSocketSessionImpl.java:292) ~[?:?] at com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.postMessage(SlackMessageEventHandlerRT.java:122) ~[?:?] at com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.eventRaised(SlackMessageEventHandlerRT.java:67) ~[?:?] at com.serotonin.m2m2.rt.EventManagerImpl.handleRaiseEvent(EventManagerImpl.java:1048) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.EventManagerImpl.raiseEvent(EventManagerImpl.java:243) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.event.detectors.PointEventDetectorRT.raiseEvent(PointEventDetectorRT.java:43) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.event.detectors.PointChangeDetectorRT.pointChanged(PointChangeDetectorRT.java:54) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:817) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.6.4.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.6.4.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.6.4.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) ~[mango-3.6.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

Couldn't find out what is triggering these changes that raise the event detector.

Tried turning off the "Error" data point but now the "Warning" one is also sending weird messages, like this one:

WARN 2019-10-29T12:01:45,429 (com.serotonin.m2m2.web.dwr.MiscDwr.jsError:167) - Javascript error Description: Error: Tried to register widget with id==chartRendererSelect but that id is already registered Page: http://192.168.100.33:8088/resources/dojo/dojo.js?v=1571864830 line 345 > Function line 3 > eval Line: 2 Browser name: Firefox Browser version: 71 osName: Windows location: http://192.168.100.33:8088/data_source_edit.shtm?dsid=498 @ Oct 29, 2019 12:01 PM

We are getting this message when we try to edit the Log4j data points in the old UI (so we can access the event detectors and the purge now button). Explanation below:

Strangely, when we try to edit the data source in the old UI, the "Data points" tab is not existent inside the data source. So I look for the points in the global "Data points" tab, where all data points appear, I filter them by name to find the Log4j Slack Messages Data Points, but most times, when I click the edit (pencil) button on any of the 4 data points, I only get the above "Warning" message in Slack and this same message on a blue box in the old UI:

Error: Tried to register widget with id==chartRendererSelect but that id is already registered

It works super randomly; I seldom get to edit the data point. It's a strange behaviour, something I never saw before with any other data source. I did add the line @terrypacker suggested in the log4j2.xml, inside the loggers section, and restarted it btw.

Is there a quick fix to this? Thanks!

@terrypacker Just finished configuring the Log4j approach and you were absolutely right, it's perfect for what we wanted. Works well too.

Thanks a lot!

@jared-wiltshire said in AI, Machine Learning, Neural Networks, TensorFlow:

Yes, no problem running TensorFlow JS in the cloud on NodeJS. However I am not sure what advantage this would afford over the native binaries.

True, I'm not sure either, just trying to think about all the possibilities, hehe.

@terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

@jflores13 said in AI, Machine Learning, Neural Networks, TensorFlow:
Something to look into in the future for sure, but that approach would have its own problems to overcome. For example getting the data from Mango to the model.

Yeeeah, I don't visualize how that interaction could be simplified. I guess it's just not as simple as I was hoping.

I'll keep looking for possible solutions to this and will share anything I can find.

Thanks!

@terrypacker ahhh, I see! Alright, don't worry, thanks for that quick fix, I will definitely try the Log4J approach.

@terrypacker said in Slack Log Settings - Only one log level?:

That would be a good feature for this module to have, a way to queue and then retry messages that have been rejected for rate limiting reasons. This thread will be a good reference if/when we can get enough support to do more development on this module. I could be persuaded to create a public fork of the Slack module if anyone wants to make some changes and submit a pull request...

I would love to contribute but, to be honest, I don't think my coding skills are enough for that yet. I'll keep learning and hopefully I can persuade you soon.

Thanks again, Terry! :)

Hello,

@craigweb said in Slack Log Settings - Only one log level?:

Hi @jflores13
This is not possible with the slack log publisher.

Alright, thanks, @CraigWeb.

@terrypacker said in Slack Log Settings - Only one log level?:

After thinking about this more I believe you could use the LOG4J data source with an event detector which would raise events that could be handled by Slack Event Handlers. This would work as follows:

1. Create Log4J data source with data points for the various LOG levels you want to send to slack
2. Add an on change event detector to each point.
3. Add a Slack event handler to each detector to publish the event message to the channel.

Note that I just released a new version of this module for Mango 3.6 as while I was testing this I found a bug and also realized that it did not post the latest value of the point in the event message (which it does now).

Wow, @terrypacker, thanks for taking the time to think it through and to investigate. We will definitely try this LOG4J approach.

We actually noticed strange behaviour before, because when we changed the log level to Information, the messages we started getting were empty, but we thought we were doing something wrong. Today, we updated the module at 13:52. We got the info messages of the startup and modules loading correctly now, but right after, the extreme error messages started coming. This is the error message copied from our Slack channel:

ERROR 2019-23-10T16:31:51.825  (com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connectImpl:337) - Error during authentication : ratelimited

This same error was sent 4,050 times from 13:53 to 14:17, then it suddenly stopped. We then decided to add a module to test if it happened again, and it did. When Mango restarted, we got 3,626 error messages from 16:08 to 16:31. Apparently, all of them are the same error.

I looked that up in Slack and found out about the rate limits, and the raw HTTP response that you get when you hit the limits. Apparently, you get an HTTP 429 Too Many Requests error. According to Slack, the raw response could look like this:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

I'm thinking that the handling of this Retry-After header probably has something to do with this, am I wrong?

Thanks again!
Greetings!

Hello, @terrypacker and @Jared-Wiltshire.

@terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

The examples you have seen were my attempts to do some modeling directly in Java which does work nicely but is quite time consuming since many features that are built into ML languages have to be hand coded in Java. I've considered adding support for Python and R but we haven't really enough interest yet to drive the development.

Interesting, it's pretty cool that you've tried it. Is it possible that something cloud-based, maybe a combo similar to lambda functions + Amazon API Gateway, could make the ML model more available and reduce the need for Python or R support in the same environment?

I have no idea if this could be done, but if the answer is yes, I think it could be charged as an extra service/module. Still, I would totally understand if the development doesn't start until more people show interest, which I hope happens soon.

@terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

However I tried to download and add the minified version as a global script and it didn't work.

I was thinking about trying that, thanks for saving me the test time.

@jared-wiltshire said in AI, Machine Learning, Neural Networks, TensorFlow:

It is highly likely that it wont work in Nashorn (the Java JS engine) as it doesn't support the latest ECMAScript features. Not to mention that there may be memory/script size limits.

Hmmm. I understand. Well, maybe the memory/size limits could also be avoided with the cloud-based approach, right?

Alright, thanks a lot for your time and answers, guys!

Hello, dear IAS devs.

We are using Slack for our alerts and, for dev purposes, we wanted sepparate channels for the different log levels.

I don't know if we're missing out on something, but my question is, what if we wanted to send different log levels to different channels? Is this possible?

Via the System Settings UI, there doesn't seem to be an option to do this, but through my Mango learning curve I've found out multiple times that I'm just looking for stuff in the wrong place. Hope this is also the case.

Thanks in advance!
Greetings.

Hello,

I would like to start interacting with ML to get ahead of the competition, and I'm curious, is there a way to implement TensorFlow or something similar to the Mango as a module? Is this line of development visualized for the future?

Last week, I was exploring Github repositories related to Infinite Automation, and found that @terrypacker has worked in an experimental ML module. Am I right? Is the project still in development?

Thanks!

Hey, Phil!
I know, many thanks for your lots of answers.

Yes, They are stored in the same property on the point locator because they can be. If a numeric point, the value format is interpreted as a java.text.DecimalFormat, and if binary it is simply string matched with what is captured in the value regex (capture group 1)

Perfect! I figured that was the case, just wanted to make sure. I had gotten the last part wrong though, so thanks for explaining that!

That definitely seems like it would be part of the problem. I'm not sure why the POP3 data source didn't expose the port as an option, it's certainly easy to do, so I have opened a git issue about this in the repository the module is in. Somewhat impassable issue until that is configurable

Alright then, I'll be waiting.

Thanks again! Greetings.