Ideas/suggestions for auto creating data points

iperry

Heya,

I am looking for ideas/suggestions on how to auto create data points based on an event.

The workflow is:

A new sensor node is added to the network
The new sensor begins transmitting messages to the MQTT server
The MQTT server creates new topics for the data in the messages
Mango generates warnings that the topic is unknown (as there are no data points associated to that topic)

WARN 2018-12-10T00:27:48,866 (com.infiniteautomation.m2m2.mqtt.client.MqttClientDataSourceRT.messageArrived:355) - Unknown topic "nodeData/L1-CS03-01/TEMP" payload: "{"rawTemperature":362,"temperature":22.625,"collectTstamp":"2018-12-10T08:28:00.696Z","deviceDataTypeId":0,"sequenceNum":34,"deviceId":"L1-CS03-01","status":1,"snr":9}"

My goal is to create an event handler for this warning that will trigger the creation of all the data points (raw, virtual, meta, event handlers, etc) needed for that topic. Then the next time data is published to that topic, the data will be loaded into the Mango data source as expected. I'm fine with missing the data from the first publishing of the data.

I have explored templates a bit but I am a little unsure how to go about configuring them to work automatically, linking them to an event for example. My experience with them so far is using them from the template configuration page.

I am a little stuck as to if this is a good direction to take or if there might be a better option. Any suggestions would be appreciated.

Thanks
Ian

MattFox

I'll come back later to give more input, (I know someone at inifninte will most certainly have a handler you could likely use to callback and generate the points required). But one thing I do note is the fact you ensure you have a consistent naming convention, 2) What will you do if a different sensor connects and requires a different number of points? You'll need some form of identifier to generate the correct points - I'd recommend a JSON object to use as a template to state the points required.

Finally if you're using an mqtt server, if it were powered by nodeJS, you could use the mangoAPI to generate all you need from the server itself after firing a few tests when a new topic is announced then generate and fire.
Just my 10c.

phildunlap

Hi Ian,

I would do this inside a Email Event Handler, but you could use an Set Point Event Handler as another option. There is a script utility JsonEmport that I gave an example usage of here: https://forum.infiniteautomation.com/topic/3708/a-script-jsonemport-example and I've done similar things in other threads, such as this one: https://forum.infiniteautomation.com/topic/2999/intesis-integration

TLDR you can do something like,

var basePoint = { //The JSON from an internal point's export
      "purgePeriod": 1,
      "setExtremeLowLimit": -1.7976931348623157e+308,
      "intervalLoggingType": "INSTANT",
      "dataSourceXid": "internal_mango_monitoring_ds",
      "discardHighLimit": 1.7976931348623157e+308,
      "deviceName": "Mango Internal",
      "enabled": true,
      "rollup": "NONE",
      "purgeOverride": false,
      "chartRenderer": null,
      "xid": "internal_mango_num_excel_reports",
      "eventDetectors": [],
      "textRenderer": {
        "unit": "",
        "useUnitAsSuffix": false,
        "renderedUnit": "",
        "type": "PLAIN",
        "suffix": ""
      },
      "loggingType": "ON_CHANGE",
      "overrideIntervalLoggingSamples": false,
      "defaultCacheSize": 1,
      "readPermission": null,
      "setPermission": null,
      "intervalLoggingPeriodType": "MINUTES",
      "intervalLoggingSampleWindowSize": 0,
      "tolerance": 0,
      "purgeType": "YEARS",
      "plotType": "STEP",
      "intervalLoggingPeriod": 15,
      "discardExtremeValues": false,
      "tags": {},
      "unit": "",
      "pointLocator": {
        "monitorId": "com.infiniteautomation.mango.excelreports.dao.ExcelReportDao.COUNT"
      },
      "discardLowLimit": -1.7976931348623157e+308,
      "setExtremeHighLimit": 1.7976931348623157e+308,
      "name": "Excel reports",
      "chartColour": "",
      "preventSetExtremeValues": false,
      "simplifyType": "NONE"
    };

basePoint.xid = "Some new XID!";

JsonEmport.setImportDuringValidation(true); //Otherwise it must be run normally to import
//So you don't actually need that line in your solution

print(JsonEmport.doImportGetStatus(JSON.stringify( {"dataPoints":[basePoint]} )));
// ^ This will not be so readable, but it would be a list one could iterate
// and see the same messages that feed back to the regular import / export page
// You would want to use: 
// JsonEmport.doImport(JSON.stringify( {"dataPoints": yourDataPointsToCreateList} ));

And you'll create a point with some new xid on the data source you took the example JSON from.

In an event handler's script, you can see available methods on the event that triggered the script if you wanted to parse out a new point name or something, like print(event); //event handlers only

In an email handler, you can return CANCEL; at the end of the script to abort sending the email. return UNCHANGED; also works, and can be used in a set point handler script to not set a value to the target point.

iperry

Hi Phil,

I am circling back to this task and have been successful in creating MQTT and virtual data points. Creating meta data points doesn't seem to be that much different but I am stuck on how to set the var(iable) names for context points.

I see that when I create meta points with context via the UI, the variable name will be automatically populated, such as "p500". How/where is this var name determined? When I create multiple meta points using the same data point in context, the same var name is used so I figured it must be maintained somewhere. However, I have yet been able to find the method that constructs it. I figure I could use my own var names but ideally I would like to use existing methods for creating/using these names for consistency.

Any suggestions would be great.

Thanks
Ian

phildunlap

Hi Ian,

The number next to the p is the dataPointId, its id column in the database. I would not consider it particularly important that all variable names abide this convention. In fact, should you recreate your system via JSON import at some point all those numbers may change!

If you needed to know that while structuring the JSON to import a meta point, you could use the DataPointQuery.byXid("context point xid here").getId() for points that exist before the meta point. If you were trying to do the import at the same time you'd have to track the auto_increment state of the dataPoints table in the database, which is assuredly more trouble than its worth (in my eyes).

iperry

Hi Phil,

I decided to use the data point id at this time as it was easy to use and meant I didn't need to construct my own unique id process. I tend to avoid using internal id values like that for external references but it works well enough for now.

My process currently creates points for a "device" which includes 9 MQTT, 4 virtual and 12 meta points. I have created a basic template for each of the 3 types of points, which the process then populates based on the a list of points required. Works quite nicely.

Now that I am trying to connect the process to email event handler (or set point handler) I am stuck with a simple problem: what is the correct event object? My handler script is:

LOG.info('Unknown topic handler executed...');
topicTest(event.getMessage());

return CANCEL;

Where topicTest is a global function that logs the message.

If I validate the script, the log messages appear in the page.
Success, email sending cancelled
INFO 2019-03-12 14:55:11,950 - Unknown topic handler executed...
INFO 2019-03-12 14:55:11,952 - Unknown topic received from MQTT server:Test

When I trigger the handler for the first time, I get an error in the log that "event" is not defined. Any further triggers of the handler appear to be ok (the original warning message is logged).

However, I only see the original warning in the ma.log file but not the 2 info messages. Is there something else I need to do?

Thanks
Ian

phildunlap

Hi Ian,

It looks like evt is the right key in email scripts. The validation code uses the same code as the set point handler, so that's why the validation would have worked. evt was probably used because it was already in the model under that key being passed to the Mango/ftl/ files. It's easy to have it under both keys, though. Thanks for bringing it to our attention!

Edit:
evt is not quite the same. It is an EventInstance instead of an EventInstanceWrapper, so you'd perhaps want evt.getMessageString() instead of evt.getMessage() if you use evt . I will add the EventInstanceWrapper under the event key.

iperry

Hi Phil,

I am hoping I am missing something very basic here. When I try to log the message string from evt, I still get an exception that the object is not defined.

LOG.info(evt.getMessageString());

return CANCEL;

Failure: ReferenceError: "evt" is not defined in <eval> at line number 2 in <eval> at line number 2

I had tried using evt before as I saw it being referenced in the ftl markup (and pop-up help) and appeared to be set to the model in EmailHandlerRT but had no success then.

It appears that the evt object is available in the model (and thus the ftl) but not to the script.

Thanks
Ian

phildunlap

That's during validation, right? Unfortunately for the email handler they're not quite the same. You could have a little preamble like this,

if(typeof evt === 'undefined')
  evt = event;
if(typeof event === 'undefined')
  event = evt;

I did fix this for 3.6, adding event into the runtime of the Email Handler as an EventInstanceWrapper, and evt as the EventInstance in validation.