Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

Ideas/suggestions for auto creating data points


  • Heya,

    I am looking for ideas/suggestions on how to auto create data points based on an event.

    The workflow is:

    1. A new sensor node is added to the network
    2. The new sensor begins transmitting messages to the MQTT server
    3. The MQTT server creates new topics for the data in the messages
    4. Mango generates warnings that the topic is unknown (as there are no data points associated to that topic)

    WARN 2018-12-10T00:27:48,866 (com.infiniteautomation.m2m2.mqtt.client.MqttClientDataSourceRT.messageArrived:355) - Unknown topic "nodeData/L1-CS03-01/TEMP" payload: "{"rawTemperature":362,"temperature":22.625,"collectTstamp":"2018-12-10T08:28:00.696Z","deviceDataTypeId":0,"sequenceNum":34,"deviceId":"L1-CS03-01","status":1,"snr":9}"

    My goal is to create an event handler for this warning that will trigger the creation of all the data points (raw, virtual, meta, event handlers, etc) needed for that topic. Then the next time data is published to that topic, the data will be loaded into the Mango data source as expected. I'm fine with missing the data from the first publishing of the data.

    I have explored templates a bit but I am a little unsure how to go about configuring them to work automatically, linking them to an event for example. My experience with them so far is using them from the template configuration page.

    I am a little stuck as to if this is a good direction to take or if there might be a better option. Any suggestions would be appreciated.

    Thanks
    Ian


  • I'll come back later to give more input, (I know someone at inifninte will most certainly have a handler you could likely use to callback and generate the points required). But one thing I do note is the fact you ensure you have a consistent naming convention, 2) What will you do if a different sensor connects and requires a different number of points? You'll need some form of identifier to generate the correct points - I'd recommend a JSON object to use as a template to state the points required.

    Finally if you're using an mqtt server, if it were powered by nodeJS, you could use the mangoAPI to generate all you need from the server itself after firing a few tests when a new topic is announced then generate and fire.
    Just my 10c.


  • Hi Ian,

    I would do this inside a Email Event Handler, but you could use an Set Point Event Handler as another option. There is a script utility JsonEmport that I gave an example usage of here: https://forum.infiniteautomation.com/topic/3708/a-script-jsonemport-example and I've done similar things in other threads, such as this one: https://forum.infiniteautomation.com/topic/2999/intesis-integration

    TLDR you can do something like,

    var basePoint = { //The JSON from an internal point's export
          "purgePeriod": 1,
          "setExtremeLowLimit": -1.7976931348623157e+308,
          "intervalLoggingType": "INSTANT",
          "dataSourceXid": "internal_mango_monitoring_ds",
          "discardHighLimit": 1.7976931348623157e+308,
          "deviceName": "Mango Internal",
          "enabled": true,
          "rollup": "NONE",
          "purgeOverride": false,
          "chartRenderer": null,
          "xid": "internal_mango_num_excel_reports",
          "eventDetectors": [],
          "textRenderer": {
            "unit": "",
            "useUnitAsSuffix": false,
            "renderedUnit": "",
            "type": "PLAIN",
            "suffix": ""
          },
          "loggingType": "ON_CHANGE",
          "overrideIntervalLoggingSamples": false,
          "defaultCacheSize": 1,
          "readPermission": null,
          "setPermission": null,
          "intervalLoggingPeriodType": "MINUTES",
          "intervalLoggingSampleWindowSize": 0,
          "tolerance": 0,
          "purgeType": "YEARS",
          "plotType": "STEP",
          "intervalLoggingPeriod": 15,
          "discardExtremeValues": false,
          "tags": {},
          "unit": "",
          "pointLocator": {
            "monitorId": "com.infiniteautomation.mango.excelreports.dao.ExcelReportDao.COUNT"
          },
          "discardLowLimit": -1.7976931348623157e+308,
          "setExtremeHighLimit": 1.7976931348623157e+308,
          "name": "Excel reports",
          "chartColour": "",
          "preventSetExtremeValues": false,
          "simplifyType": "NONE"
        };
    
    basePoint.xid = "Some new XID!";
    
    JsonEmport.setImportDuringValidation(true); //Otherwise it must be run normally to import
    //So you don't actually need that line in your solution
    
    print(JsonEmport.doImportGetStatus(JSON.stringify( {"dataPoints":[basePoint]} )));
    // ^ This will not be so readable, but it would be a list one could iterate
    // and see the same messages that feed back to the regular import / export page
    // You would want to use: 
    // JsonEmport.doImport(JSON.stringify( {"dataPoints": yourDataPointsToCreateList} ));
    

    And you'll create a point with some new xid on the data source you took the example JSON from.

    In an event handler's script, you can see available methods on the event that triggered the script if you wanted to parse out a new point name or something, like print(event); //event handlers only

    In an email handler, you can return CANCEL; at the end of the script to abort sending the email. return UNCHANGED; also works, and can be used in a set point handler script to not set a value to the target point.


  • Hi Phil,

    I am circling back to this task and have been successful in creating MQTT and virtual data points. Creating meta data points doesn't seem to be that much different but I am stuck on how to set the var(iable) names for context points.

    I see that when I create meta points with context via the UI, the variable name will be automatically populated, such as "p500". How/where is this var name determined? When I create multiple meta points using the same data point in context, the same var name is used so I figured it must be maintained somewhere. However, I have yet been able to find the method that constructs it. I figure I could use my own var names but ideally I would like to use existing methods for creating/using these names for consistency.

    Any suggestions would be great.

    Thanks
    Ian


  • Hi Ian,

    The number next to the p is the dataPointId, its id column in the database. I would not consider it particularly important that all variable names abide this convention. In fact, should you recreate your system via JSON import at some point all those numbers may change!

    If you needed to know that while structuring the JSON to import a meta point, you could use the DataPointQuery.byXid("context point xid here").getId() for points that exist before the meta point. If you were trying to do the import at the same time you'd have to track the auto_increment state of the dataPoints table in the database, which is assuredly more trouble than its worth (in my eyes).


  • Hi Phil,

    I decided to use the data point id at this time as it was easy to use and meant I didn't need to construct my own unique id process. I tend to avoid using internal id values like that for external references but it works well enough for now.

    My process currently creates points for a "device" which includes 9 MQTT, 4 virtual and 12 meta points. I have created a basic template for each of the 3 types of points, which the process then populates based on the a list of points required. Works quite nicely.

    Now that I am trying to connect the process to email event handler (or set point handler) I am stuck with a simple problem: what is the correct event object? My handler script is:

    LOG.info('Unknown topic handler executed...');
    topicTest(event.getMessage());

    return CANCEL;

    Where topicTest is a global function that logs the message.

    If I validate the script, the log messages appear in the page.
    Success, email sending cancelled
    INFO 2019-03-12 14:55:11,950 - Unknown topic handler executed...
    INFO 2019-03-12 14:55:11,952 - Unknown topic received from MQTT server:Test

    When I trigger the handler for the first time, I get an error in the log that "event" is not defined. Any further triggers of the handler appear to be ok (the original warning message is logged).

    However, I only see the original warning in the ma.log file but not the 2 info messages. Is there something else I need to do?

    Thanks
    Ian


  • Hi Ian,

    It looks like evt is the right key in email scripts. The validation code uses the same code as the set point handler, so that's why the validation would have worked. evt was probably used because it was already in the model under that key being passed to the Mango/ftl/ files. It's easy to have it under both keys, though. Thanks for bringing it to our attention!

    Edit:
    evt is not quite the same. It is an EventInstance instead of an EventInstanceWrapper, so you'd perhaps want evt.getMessageString() instead of evt.getMessage() if you use evt . I will add the EventInstanceWrapper under the event key.


  • Hi Phil,

    I am hoping I am missing something very basic here. When I try to log the message string from evt, I still get an exception that the object is not defined.

    LOG.info(evt.getMessageString());

    return CANCEL;

    Failure: ReferenceError: "evt" is not defined in <eval> at line number 2 in <eval> at line number 2

    I had tried using evt before as I saw it being referenced in the ftl markup (and pop-up help) and appeared to be set to the model in EmailHandlerRT but had no success then.

    It appears that the evt object is available in the model (and thus the ftl) but not to the script.

    Thanks
    Ian


  • That's during validation, right? Unfortunately for the email handler they're not quite the same. You could have a little preamble like this,

    if(typeof evt === 'undefined')
      evt = event;
    if(typeof event === 'undefined')
      event = evt;
    

    I did fix this for 3.6, adding event into the runtime of the Email Handler as an EventInstanceWrapper, and evt as the EventInstance in validation.