Template Import JSON markup example

sstuhlmann

Can you provide an example of a marked up JSON that will work with the FTL conversion process? The documentation for template creation is thin. I was able to get a marked up version to convert to FTL but the output was not as expected. I was able to alter the FTL manually to get the required output but I would definitely like something with a few different markup types in order to hash out the nuances of the parcer and save myself loads of time.

Oh and FYI, the h2ftl script error-ed out at the 'print m.group(4)' statement. for reference, I am running python 3.6.4 for windows. I was able to comment out that line and it ran fine.

phildunlap

Hi ssthlmann,

Thanks for the question! Sometimes I wonder if this module goes overlooked, so I'm glad you're giving it a try!

In the course of answering this question, I found a bug in using the quadruple dollar sign without a property name, but I have fixed this and we'll release version 3.3.1 of the module shortly. Thanks for the note about the print statement. I will remove that from the script and then it should be compatible with python 2 or 3.

Here's my pre-processed JSON:

{
   "dataSources":[
      {
         "xid":"DS_Template_%%deviceNumber%%",
         "name":"%%deviceName%%",
         "enabled":true,
         "type":"VIRTUAL",
         "alarmLevels":{
            "POLL_ABORTED":"URGENT"
         },
         "purgeType":"YEARS",
         "updatePeriodType":"SECONDS",
         "polling":true,
         "updatePeriods":5,
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"DP_Templated_1",
         "name":"$$DataPointName1$$",
         "enabled":true,
         "loggingType":"ON_CHANGE",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"INSTANT",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"ALPHANUMERIC",
            "changeType":{
               "type":"NO_CHANGE",
               "startValue":"$$$$"
            },
            "settable":true
         },
         "eventDetectors":[
         ],
         "plotType":"STEP",
         "rollup":"NONE",
         "unit":"",
         "templateXid":"Alphanumeric_Default",
         "chartColour":"",
         "chartRenderer":{
            "type":"TABLE",
            "limit":10
         },
         "dataSourceXid":"DS_Template_%%deviceNumber%%",
         "defaultCacheSize":1,
         "deviceName":"%%deviceName%%",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":15,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "tags":{
         },
         "textRenderer":{
            "type":"PLAIN",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "suffix":""
         },
         "tolerance":0.0
      }
   ]
}

I ran the h2ftl.py script and created output.ftl, which looks like this:

<#--===MANIFEST===
deviceNumber:Global:Global
deviceName:Global:Global
name:prefix:DataPointName1
startValue:prefix:
===END MANIFEST===-->

{
   "dataSources":[
      {
         "xid":"DS_Template_<#if deviceNumberGlobalGlobal??>${deviceNumberGlobalGlobal}<#else>${defaultdeviceNumberGlobalGlobal}</#if>",
         "name":"<#if deviceNameGlobalGlobal??>${deviceNameGlobalGlobal}<#else>${defaultdeviceNameGlobalGlobal}</#if>",
         "enabled":true,
         "type":"VIRTUAL",
         "alarmLevels":{
            "POLL_ABORTED":"URGENT"
         },
         "purgeType":"YEARS",
         "updatePeriodType":"SECONDS",
         "polling":true,
         "updatePeriods":5,
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"DP_Templated_1",
         "name":"<#if nameprefixDataPointName1??>${nameprefixDataPointName1}<#else>${defaultnameprefixDataPointName1}</#if>",
         "enabled":true,
         "loggingType":"ON_CHANGE",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"INSTANT",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"ALPHANUMERIC",
            "changeType":{
               "type":"NO_CHANGE",
               "startValue":"<#if startValueprefix??>${startValueprefix}<#else>${defaultstartValueprefix}</#if>"
            },
            "settable":true
         },
         "eventDetectors":[
         ],
         "plotType":"STEP",
         "rollup":"NONE",
         "unit":"",
         "templateXid":"Alphanumeric_Default",
         "chartColour":"",
         "chartRenderer":{
            "type":"TABLE",
            "limit":10
         },
         "dataSourceXid":"DS_Template_<#if deviceNumberGlobalGlobal??>${deviceNumberGlobalGlobal}<#else>${defaultdeviceNumberGlobalGlobal}</#if>",
         "defaultCacheSize":1,
         "deviceName":"<#if deviceNameGlobalGlobal??>${deviceNameGlobalGlobal}<#else>${defaultdeviceNameGlobalGlobal}</#if>",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":15,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "tags":{
         },
         "textRenderer":{
            "type":"PLAIN",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "suffix":""
         },
         "tolerance":0.0
      }
   ]
}

which I then placed in Mango/web/templateConfigruation/web/ftl/ and navigated to the /templateConfiguration.shtm page. Upon importing my template, I got this dialogue:

Which imported a data source and one point that were these:

{
   "dataSources":[
      {
         "xid":"DS_Template_555",
         "name":"CoolTemplate",
         "enabled":true,
         "type":"VIRTUAL",
         "alarmLevels":{
            "POLL_ABORTED":"URGENT"
         },
         "purgeType":"YEARS",
         "updatePeriodType":"SECONDS",
         "polling":true,
         "updatePeriods":5,
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"DP_Templated_1",
         "name":"My Data Point",
         "enabled":true,
         "loggingType":"ON_CHANGE",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"INSTANT",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"ALPHANUMERIC",
            "changeType":{
               "type":"NO_CHANGE",
               "startValue":"foobar"
            },
            "settable":true
         },
         "eventDetectors":[
         ],
         "plotType":"STEP",
         "rollup":"NONE",
         "unit":"",
         "templateXid":"Alphanumeric_Default",
         "chartColour":"",
         "chartRenderer":{
            "type":"TABLE",
            "limit":10
         },
         "dataSourceXid":"DS_Template_555",
         "defaultCacheSize":1,
         "deviceName":"CoolTemplate",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":15,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "tags":{
         },
         "textRenderer":{
            "type":"PLAIN",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "suffix":""
         },
         "tolerance":0.0
      }
   ]
}

This is the issue I fixed: https://github.com/infiniteautomation/ma-core-public/issues/1204

But you will not be able to use the example provided here until 3.3.1 of the template configuration module is released. However, you could modify two lines in output.ftl and it will work:

startValue:prefix: --> startValue:prefix:A
"startValue":"<#if startValueprefix??>${startValueprefix}<#else>${defaultstartValueprefix}</#if>"
-->
"startValue":"<#if startValueprefixA??>${startValueprefixA}<#else>${defaultstartValueprefixA}</#if>"

sstuhlmann

So an update. . .
I was better able to parse out the tags issue. I am in the process of re-writing the python script to better handle multiple globals on a single line. Example:

"xid" : "%%Prefix%%_%%Device Name%%-suffix"
when imported converts to:
"xid" : "PrefixInput_DeviceNameInput-suffix"
where Prefix and Device Name are globals.

I also added code into the python script to create some Freemarker variables with default data. Combined with an altered <if> conditional, it prevents fields left blank from causing errors or appearing as blank data. I'm sure I'll code into it to allow the data inputer to designate when they want to actually leave a data block blank, ie for a data point property value that you want blank like say unit.

phildunlap

Nice! I would have been happy to attack that issue, it does seem like you should be able to use multiple annotations on a single line (but you're right, that currently wouldn't work). If you want to share back what you come up with we'd certainly enjoy taking a look at it! And, similarly, if you find some behavior in the Java code is undesirable let us know (specifically, I can't remember if there's any checks inside the Java code or the UI that prevents you from having blank fields) and I'll have a look, as that module is not open sourced.

sstuhlmann

I haven't fully tested but I did get some errors when leaving certain things completely blank. It did allow me to put spaces into the xid: fields although I expected an error.

I'm using this to create templates for my virtual devices. I'll use these in sales and testing demos. Once I get all of my SQL structure completed and all of the data imported into it, I will write a fully functioning separate app to create templates dynamically. This will give me maximum flexibility with minimal data entry time.

I'll post the new python script once I have fully functioning code.

On a similar note, I have another question. Is the JSON structure that exports for data sources, data points, point property template, etc a fully realized data model? Meaning, if I make and export a modbus/ip source, are all of the JSON tags for a modbus/IP source there? Even if I don't utilize that structure component. From a quick glance, it looks like it is. I ask because I want to use that as a model for conversion of data from my program to an import capable JSON. I would just need to create and export all of the data source types, data points, etc. What is the likelihood of changes to the JSON data model for a given data source within the same major revision?

Some of these questions are probably better suited for the developer forum.

phildunlap

Thanks!

Yes, it is fully realized (with few special exceptions in the system settings: you cannot import database schema version numbers). It is unlikely that data sources or points would change in such a way as to invalidate previous JSON. If there were a change, it would most likely be the addition of a property that would have a class default, so it could be foregone in old JSON without issue. We try to stay conscious of this so that people's backups do not get invalidated.