Temporal observational data on collected data

[chrapchp]

We want to set observations on hydroponic system. We have growing chambers and different sources of nutrients. Some of the data collected includes CO2, temps, humidity, EC, pH, etc. What we would like to do is add meta data. We tried via virtual data sources and question if there is a better way as adding a tag by chamber seems cumbersome.

I was thinking of a JSON store. What makes it unique is the growing chamber + nutrient and the plant type.

Here is a usage scenario
At time t, add an observation e.g. for growing chamber X, nutrient tank Y, plant type Z, such as the characteristics of the plants, and other types of observations that could influence the growth. In the future, the data collected could be analyzed in the context of these observations.

Is the JSON route appropriate?
peter

[terrypacker]

Peter,

Our MangoNoSQL database has this ability built in. It sounds like a JSON store would work but since you are going to track time series data the NoSQL auxiliary database would be a better place for the meta data.

It basically works by allowing JSON data to be stored at any given timestamp and linked to an existing data point in Mango. You can only store 1 JSON object at any given timestamp for a point, but this is easily managed by editing any data that already exists instead of replacing it.

See the REST endpoints at: rest/v2/nosql/point-data

[phildunlap]

Hi peter,

It certainly could work. You're thinking make some kind of XID for the JSON store item out of the chamber-nutrient-plant pairs and then store something like

{"observations":[ {"time": "2018-1-1 12:00:00", "note": "Oddly purple..."} ]/*, other top level information */ }

? If that seems straightforward to you (easily manipulating that list is less cumbersome than the data point), then go for it! It will certainly run well up to tens of thousands of entries.

Another way to do this could be to have a scripting data source with a single point that you make observations into. This point updates the script's context, so the script runs. It checks the point's identifier and information, and creates the new observations point if necessary. Like so,

function createObservationPoint(chamberKey) { //This will create the new points for us
  //baseObservationPoint is the xid of an enabled, settable alphanumeric point on the scripting data source that runs this script.
  var basePoint = 
JSON.parse(JsonEmport.dataPointQuery("eq(xid,baseObservationPoint)")).dataPoints[0];
  basePoint.xid = chamberKey + "_obs";
  basePoint.deviceName = chamberKey;
  basePoint.pointLocator.varName = chamberKey;
  JsonEmport.doImport( JSON.stringify( {"dataPoints": [ basePoint ] } ));
}

//So there is one virtual point in the script's context: an alphanumeric with the variable name 'observation'
//As written here, observation would be set to a value like... C6_Tulip_228-Possible cold overexposure
if(observation.value !== "") {
  var obsData = observation.value.split("-"); //Left of hyphen is chamberKey, right is observation
  var chamberKey = obsData[0];
  if(typeof this[chamberKey] === 'undefined') { //create if need be
    createObservationPoint(chamberKey);
    LOG.info("Created observation point for " + chamberKey); //keep track of creating things at least a little
  }
  this[chamberKey].set(obsData[1]); //set the observation into the right point.
  observation.set("");
}

The downside there may be that people may make mistakes entering the identifier for the observation, which would just create a new point. But the upside is your observations would be visible through the regular mechanisms of points, such as the latest values table.

There are probably other ways to do it yet still. Edit: Ah yes Terry's suggestion is another good one. You can directly associate JSON objects to time series data. ("string" is valid JSON if you only wish to store a string).

Edit again: It appears I missed a paren and a .value. But here's the JSON for such a scripting data source:

{
   "dataSources":[
      {
         "xid":"DS_7868778e-0043-43a2-954e-2793f88adc5a",
         "name":"Observer",
         "enabled":true,
         "type":"SCRIPTING",
         "alarmLevels":{
            "SCRIPT_ERROR":"URGENT",
            "DATA_TYPE_ERROR":"URGENT",
            "POLL_ABORTED":"URGENT",
            "LOG_ERROR":"URGENT"
         },
         "purgeType":"YEARS",
         "updateEvent":"CONTEXT_UPDATE",
         "context":[
         ],
         "logLevel":"NONE",
         "cronPattern":"0 * * * * ? 2019",
         "executionDelaySeconds":0,
         "historicalSetting":false,
         "script":"function createObservationPoint(chamberKey) { \/\/This will create the new points for us\r\n  \/\/baseObservationPoint is the xid of an enabled, settable alphanumeric point on the scripting data source that runs this script.\r\n  var basePoint = \r\nJSON.parse(JsonEmport.dataPointQuery(\"eq(xid,baseObservationPoint)\")).dataPoints[0];\r\n  basePoint.xid = chamberKey + \"_obs\";\r\n  basePoint.deviceName = chamberKey;\r\n  basePoint.pointLocator.varName = chamberKey;\r\n  JsonEmport.doImport( JSON.stringify( {\"dataPoints\": [ basePoint ] } ));\r\n}\r\n\r\n\/\/So there is one virtual point in the script's context: an alphanumeric with the variable name 'observation'\r\n\/\/As written here, observation would be set to a value like... C6_Tulip_228-Possible cold overexposure\r\nif(observation.value !== \"\") {\r\n  var obsData = observation.value.split(\"-\"); \/\/Left of hyphen is chamberKey, right is observation\r\n  var chamberKey = obsData[0];\r\n  if(typeof this[chamberKey] === 'undefined') { \/\/create if need be\r\n    createObservationPoint(chamberKey);\r\n    LOG.info(\"Created observation point for \" + chamberKey); \/\/keep track of creating things at least a little\r\n  }\r\n  this[chamberKey].set(obsData[1]); \/\/set the observation into the right point.\r\n  observation.set(\"\");\r\n}",
         "scriptPermissions":{
            "customPermissions":"",
            "dataPointReadPermissions":"superadmin",
            "dataPointSetPermissions":"superadmin",
            "dataSourcePermissions":"superadmin"
         },
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"baseObservationPoint",
         "name":"Observation",
         "enabled":true,
         "loggingType":"ALL",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"INSTANT",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"ALPHANUMERIC",
            "contextUpdate":true,
            "settable":true,
            "varName":"observation"
         },
         "eventDetectors":[
         ],
         "plotType":"STEP",
         "rollup":"NONE",
         "unit":"",
         "templateXid":"Alphanumeric_Default",
         "chartColour":"",
         "chartRenderer":{
            "type":"TABLE",
            "limit":10
         },
         "dataSourceXid":"DS_7868778e-0043-43a2-954e-2793f88adc5a",
         "defaultCacheSize":1,
         "deviceName":"Observer",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":15,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "textRenderer":{
            "type":"PLAIN",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "suffix":""
         },
         "tolerance":0.0
      }
   ]
}

This script will not validate if the point was supposed to have been created, as imports are not run during validation. After points are created, they're reasonable to use to directly make observations to, with the hyphen and identifier being unnecessary then.

[chrapchp]

@phildunlap Thanks for the replies. Both Terry's and your response provide some food for thought. I will explore both options, yet I do like the one-stop approach result of using the scripts. The goal is to architect a solution that both control engineers and web-type developers can co-exist in the same sandbox and still allow a data analyst to extract information. That said, the data will be exported and manipulated with external python based tools so the JSON approach is viable as well.

peter

[Jared Wiltshire]

@chrapchp said in Temporal observational data on collected data:

We tried via virtual data sources and question if there is a better way as adding a tag by chamber seems cumbersome.

@chrapchp said in Temporal observational data on collected data:

At time t, add an observation e.g. for growing chamber X, nutrient tank Y, plant type Z, such as the characteristics of the plants, and other types of observations that could influence the growth.

I'm also going to chime in and suggest that a alpha-numeric data point might be good way to go. We have added data point tags in v3.3 so you will be able to tag the data points with the growingChamber=X, nutrientTank=Y and plantType=Z. You can then locate all the data points with a given set of tags.

We are hoping to have v3.3 released within the next few weeks, the data point tags will be usable via the REST API upon release but it might be another couple of weeks before the tagging functionality will be added to the UI.

[chrapchp]

Thanks for the info. We will look into this.

In the long run this may span buildings and many chambers so we would be looking at something that scales without too much data entry.

Peter

[thutchis]

@chrapchp said in Temporal observational data on collected data:

I'm also going to chime in and suggest that a alpha-numeric data point might be good way to go. We have added data point tags in v3.3 so you will be able to tag the data points with the growingChamber=X, nutrientTank=Y and plantType=Z. You can then locate all the data points with a given set of tags.

We are hoping to have v3.3 released within the next few weeks, the data point tags will be usable via the REST API upon release but it might be another couple of weeks before the tagging functionality will be added to the UI.

Jarad,
Where can I find more information on the data point tags? Any examples of using them with the REST API? I would like to tag my meta data and data points created in a batch process time span with a batch number tag for easy retrieval when creating a batch report.

[Jared Wiltshire]

@thutchis Turn on the Swagger API documentation by setting swagger.enabled=true in your overrides env.properties file. Check that swagger.mangoApiVersion=v[12] too.

You can then access the REST API documentation at /swagger/index.html

Just quickly though-

Get tags for a data point XID:
GET /rest/v2/data-point-tags/{xid}

Set tags for a data point XID:
POST /rest/v2/data-point-tags/{xid}
Body is a JSON object

{
  "tagKey": "tagValue",
  "site": "Denver"
}

To query for data points with a given set of tags:
GET /rest/v2/data-points?tags.site=Denver&tags.device=MachineX

Note there are psuedo-tags device and name which map to the data point's device name and name respectively.