how to handle missing datapoints

Phillip Weeks

I have a mango es collecting data and sometimes it has crashed. it polls a accumulator counter once a minute. there are now many data holes where there is no data values for time blocks. how can I fill in the missing datapoint values based on the average between the closest two point values using script.
I generate a metapoint on this counter to calculate the change from one minute to the next so I'm unclear what happens when generating meta history and it hits a null value for the p.past(MINUTE,5).maximumValue ?? .

phildunlap

Hi Phillip,

Did you seek the cause of the crash? Presumably there will be a meaningful log message in Mango/logs/ma.log (maybe Mango/bin/logs/ma.log on Windows).

For a meta point, you can just set the history generation period prior to the shutdown and regenerate. If you are not using the NoSQL module, you will probably want to delete the values after the history generation period start before generating, otherwise you will have multiple values in the database.

If this is a different kind of point, you can probably still fill in the data gap using a script's 'set historical values' setting and some clever coding.

I would expect if you have a meta point and its body is "return null;" I would expect Binary to get false, Numeric to get 0.0, Multistate to get 0, and Alphanumeric to get null as the value when the script runs. So, if there is no data in your period, p.past(MINUTE,5).maximumValue should be null and you would get a zero.

Phillip Weeks

Oh yes indeed JAVA runs out of memory malloc fails
I'm beyond this now lol as we are publishing the data to a cloud server with 4gb for metapoint calculations and this has solved the crash issues. in total we are calculating about 1100 metapoints on data and this is a small 72 unit test with a 300+ unit test looming.

Anyway I would like to run a script back over the modbus points and insert timestamps and a number that would be the difference between the oldest point value before the null the oldest after the null and divide this by the number of timestamp intervals between these points and then insert that value and call the same script again to find the next null.

Am I thinking about this correctly?
Can you offer an example that would accomplish this setting TIMESTAMP etc.
anything would be helpful as javascript is new to me.
Thanks in advance.

phildunlap

Hmm. I don't suppose you've set any heap parameters using the Mango/bin/ext-available/ scripts?

There is a brief discussion in this thread about using the last function to make meta points looking for specific statistics: https://infiniteautomation.com/forum/topic/2483/high-cpu-usage/18

For your example, likely the most effecient way to get the maximum value would be a script body of return my.value; and a logging type of 5 minute interval maximum.

To the question of filling historical values, there are multiple ways. If you export a CSV from the data point details' chart, this format can be used to import values through the data import module. Or, you could have a scripting data source with "Saves historic" checked (this setting bypasses generating events and logging settings to save specific values) with a body like,

function interpolate(valueTime1, valueTime2, time) {
  //You can implement this other ways, but this is simple linear I believe
  return valueTime1.value + (valueTime2.value - valueTime1.value) * (time - valueTime1.time) / (valueTime2.time - valueTime1.time);
}

var minimumDataInterval = 5000; //5 seconds
var periodBegin = new Date(2017, 4, 12).getTime(); //You probably wish to set these ranges
var periodEnd = new Date(2017, 4, 13).getTime();
var values = p.pointValuesBetween( periodBegin, periodEnd );
for(var k = 0; k < values.length-1; k+=1) {
  var val1 = values[k];
  var val2 = values[k+1];
  if( val2.time - val1.time > minimumDataInterval ) {
    var newDataTime = val1.time + minimumDataInterval;
    while( newDataTime < val2.time ) {
      p.set( interpolate(val1, val2, newDatatime), newDataTime );
      newDataTime += minimumDataInterval;
  }
}

For this script, the second time it runs shouldn't do anything, so there's a couple options. You can set it to a cron like "0 * * * * ?" and disable it after the first new minute ticks by. You can create a settable binary point on the scripting data source and have some if( runScript.value ) { ...; runScript.set(false);} wrapping the work in the script (then you'll have to set that point true, and have its variableName be runScript).

One could even get crazier if they wanted and have this work for an arbitrary point, by having a data point on the scripting data source that was alphanumeric, accepted let's say a comma delineated list of XIDs to generate history over, and have some larger loop like....

if( xidList.value != null && xidList.value.length > 0 ) {
  var xids = xidList.value.split(",");
  for(var i = 0; i < xids.length; i+=1) {
    var dps = DataPointQuery.query("eq(xid, "+xids[ i ]+")"); //apparently * didn't render as [ i ]
    if(dps.length < 1 || dps[0].runtime == null)
      continue;
    p = dps[0].runtime;
    //The other part of the script referenced p
    /* code from above */
  }
  xidList.set(""); //note that if the point was disabled we did nothing yet this still clears
   //but hey it's just an example!
}

Phillip Weeks

Thanks so much Phil, I really appreciate that and this was exactly what I needed for a much better understanding on this . Yes I've been using a script point latch in running code on cron so I'll pop fit it in there. ya gotta love recursion :)
I've got 72 points to clean up. So if I insert these points into the published data on our cloud server, is there a way for those inserts to propagate back to the ES?

I have one more question about metapoints .. I have a source with 311 Meta points and they were all created through CSV import and excel and all imported fine... oddly some of them will not take a value and continue to state "active source point has no value" I can find no reason for how only some of these points to have no value and yet almost all the others are populating every minute from their scripts? I run a history on these points as it comes back 0 records. I have destroyed and recreated and copied a working point and just changed the context variable to the correct one. No difference and yet the context variable does have a history.

Any ideas on what causes this situation?
Thanks in advance.

phildunlap

Values set to your cloud server will not propagate backward from receiver to publisher. You could insert the values on the ES, but you'd have to reset the history synchronization time on the publisher to prior to the period you're filling in values, then do a big history sync. If you set them on the cloud instance and the new values are in the period that still has to history sync, this wouldn't work as intended, as having more values (or unmatched values) in the receiver than the publisher for a time range will cause synchronization issues. You could do an export of the CSV from the cloud and import that at the ES, start at the ES, or decide the ES doesn't actually need that interpolation.
The "active source point has no value" messages are not from your Meta points. Why do you think they are? The code appears to suggest only a Set Point event handler will generate that message.

phildunlap

Okay I understand. I didn't realize that was the same text for mouse overs on the data point icon in the data point table on the data source page. What's the JSON for one such nonfunctioning point?

Phillip Weeks

I recreated the point from scratch and it started working. I do not know what the difference could have been. all other working points in this DS had the same import template curious? must be something.

phildunlap

"must be something" if you really think so you can provide me the template's JSON and the JSON for the non-functioning meta point and I'll try to see

Phillip Weeks

Great suggestion............ I compared the two jsons and every 11th point had the updates context set off ... Thanks for the great tips..

Phillip Weeks

Phil, I tried the code you provided and this part of the script returns the values array with 0 length {} even though I see hundreds of pointvalues on this context point between these dates. Anything obvious?

.var periodBegin = new Date(2017, 4, 1).getTime();
var periodEnd = new Date(2017, 4, 14).getTime();
var values = p.pointValuesBetween( periodBegin, periodEnd );

This is a printout of the result:

0
{
value: 228.5,
time: 1492146208215,
millis: 215,
second: 28,
minute: 3,
hour: 1,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(228.5@2017/04/14 01:03:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

This is the context point stats for the past 15 days:
Statistics (as of 2017/04/14 01:59:14) Time period: 15
Get statistics
Start: 2017/03/30 23:00
End: 2017/04/14 01:59
Minimum: 23.00 L @ Mar 30 23:00
Maximum: 228.50 L @ Apr 13 14:05
Average: 107.36 L
Integral: 131019642.6 L·s
Sum: 1470286.00 L
Log entries: 15264

phildunlap

It's probably because I spaced on months being indexed to 0, so 4 is May instead of April. Day of the month is indexed to 1, though, so I guess you're looking for new Date(2017, 3, 1) and new Date(2017, 3, 14)

Phillip Weeks

OK yes that was it
Great I have it inserting 5000 points for one point history now Id like to get this working..
For(i=1;i<73;i++){....
eval('var values = p'+i+'.pointValuesBetween( periodBegin, periodEnd )');

eval('p'+i+'.set((val1.value + (val2.value - val1.value) * (newDataTime - val1.time) / (val2.time - val1.time)), newDataTime )');

It complains of error
Any ideas..

phildunlap

Capital F in for?

I guess you're in the state where the script doesn't output the error message? Sometimes clearing your cache and refreshing can relieve that.

Phillip Weeks

Hi Phil .. almost done with this issue.. using var x = eval('p'+i) I can assign x = p1 thru p9 but for some reason it won't assign p10 or higher to x generates an error Any ideas why?

if (!INTERPOLATE_FLAG.value===true){
var DatapointsAdded = 0;
var LongestSpan = 0;
var minimumDataInterval = 60000; //60 seconds
var periodBegin = new Date(2017, 2, 1).getTime(); //You probably wish to set these ranges
var periodEnd = new Date(2017, 3, 14).getTime();

INTERPOLATE_FLAG.set(false,periodEnd);

for (var i = 1; i<9; i++){
var x = eval('p'+i);
print(x);
var values = x.pointValuesSince(periodBegin);
for(var k = 0; k < values.length-1; k+=1) {
var CurrentSpan =0;
var val1 = values[k];
var val2 = values[k+1];
if( val2.time - val1.time > minimumDataInterval ) {
var newDataTime = val1.time + minimumDataInterval;
while( newDataTime < val2.time ) {
x.set((val1.value + (val2.value - val1.value) * (newDataTime - val1.time) / (val2.time - val1.time)), newDataTime );
DatapointsAdded += 1;
CurrentSpan += 1;
if (CurrentSpan > LongestSpan){
LongestSpan = CurrentSpan;
}
}
}
}
}

Datapoints_Added.set(DatapointsAdded,periodEnd);
Longest_Span.set(LongestSpan,periodEnd);

}
RESULTS:

Setting point START INTERPOLATION to false @14/04/2017 00:00:00
{
value: 228.5,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(228.5@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 48.0,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(48.0@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 1401.0,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(1401.0@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 821.5,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(821.5@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 836.5,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(836.5@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 250.5,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(250.5@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 1005.5,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(1005.5@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

{
value: 0.0,
time: 1492189828215,
millis: 215,
second: 28,
minute: 10,
hour: 13,
day: 14,
dayOfWeek: 6,
dayOfYear: 104,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(0.0@2017/04/14 13:10:28.215),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

Setting point NEW_DATAPOINTS_ADDED to 40841.0 @14/04/2017 00:00:00
Setting point Longest Datapoint Gap to 2232.0 @14/04/2017 00:00:00

phildunlap

Somewhere between anonymous functions and an eval() function there is plenty of room to make JavaScript confusing. I would avoid the eval function, personally, as its cleverness is often not worth its extra complexity or risk. But, that's up to you.

The likelihood is, if it has the problem on #10, that p10 is not a valid variable name or p10 is disabled.

phildunlap

Hi Phillip,

I just put Scripting version 1.2.1 into the store. It has a fix for the reason the error messages may not be printing beneath the script box. Could be helpful!

Phillip Weeks

Thanks Phil I'll try it out.

phildunlap

Hi Phillip,

It looks like I made some mistakes when writing the interpolate script. From testing, here's an updated version of the base functionality...

function interpolate(valueTime1, valueTime2, time) {
  //You can implement this other ways, but this is simple linear I believe
  return parseFloat(valueTime1.value) + ((valueTime2.value - valueTime1.value) * (time - valueTime1.time)) / (valueTime2.time - valueTime1.time);
}

var minimumDataInterval = 5000; //5 seconds
var periodBegin = new Date(2017, 3, 12).getTime(); //You probably wish to set these ranges
var periodEnd = new Date(2017, 3, 13).getTime();
var values = p.pointValuesBetween( periodBegin, periodEnd );
for(var k = 0; k < values.length-1; k+=1) {
  var val1 = values[k];
  var val2 = values[k+1];
  if( val2.time - val1.time > minimumDataInterval ) {
    var newDataTime = val1.time + minimumDataInterval;
    while( newDataTime < val2.time ) {
      p.set( interpolate(val1, val2, newDataTime), newDataTime );
      newDataTime += minimumDataInterval;
  }
}
}

Here's the setup from my test:

{
   "dataSources":[
      {
         "xid":"DS_3b6b1d05-2aad-456f-895b-fe891ebc10a7",
         "name":"Scripting-BaseData",
         "enabled":false,
         "type":"SCRIPTING",
         "alarmLevels":{
            "SCRIPT_ERROR":"URGENT",
            "DATA_TYPE_ERROR":"URGENT",
            "POLL_ABORTED":"URGENT",
            "LOG_ERROR":"URGENT"
         },
         "purgeType":"YEARS",
         "context":[
            {
               "varName":"p1",
               "dataPointXid":"DP_ae686073-a60f-481e-9d25-3ddc3e78ef88"
            }
         ],
         "logLevel":"NONE",
         "cronPattern":"0\/1 * * * * ?",
         "executionDelaySeconds":0,
         "historicalSetting":false,
         "script":"if(!p1.value) {\n    p1.set(0, 0);\n    p1.set(1000, 1000);\n}",
         "scriptPermissions":{
            "customPermissions":"",
            "dataPointReadPermissions":"superadmin",
            "dataPointSetPermissions":"superadmin",
            "dataSourcePermissions":"superadmin"
         },
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      },
      {
         "xid":"DS_af7afc6b-50b5-4c33-84e0-f9297514df28",
         "name":"Scripting-Interpolate",
         "enabled":false,
         "type":"SCRIPTING",
         "alarmLevels":{
            "SCRIPT_ERROR":"URGENT",
            "DATA_TYPE_ERROR":"URGENT",
            "POLL_ABORTED":"URGENT",
            "LOG_ERROR":"URGENT"
         },
         "purgeType":"YEARS",
         "context":[
            {
               "varName":"p",
               "dataPointXid":"DP_ae686073-a60f-481e-9d25-3ddc3e78ef88"
            }
         ],
         "logLevel":"NONE",
         "cronPattern":"0\/1 * * * * ?",
         "executionDelaySeconds":0,
         "historicalSetting":true,
         "script":"function interpolate(valueTime1, valueTime2, time) {\n  \/\/You can implement this other ways, but this is simple linear I believe\n  return parseFloat(valueTime1.value) + ((valueTime2.value - valueTime1.value) * (time - valueTime1.time)) \/ (valueTime2.time - valueTime1.time);\n}\nif(p.last(100).length < 5) {\nvar minimumDataInterval = 2; \/\/5 seconds\nvar periodBegin = 0; \/\/You probably wish to set these ranges\nvar periodEnd = 1001;\nvar values = p.pointValuesBetween( periodBegin, periodEnd );\nfor(var k = 0; k < values.length-1; k+=1) {\n  var val1 = values[k];\n  var val2 = values[k+1];\n  if( val2.time - val1.time > minimumDataInterval ) {\n    var newDataTime = val1.time + minimumDataInterval;\n    while( newDataTime < val2.time ) {\n      p.set( interpolate(val1, val2, newDataTime), newDataTime );\n      newDataTime += minimumDataInterval;\n  }\n}\n}}",
         "scriptPermissions":{
            "customPermissions":"",
            "dataPointReadPermissions":"superadmin",
            "dataPointSetPermissions":"superadmin",
            "dataSourcePermissions":"superadmin"
         },
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      },
      {
         "xid":"DS_d68e5a18-5457-4fe3-a76b-3fc4c1c7120b",
         "name":"vrtds",
         "enabled":true,
         "type":"VIRTUAL",
         "alarmLevels":{
            "POLL_ABORTED":"URGENT"
         },
         "purgeType":"YEARS",
         "updatePeriodType":"MINUTES",
         "polling":false,
         "updatePeriods":5,
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"DP_ae686073-a60f-481e-9d25-3ddc3e78ef88",
         "name":"test",
         "enabled":true,
         "loggingType":"ALL",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"AVERAGE",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"NUMERIC",
            "changeType":{
               "type":"NO_CHANGE",
               "startValue":"0"
            },
            "settable":true
         },
         "eventDetectors":[
         ],
         "plotType":"SPLINE",
         "rollup":"NONE",
         "unit":"",
         "chartColour":"",
         "chartRenderer":{
            "type":"IMAGE",
            "timePeriodType":"DAYS",
            "numberOfPeriods":1
         },
         "dataSourceXid":"DS_d68e5a18-5457-4fe3-a76b-3fc4c1c7120b",
         "defaultCacheSize":1,
         "deviceName":"vrtds",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":1,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "textRenderer":{
            "type":"ANALOG",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "format":"0.00"
         },
         "tolerance":0.0
      }
   ]
}

Phillip Weeks

Thanks again Phil. That helped :)