How To Copy point data to a scripting point

Phillip Weeks

I am trying to copy all points since a specific date into a scripting point with the code below...
I don't understand what is happening at all.. sometimes the script runs and produces 100 points. I purge the points and rerun the script and it produces 6 point values, purge again and run again and it creates 1 point value.
The script always says 7684 points were set according to the result printout.. But when it runs only one point was created or sometimes 10 points or another number of points? Never creates the 7684 number it says??
I have no clue what determines this outcome and any help understanding this would be appreciated.

Run Script is : true
Points Found: 7684
Setting point PointCopy to 541.0 @13/04/2017 00:00:28
Setting point PointCopy to 541.0 @13/04/2017 00:01:28
Setting point PointCopy to 541.0 @13/04/2017 00:02:28
:
Setting point PointCopy to 636.5 @18/04/2017 07:55:28
Setting point PointCopy to 636.5 @18/04/2017 07:56:28
Setting point Run Script #2 Flag1 to false @18/04/2017 11:02:25
Points Copied = 7683
Points fixed = 12
{
value: 636.5,
time: 1492522230001,
millis: 1,
second: 30,
minute: 0,
hour: 11,
day: 18,
dayOfWeek: 3,
dayOfYear: 108,
month: 4,
year: 2017,
last(count): PointValueTime[count],
lastValue: PointValueTime(636.5@2017/04/18 11:00:30.001),
lastValue(count): PointValueTime,
set(value): ,
set(value, timestamp): ,
pointValuesBetween(timestamp, timestamp): PointValueTime[],
pointValuesSince(timestamp): PointValueTime[],
pointValuesBefore(timestamp): PointValueTime[],
pointValuesAfter(timestamp): PointValueTime[],
pointValueAt(timestamp): PointValueTime,
ago(periodType): double,
ago(periodType, periods): double,
past(periodType): AnalogStatisticsWrapper,
past(periodType, periods): AnalogStatisticsWrapper,
prev(periodType): AnalogStatisticsWrapper,
prev(periodType, periods): AnalogStatisticsWrapper,
previous(periodType): AnalogStatisticsWrapper,
previous(periodType, periods): AnalogStatisticsWrapper,
stats(from, to): AnalogStatisticsWrapper,
}

THIS IS THE SCRIPT IN QUESTION....

print("Run Script is : " +RUNSCRIPT2_FLAG1.value);
var periodBegin = new Date(2017, 3, 13).getTime()
var copied = 0; fixed=0;

if (RUNSCRIPT2_FLAG1.value === true){
var values = pointx.pointValuesSince(periodBegin);
print ("Points Found: " + values.length);
for(var k = 0; k < values.length-1; k+=1) {
var val1 = values[k];
var lastval = val1.value;

        if (val1.value >= 1){
            lastval = val1.value;
            CopiedPoint.set(lastval,val1.time);
             copied +=1;
        } 
        else 
        {
            CopiedPoint.set(lastval,val1.time);
            copied +=1;
            fixed +=1;
        }
  }     
     
 
  }

RUNSCRIPT2_FLAG1.set(false);
print("Points Copied = " + copied);
print("Points fixed = " + fixed);
print(CopiedPoint);

phildunlap

Hi Phillip,

Does this scripting data source have "Saves historic" checked? (I guess yes)
How are you assessing how many points were created? The number of values PointCopy has? How long did you wait after running the script?
Is your script difficult to disable once enabled? (I guess no)

My intuition suggests you didn't wait long enough after the script ran for the values to appear, but you could enable logging in the script and put in LOG.info("Copied: " + copied " + " point values."); to get log messages into Mango/logs/scripting-[data source id].log

Values that are not new values (latest timestamps in the series) can take a minute or two (actually configurable) if you're using the NoSQL module to appear, as it is more efficient to wait for the backdates to accumulate than insert them one at a time. That could be what you're seeing? It could also be the logging type of the CopiedPoint is not "All data" and it's something to do with that.

Phillip Weeks

Phil; I have the script copying now and writing to a log file, perhaps I was not waiting long enough. Yes, I control the script running with flags as you noticed, so pretty easy to enable and disable them.
I am still trying to validate the interpolate script you helped me with as working correctly on this copied point before I implement it on the original modbus data.
In testing the script I get duplicates entries for the same timestamp. Is this normal because I thought a point.set with a timestamp that existed would overwrite that value. how do I prevent duplicates?

Also other than the sparse documentation is there a complete and detailed documentation on the objects, properties and methods exposed? I seem to be learning this data paradigm piece-meal. Thanks in advance.

phildunlap

Do you have the mangoNoSqlDatabase module installed?

This article is a useful introduction to Mango's JavaScript: http://help.infiniteautomation.com/support/solutions/articles/14000022520-about-mango-java-script

It's a reproduction of the contextual help in the related items for most things which have scripts (click blue ? on data source edit page, scroll to the bottom, click "Mango Java Script")

Phillip Weeks

@phildunlap said in How To Copy point data to a scripting point:

mangoNoSqlDatabase
Do we require the mangoNoSqlDatabase module do to solve this duplicate issue?
I googled mangoNoSqlDatabase and get no results at all related to mango. NOTHING.
Even on the MANGO help site it returns virtually nothing about mangoNoSqlDatabase .
AND I have read those pages you directed me to, and nothing on duplicate records.

phildunlap

Hi Phillip

I foresaw the duplicate values if you are not using NoSQL in the original thread: http://infiniteautomation.com/forum/topic/2724/how-to-handle-missing-datapoints/3

To check if you have the module installed, you would navigate to the /modules.shtm page in the browser and see if it's in the list.

Phillip Weeks

no its not in the list :(
I'm guessing it is required to prevent duplicates?

phildunlap

Yes, you cannot have two values at the same millisecond using the NoSQL module, the latest insert prevails.

phildunlap

You may want to use the Data Import module to delete the duplicates. You can export the point values on the Data Point Details page and mark rows to be removed, then import on /dataImport.shtm

Phillip Weeks

Even without this module I am wondering why it duplicates points because we only set points with a timestamp + interval where there was no timestamp within the interval of the current point so this should not create duplicates should it?

phildunlap

I can't say. I did some testing and while I have to correct some other errors in what I provided you it didn't seem to have any issues that would have caused duplication points. There was an issue with the variable name passed to the interpolation function, though. The updated version of that script is in the other thread.