Data File importer duplicate values

[psysak]

Is it possible to have the data file importer skip importing a value if one already exists? I am pulling a csv off a website which is "year to now". If I pull it once a day I get duplicates for each value. Is that something I would have to code in the importer so that it checks to see if that value exists?

[MattFox]

Well at least if you did that you'd have more control over filtering the data. Set up a list with the timestamp as the key. At least you'd be able to set the value for that particular time. Then iterate and insert.

That's how I'd implement it at least...

Fox

[phildunlap]

Hi psysak,

I think having the importer check to see if the value exists is a possible and not incorrect direction. You should be able to import whatever classes you like and use them however you like. This may be the right time to use a com.serotonin.m2m2.rt.dataImage.PointValueFacade: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/serotonin/m2m2/rt/dataImage/PointValueFacade.java

---

There would be a hacky way of getting that information more available to the CSV importer somewhat simply by adding a point to the data file data source in question, and then updating that point's "Identifier in import document" through a meta point triggered by the new data imported.

var timePoint = JSON.parse(JsonEmport.dataPointQuery("eq(xid,DP_time_informer)"))["dataPoints"][0]
timePoint.pointLocator.fromIdentifier = String( targetPoint.time );
JsonEmport.doImport( JSON.stringify( { "dataPoints": [timePoint] } ));

Then changing the importer with something like...

private long lastImportTime;

@Override
public boolean takesPointIdentifiers() {
  return true;
}

@Override
public void setIdentifiers(String[] identifiers) {
  //super.setIdentifier(identifiers) if you want this.identifiers = identifiers
  for(String s : identifiers)
    if( !"".equals(s) ) { //Presuming all other identifier fields are empty
       lastImportTime = Long.valueOf(s);
       break;
    }
}

---

It could also be that the best solution is to modify the data file importer to import a digested data set into an alphanumeric point. Say, pick some format to encode the last ten days into an alphanumeric point, and then use a script to digest it farther and check if the value for that day / time already existed.

var valuesList = alphanum.value.split("~"); //5@15~10@20
//iterate valuesList, split the time from the value and set to the end point
for(var k = 0; k < valuesList.length; k+=1) {
  var valuePair = valuesList[k].split("@");
  output.set( Number(valuePair[0]), Number(valuePair[1]) );
}

---

I didn't test any of that, but it should work....

Edit: Also, if using NoSQL this shouldn't be an issue. The date should always resolve to the same millisecond and it should overwrite the previous value at that millisecond.