• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Data File importer duplicate values

    Scheduled Pinned Locked Moved User help
    3 Posts 3 Posters 1.0k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      psysak
      last edited by

      Is it possible to have the data file importer skip importing a value if one already exists? I am pulling a csv off a website which is "year to now". If I pull it once a day I get duplicates for each value. Is that something I would have to code in the importer so that it checks to see if that value exists?

      1 Reply Last reply Reply Quote 0
      • MattFoxM Offline
        MattFox
        last edited by

        Well at least if you did that you'd have more control over filtering the data. Set up a list with the timestamp as the key. At least you'd be able to set the value for that particular time. Then iterate and insert.

        That's how I'd implement it at least...

        Fox

        Do not follow where the path may lead; go instead where there is no path.
        And leave a trail - Muriel Strode

        1 Reply Last reply Reply Quote 0
        • phildunlapP Offline
          phildunlap
          last edited by phildunlap

          Hi psysak,

          I think having the importer check to see if the value exists is a possible and not incorrect direction. You should be able to import whatever classes you like and use them however you like. This may be the right time to use a com.serotonin.m2m2.rt.dataImage.PointValueFacade: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/serotonin/m2m2/rt/dataImage/PointValueFacade.java


          There would be a hacky way of getting that information more available to the CSV importer somewhat simply by adding a point to the data file data source in question, and then updating that point's "Identifier in import document" through a meta point triggered by the new data imported.

          var timePoint = JSON.parse(JsonEmport.dataPointQuery("eq(xid,DP_time_informer)"))["dataPoints"][0]
          timePoint.pointLocator.fromIdentifier = String( targetPoint.time );
          JsonEmport.doImport( JSON.stringify( { "dataPoints": [timePoint] } ));
          

          Then changing the importer with something like...

          private long lastImportTime;
          
          @Override
          public boolean takesPointIdentifiers() {
            return true;
          }
          
          @Override
          public void setIdentifiers(String[] identifiers) {
            //super.setIdentifier(identifiers) if you want this.identifiers = identifiers
            for(String s : identifiers)
              if( !"".equals(s) ) { //Presuming all other identifier fields are empty
                 lastImportTime = Long.valueOf(s);
                 break;
              }
          }
          

          It could also be that the best solution is to modify the data file importer to import a digested data set into an alphanumeric point. Say, pick some format to encode the last ten days into an alphanumeric point, and then use a script to digest it farther and check if the value for that day / time already existed.

          var valuesList = alphanum.value.split("~"); //5@15~10@20
          //iterate valuesList, split the time from the value and set to the end point
          for(var k = 0; k < valuesList.length; k+=1) {
            var valuePair = valuesList[k].split("@");
            output.set( Number(valuePair[0]), Number(valuePair[1]) );
          }
          

          I didn't test any of that, but it should work....

          Edit: Also, if using NoSQL this shouldn't be an issue. The date should always resolve to the same millisecond and it should overwrite the previous value at that millisecond.

          1 Reply Last reply Reply Quote 0
          • First post
            Last post