Import EnergyData from XML

mircsicz

Hi Philip,

thx a ton for looking into my case and replying...

I've copied your RMCU.java to web/modules/dataFile/web/CompilingGrounds/XML and I've added the mentioned

import com.infiniteautomation.datafilesource.dataimage.ImportPoint;
import com.infiniteautomation.datafilesource.dataimage.NumericImportPoint;
import com.infiniteautomation.datafilesource.contexts.AbstractXMLDataSource;
//We'll also use...
import java.text.SimpleDateFormat;
import java.text.ParseException;
import java.util.Map;
import java.util.HashMap;

line's to my TP and VALUES.java files, besides that I've also removed the

package generated;

line. And I successfully compiled that new import Template!!! 8-)

Next thing I did was throwing 40k XML's in the directory:
alt text

So where can I now monitor the progress? I already have a multitail open monitoring ma.log, but that only stated:

INFO  2018-03-06T16:56:49,712 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:407) - Data source 'WSE_KA-DB' initialized
INFO  2018-03-06T16:56:49,713 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:412) - Data source 'WSE_KA-DB' took 3.304719ms to start

Edit:
just got an urgent Event:

'WSE_KA-DB': JAXBException: 1 counts of IllegalAnnotationExceptions

But I don't see it in the log and the server still seems busy with a load between 0.7 and 1...

How do I look under the hood? I still get those messages and no new data points show up

6.3.2018 17:15:26	device_hub  'WSE_KA-DB': Event from import class: java.lang.NullPointerException	No RTN	N/A	
6.3.2018 17:15:26	device_hub  'WSE_KA-DB': JAXBException: 1 counts of IllegalAnnotationExceptions	No RTN	N/A

Seems not to get it done:

WARN  2018-03-06T17:30:30,753 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source WSE_KA-DB aborted 1 polls since it started.

phildunlap

Hi mircsicz,

Did you modify the class before getting the IllegalAnnotationException? I had to work through one of those when I had added the simple date format as a class member variable. You shouldn't declare any additional variables in the automatically generated class, try to keep those variables all within the getParsedPoints method. If you use the API instead of poll to try importing the file, you can monitor status at the temporary resource location it provides you. Through the UI, you should be redirected to a page you could refresh by clicking...

Other than that, printing or throwing exceptions from your implementing class is the easiest way to get debug information. If you throw new java.lang.RuntimeException("my message"); you'll get a stack trace to that part of your code in the log and in an event, or you can simply System.out.println() as I did, but you would need to be watching stdout

mircsicz

Thank you Philip,

but currently I'm locked out. I changed my passwd and probably had twice the same typo, so I can't relogin. And sadly I can't reach the H2 WebConsole as of this error

mircsicz

Hi Philip,

I've uploaded using the API and clicked "Submit, get monitor":

{"resourceId":"DF_672777d1-dc31-4ec8-bb0f-091d52697197","progress":0,"finished":true,"cancelled":false,"errors":["Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions"],"unfoundIdentifiers":[],"createdPoints":[],"failedPoints":[],"totalImported":0,"priority":2,"queueSize":0,"taskId":"TR_DF_672777d1-dc31-4ec8-bb0f-091d52697197","threadName":"Temporary Resource Timeout for : DF_672777d1-dc31-4ec8-bb0f-091d52697197","expires":1520368586474}

phildunlap

Refresh that page, you should get some status information

mircsicz

@phildunlap said in Import EnergyData from XML:

Refresh that page, you should get some status information

That's all I get... :-(

phildunlap

@mircsicz said in Import EnergyData from XML:

Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions

Oh, yeah, didn't look closely. You have the same error: Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions

Did you see my note about adding member variables? Are you using the class I provided? I did test it using the XML you provided....

mircsicz

Yes I saw your note and all I added to the other two *.java files was the import part's you mentioned...

And yes copy/pasted the provided class ;-)

phildunlap

You can supply XML if you think something went awry with your sample data, but the snippet you posted I have imported successfully again just now, using a copy-paste of the class I supplied as RMCU.java and the TP.java / VALUES.java files that were generated (but I removed package generated from them, you didn't need to modify TP and VALUES any other way). The illegal annotation suggests to me something in that process didn't go right.

mircsicz

Argghhh, I've removed the additional "import" line's from TP and VALUES.java but I still get no positive feedback:

{"resourceId":"DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","progress":0,"finished":true,"cancelled":false,"errors":["Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions"],"unfoundIdentifiers":[],"createdPoints":[],"failedPoints":[],"totalImported":0,"priority":2,"queueSize":0,"taskId":"TR_DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","threadName":"Temporary Resource Timeout for : DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","expires":1520371072215}```

mircsicz

THX to phil's support it's now importing and creating data point's

But it's so far not the correct data point seperation!

It needs to be seperated by TPID's

EDIT: Sorry was in a rush as my son woke up while I was typing...

The structur of the XML is as follow's:

RMCU
   TP ID=NAME-OF-METERING-POINT
      VALUES
        DATETIME
         EE (Electric Energy)
         TMP (Temperatur)
         PO (Power)
         HQ (Heat Quantity)
      VALUES
   TP
ID
 NAME-OF-DATALOGGER
ID
SERIAL
  SERIAL-OF-DATALOGGER
SERIAL
IP
   IP-OF-DATALOGGER
IP
RMCU

Each TP-ID's VALUE tagged combined with DATETIME needs to become it's own Data Point

phildunlap

Glad you got it working. You probably want to get those TPIDs into the identifier for the NumericImportPoint (the first argument in creating these objects).

mircsicz

Hi Philip,

yeap that's exactly what I want! But I can't realize the change, I guess I've to change "This EE point xid" in

if(value.getEE() != null)    
              result.add(new NumericImportPoint("This EE point xid", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));

But as I don't have any java experience I simply don't know how, So if you could phrase it for me once I'ld hopefully be done and gone... ;-)

EDIT: I just saw that the Data Point is literally named: "This EE point xid" so if you'ld be so kind to tell me how to parse the string behind the "tp" var into that place that would probably be it. It expect the DataPoint name to be something like:

GHV1 3P6 5OG_EE
Case-Temp_TMP
or in other cases:
GHV1 3P6 5OG_PO
etc

phildunlap

Correct part of the code. I'm guessing you want the ID from the TP element and the EE/TMP nature of the value. Something like,

if(value.getEE() != null)
  result.add(new NumericImportPoint(tp.getID() + " EE value", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));
if(value.getTMP() != null)
  result.add(new NumericImportPoint(tp.getID() + " TMP value", value.getTMP().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));

phildunlap

Seeing you comment on the schema two posts ago, you also will need to add conditions for if(value.getPO() != null) and probably HQ as well.

mircsicz

Yeap Phil, I'm aware of that. But with a working basic example like this one I can abstract the rest...

THX a ton Philip I reran the import on al 50k XML files and I now have the expected data point's.... I'll now start to adopt for other value type's!!!