Import EnergyData from XML

mircsicz

I've found some documentation inside the "data source" type "data file"... I'll see how far I can go with that and report back...

I'm stuck at the expectation to include

public abstract List<ImportPoint> getParsedPoints();

into the class file, I've already added “implements AbstractXMLDataSource” to what looked for me like the main class definition. But as I'm not a java programmer and more a network-ops I'm not even sure about that...

//
// Diese Datei wurde mit der JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 generiert
// Siehe <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a>
// Änderungen an dieser Datei gehen bei einer Neukompilierung des Quellschemas verloren.
// Generiert: 2018.03.04 um 09:28:57 PM CET
//


package generated;

import java.math.BigDecimal;
import java.math.BigInteger;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.annotation.XmlElementDecl;
import javax.xml.bind.annotation.XmlRegistry;
import javax.xml.bind.annotation.adapters.CollapsedStringAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
import javax.xml.namespace.QName;


/**
 * This object contains factory methods for each
 * Java content interface and Java element interface
 * generated in the generated package.
 * <p>An ObjectFactory allows you to programatically
 * construct new instances of the Java representation
 * for XML content. The Java representation of XML
 * content can consist of schema derived interfaces
 * and classes representing the binding of schema
 * type definitions, element declarations and model
 * groups.  Factory methods for each of these are
 * provided in this class.
 *
 */
@XmlRegistry
public class ObjectFactory {

    implements AbstractXMLDataSource;

    private final static QName _EE_QNAME = new QName("", "EE");
    private final static QName _STATUS_QNAME = new QName("", "STATUS");
    private final static QName _DATETIME_QNAME = new QName("", "DATETIME");
    private final static QName _IP_QNAME = new QName("", "IP");
    private final static QName _ID_QNAME = new QName("", "ID");
    private final static QName _SN_QNAME = new QName("", "SN");
    private final static QName _PO_QNAME = new QName("", "PO");
    private final static QName _TMP_QNAME = new QName("", "TMP");

    /**
     * Create a new ObjectFactory that can be used to create new instances of schema derived classes for package: generated
     *
     */
    public ObjectFactory() {
    }

    /**
     * Create an instance of {@link VALUES }
     *
     */
    public VALUES createVALUES() {
        return new VALUES();
    }

    /**
     * Create an instance of {@link RMCU }
     *
     */
    public RMCU createRMCU() {
        return new RMCU();
    }

    /**
     * Create an instance of {@link TP }
     *
     */
    public TP createTP() {
        return new TP();
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link BigDecimal }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "EE")
    public JAXBElement<BigDecimal> createEE(BigDecimal value) {
        return new JAXBElement<BigDecimal>(_EE_QNAME, BigDecimal.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link BigInteger }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "STATUS")
    public JAXBElement<BigInteger> createSTATUS(BigInteger value) {
        return new JAXBElement<BigInteger>(_STATUS_QNAME, BigInteger.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link String }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "DATETIME")
    @XmlJavaTypeAdapter(CollapsedStringAdapter.class)
    public JAXBElement<String> createDATETIME(String value) {
        return new JAXBElement<String>(_DATETIME_QNAME, String.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link String }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "IP")
    @XmlJavaTypeAdapter(CollapsedStringAdapter.class)
    public JAXBElement<String> createIP(String value) {
        return new JAXBElement<String>(_IP_QNAME, String.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link String }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "ID")
    @XmlJavaTypeAdapter(CollapsedStringAdapter.class)
    public JAXBElement<String> createID(String value) {
        return new JAXBElement<String>(_ID_QNAME, String.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link BigInteger }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "SN")
    public JAXBElement<BigInteger> createSN(BigInteger value) {
        return new JAXBElement<BigInteger>(_SN_QNAME, BigInteger.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link BigDecimal }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "PO")
    public JAXBElement<BigDecimal> createPO(BigDecimal value) {
        return new JAXBElement<BigDecimal>(_PO_QNAME, BigDecimal.class, null, value);
    }

    /**
     * Create an instance of {@link JAXBElement }{@code <}{@link BigDecimal }{@code >}}
     *
     */
    @XmlElementDecl(namespace = "", name = "TMP")
    public JAXBElement<BigDecimal> createTMP(BigDecimal value) {
        return new JAXBElement<BigDecimal>(_TMP_QNAME, BigDecimal.class, null, value);
    }


}

phildunlap

Hi mircsicz,

Edit: I see you editted your post to show you have run the xjc! Nice work! In case someone has not,

Did you attempt to run the xjc from your Java/jdk/bin/ on the xsd you generated? If you have the schema.xsd saved in /path/to/dir, you can cd /path/to/dir; /path/to/java/jdk/bin/xjc schema.xsd That should have output three Java classes into the same directory as the schema, RMCU.java, TP.java and VALUES.java. From your schema, we can see TP and VALUES objects are children of / inside the RMCU element. So, RMCU is what we'll adapt into our importing class by simply adding the interface designation after the class declaration:

public class RMCU implements AbstractXMLDataSource {

we'll want to make things simple for ourselves, so we're going to remove the package generated xjc placed at the top.

we'll have to import any objects we reference from Mango, like the ImportPoint or NumericImportPoint in the import section at the top, like,

import com.infiniteautomation.datafilesource.dataimage.ImportPoint;
import com.infiniteautomation.datafilesource.dataimage.NumericImportPoint;
import com.infiniteautomation.datafilesource.contexts.AbstractXMLDataSource;
//We'll also use...
import java.text.SimpleDateFormat;
import java.text.ParseException;
import java.util.Map;
import java.util.HashMap;

And then we'll have to implement the one function that will send the points from this automatically generated XML reading class to something Mango can use, which is the getParsedPoints method. So, as the last method in the class, I will add,

public List<ImportPoint> getParsedPoints() {
  SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
  List<ImportPoint> result = new ArrayList<ImportPoint>();
  Map<String, String> extraParams = new HashMap<>();
  List<TP> tps = getTP();
  System.out.println("TPs length: " + tps.size()); //Not in log, in console.
  for(TP tp : tps) {
    //You probably want to add NumericImportPoint instances to the result list.
    // Which I guess will be the values. Like,
    for(VALUES value : tp.getVALUES()) {
      try {
        //System.out.println("Adding value..." + sdf.parse(value.getDATETIME())); //Not in log, in console.
        //System.out.println("Adding value..." + value.getEE());
        if(value.getEE() != null)
          result.add(new NumericImportPoint("This EE point xid", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));
        if(value.getTMP() != null)
          result.add(new NumericImportPoint("This TMP point xid", value.getTMP().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));
      } catch (ParseException e) { /* gobble */ }
    }
  }
  return result;
}

To create the list of data to import. Here's the whole class. Maybe I didn't import your example data quite the way you want, but it may help. You can place RMCU.java in your Mango/web/modules/dataFile/web/CompilingGrounds/XML directory and compile it on the data source's page.

//
// This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 
// See <a href="http://java.sun.com/xml/jaxb">http://java.sun.com/xml/jaxb</a> 
// Any modifications to this file will be lost upon recompilation of the source schema. 
// Generated on: 2018.03.05 at 03:16:52 PM MST 
//
import java.text.SimpleDateFormat;
import java.text.ParseException;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
import java.util.HashMap;
import java.util.Map;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlSchemaType;
import javax.xml.bind.annotation.XmlType;
import javax.xml.bind.annotation.adapters.CollapsedStringAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;

import com.infiniteautomation.datafilesource.dataimage.ImportPoint;
import com.infiniteautomation.datafilesource.dataimage.NumericImportPoint;
import com.infiniteautomation.datafilesource.contexts.AbstractXMLDataSource;

/**
 * <p>Java class for anonymous complex type.
 * 
 * <p>The following schema fragment specifies the expected content contained within this class.
 * 
 * <pre>
 * &lt;complexType>
 *   &lt;complexContent>
 *     &lt;restriction base="{http://www.w3.org/2001/XMLSchema}anyType">
 *       &lt;sequence>
 *         &lt;element ref="{}TP" maxOccurs="unbounded"/>
 *         &lt;element ref="{}ID"/>
 *         &lt;element ref="{}SN"/>
 *         &lt;element ref="{}IP"/>
 *       &lt;/sequence>
 *     &lt;/restriction>
 *   &lt;/complexContent>
 * &lt;/complexType>
 * </pre>
 * 
 * 
 */
@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "", propOrder = {
    "tp",
    "id",
    "sn",
    "ip"
})
@XmlRootElement(name = "RMCU")
public class RMCU implements AbstractXMLDataSource {
	
    @XmlElement(name = "TP", required = true)
    protected List<TP> tp;
    @XmlElement(name = "ID", required = true)
    @XmlJavaTypeAdapter(CollapsedStringAdapter.class)
    @XmlSchemaType(name = "NCName")
    protected String id;
    @XmlElement(name = "SN", required = true)
    protected BigInteger sn;
    @XmlElement(name = "IP", required = true)
    @XmlJavaTypeAdapter(CollapsedStringAdapter.class)
    @XmlSchemaType(name = "NMTOKEN")
    protected String ip;

    /**
     * Gets the value of the tp property.
     * 
     * <p>
     * This accessor method returns a reference to the live list,
     * not a snapshot. Therefore any modification you make to the
     * returned list will be present inside the JAXB object.
     * This is why there is not a <CODE>set</CODE> method for the tp property.
     * 
     * <p>
     * For example, to add a new item, do as follows:
     * <pre>
     *    getTP().add(newItem);
     * </pre>
     * 
     * 
     * <p>
     * Objects of the following type(s) are allowed in the list
     * {@link TP }
     * 
     * 
     */
    public List<TP> getTP() {
        if (tp == null) {
            tp = new ArrayList<TP>();
        }
        return this.tp;
    }

    /**
     * Gets the value of the id property.
     * 
     * @return
     *     possible object is
     *     {@link String }
     *     
     */
    public String getID() {
        return id;
    }

    /**
     * Sets the value of the id property.
     * 
     * @param value
     *     allowed object is
     *     {@link String }
     *     
     */
    public void setID(String value) {
        this.id = value;
    }

    /**
     * Gets the value of the sn property.
     * 
     * @return
     *     possible object is
     *     {@link BigInteger }
     *     
     */
    public BigInteger getSN() {
        return sn;
    }

    /**
     * Sets the value of the sn property.
     * 
     * @param value
     *     allowed object is
     *     {@link BigInteger }
     *     
     */
    public void setSN(BigInteger value) {
        this.sn = value;
    }

    /**
     * Gets the value of the ip property.
     * 
     * @return
     *     possible object is
     *     {@link String }
     *     
     */
    public String getIP() {
        return ip;
    }

    /**
     * Sets the value of the ip property.
     * 
     * @param value
     *     allowed object is
     *     {@link String }
     *     
     */
    public void setIP(String value) {
        this.ip = value;
    }
	
    public List<ImportPoint> getParsedPoints() {    
      SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");    
      List<ImportPoint> result = new ArrayList<ImportPoint>();    
      Map<String, String> extraParams = new HashMap<>();    
      List<TP> tps = getTP();    
      System.out.println("TPs length: " + tps.size()); //Not in log, in console.    
      for(TP tp : tps) {    
        //You probably want to add NumericImportPoint instances to the result list.    
        // Which I guess will be the values. Like,    
        for(VALUES value : tp.getVALUES()) {    
          try {    
            //System.out.println("Adding value..." + sdf.parse(value.getDATETIME())); //Not in log, in console.    
            //System.out.println("Adding value..." + value.getEE());    
            if(value.getEE() != null)    
              result.add(new NumericImportPoint("This EE point xid", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));    
            if(value.getTMP() != null)    
              result.add(new NumericImportPoint("This TMP point xid", value.getTMP().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));    
          } catch (ParseException e) { /* gobble */ }    
        }    
      }    
      return result;    
    }    
    
}

You will also need the VALUES.java and TP.java files created by xjc in your CompilingGrounds/XML directory. You will need to have the data source create points (a checkbox on the data source), or create points on the data source with the identifiers "This EE point xid" or "This TMP point xid" to see this data in Mango.

mircsicz

Hi Philip,

thx a ton for looking into my case and replying...

I've copied your RMCU.java to web/modules/dataFile/web/CompilingGrounds/XML and I've added the mentioned

import com.infiniteautomation.datafilesource.dataimage.ImportPoint;
import com.infiniteautomation.datafilesource.dataimage.NumericImportPoint;
import com.infiniteautomation.datafilesource.contexts.AbstractXMLDataSource;
//We'll also use...
import java.text.SimpleDateFormat;
import java.text.ParseException;
import java.util.Map;
import java.util.HashMap;

line's to my TP and VALUES.java files, besides that I've also removed the

package generated;

line. And I successfully compiled that new import Template!!! 8-)

Next thing I did was throwing 40k XML's in the directory:
alt text

So where can I now monitor the progress? I already have a multitail open monitoring ma.log, but that only stated:

INFO  2018-03-06T16:56:49,712 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:407) - Data source 'WSE_KA-DB' initialized
INFO  2018-03-06T16:56:49,713 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:412) - Data source 'WSE_KA-DB' took 3.304719ms to start

Edit:
just got an urgent Event:

'WSE_KA-DB': JAXBException: 1 counts of IllegalAnnotationExceptions

But I don't see it in the log and the server still seems busy with a load between 0.7 and 1...

How do I look under the hood? I still get those messages and no new data points show up

6.3.2018 17:15:26	device_hub  'WSE_KA-DB': Event from import class: java.lang.NullPointerException	No RTN	N/A	
6.3.2018 17:15:26	device_hub  'WSE_KA-DB': JAXBException: 1 counts of IllegalAnnotationExceptions	No RTN	N/A

Seems not to get it done:

WARN  2018-03-06T17:30:30,753 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source WSE_KA-DB aborted 1 polls since it started.

phildunlap

Hi mircsicz,

Did you modify the class before getting the IllegalAnnotationException? I had to work through one of those when I had added the simple date format as a class member variable. You shouldn't declare any additional variables in the automatically generated class, try to keep those variables all within the getParsedPoints method. If you use the API instead of poll to try importing the file, you can monitor status at the temporary resource location it provides you. Through the UI, you should be redirected to a page you could refresh by clicking...

Other than that, printing or throwing exceptions from your implementing class is the easiest way to get debug information. If you throw new java.lang.RuntimeException("my message"); you'll get a stack trace to that part of your code in the log and in an event, or you can simply System.out.println() as I did, but you would need to be watching stdout

mircsicz

Thank you Philip,

but currently I'm locked out. I changed my passwd and probably had twice the same typo, so I can't relogin. And sadly I can't reach the H2 WebConsole as of this error

mircsicz

Hi Philip,

I've uploaded using the API and clicked "Submit, get monitor":

{"resourceId":"DF_672777d1-dc31-4ec8-bb0f-091d52697197","progress":0,"finished":true,"cancelled":false,"errors":["Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions"],"unfoundIdentifiers":[],"createdPoints":[],"failedPoints":[],"totalImported":0,"priority":2,"queueSize":0,"taskId":"TR_DF_672777d1-dc31-4ec8-bb0f-091d52697197","threadName":"Temporary Resource Timeout for : DF_672777d1-dc31-4ec8-bb0f-091d52697197","expires":1520368586474}

phildunlap

Refresh that page, you should get some status information

mircsicz

@phildunlap said in Import EnergyData from XML:

Refresh that page, you should get some status information

That's all I get... :-(

phildunlap

@mircsicz said in Import EnergyData from XML:

Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions

Oh, yeah, didn't look closely. You have the same error: Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions

Did you see my note about adding member variables? Are you using the class I provided? I did test it using the XML you provided....

mircsicz

Yes I saw your note and all I added to the other two *.java files was the import part's you mentioned...

And yes copy/pasted the provided class ;-)

phildunlap

You can supply XML if you think something went awry with your sample data, but the snippet you posted I have imported successfully again just now, using a copy-paste of the class I supplied as RMCU.java and the TP.java / VALUES.java files that were generated (but I removed package generated from them, you didn't need to modify TP and VALUES any other way). The illegal annotation suggests to me something in that process didn't go right.

mircsicz

Argghhh, I've removed the additional "import" line's from TP and VALUES.java but I still get no positive feedback:

{"resourceId":"DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","progress":0,"finished":true,"cancelled":false,"errors":["Loading file ka-db failed: 1 counts of IllegalAnnotationExceptions"],"unfoundIdentifiers":[],"createdPoints":[],"failedPoints":[],"totalImported":0,"priority":2,"queueSize":0,"taskId":"TR_DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","threadName":"Temporary Resource Timeout for : DF_a963f6b1-51fe-445e-9d3e-d255342e2f58","expires":1520371072215}```

mircsicz

THX to phil's support it's now importing and creating data point's

But it's so far not the correct data point seperation!

It needs to be seperated by TPID's

EDIT: Sorry was in a rush as my son woke up while I was typing...

The structur of the XML is as follow's:

RMCU
   TP ID=NAME-OF-METERING-POINT
      VALUES
        DATETIME
         EE (Electric Energy)
         TMP (Temperatur)
         PO (Power)
         HQ (Heat Quantity)
      VALUES
   TP
ID
 NAME-OF-DATALOGGER
ID
SERIAL
  SERIAL-OF-DATALOGGER
SERIAL
IP
   IP-OF-DATALOGGER
IP
RMCU

Each TP-ID's VALUE tagged combined with DATETIME needs to become it's own Data Point

phildunlap

Glad you got it working. You probably want to get those TPIDs into the identifier for the NumericImportPoint (the first argument in creating these objects).

mircsicz

Hi Philip,

yeap that's exactly what I want! But I can't realize the change, I guess I've to change "This EE point xid" in

if(value.getEE() != null)    
              result.add(new NumericImportPoint("This EE point xid", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));

But as I don't have any java experience I simply don't know how, So if you could phrase it for me once I'ld hopefully be done and gone... ;-)

EDIT: I just saw that the Data Point is literally named: "This EE point xid" so if you'ld be so kind to tell me how to parse the string behind the "tp" var into that place that would probably be it. It expect the DataPoint name to be something like:

GHV1 3P6 5OG_EE
Case-Temp_TMP
or in other cases:
GHV1 3P6 5OG_PO
etc

phildunlap

Correct part of the code. I'm guessing you want the ID from the TP element and the EE/TMP nature of the value. Something like,

if(value.getEE() != null)
  result.add(new NumericImportPoint(tp.getID() + " EE value", value.getEE().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));
if(value.getTMP() != null)
  result.add(new NumericImportPoint(tp.getID() + " TMP value", value.getTMP().doubleValue(), sdf.parse(value.getDATETIME()).getTime(), extraParams));

phildunlap

Seeing you comment on the schema two posts ago, you also will need to add conditions for if(value.getPO() != null) and probably HQ as well.

mircsicz

Yeap Phil, I'm aware of that. But with a working basic example like this one I can abstract the rest...

THX a ton Philip I reran the import on al 50k XML files and I now have the expected data point's.... I'll now start to adopt for other value type's!!!