Problem with defining class for CSV dataFile import

[episensor]

Hi all - would appreciate your help with something - I'm trying to define a class that will allow me to import CSV data from the local filesystem, that's in the following format:

ExportID, Date, Value
000D6F00010B760C_100333,2014-05-01T14:08:00,213855.4
000D6F0001A31EAF_100333,2014-05-01T14:08:00,348968.6
000D6F00010B75B8_100333,2014-05-01T14:09:00,347326.1
000D6F00010B7540_100333,2014-05-01T14:09:00,480619.3
000D6F00010B73A2_100333,2014-05-01T14:09:00,424276.6

The dateTime stamp is in the format yyyy-MM-ddThh:mm:ss

There's an example of a CSV import class given in the dataFile module that goes something like this:

**extends AbstractCSVDataSource {

DateTimeFormatter dtf = DateTimeFormat.forPattern("MM/dd/yyyy hh:mm");
Map<String, Integer> headerMap = new HashMap<String, Integer>();

@Override

public void importRow(String[] row, int rowNum) {

    if(rowNum == 0) for (int k = 0; k < row.length; ++k) { headerMap.put(row[k], k); }
    
    else this.parsedPoints.add(

        new NumericImportPoint(
        row[headerMap.get("ID")],
        Double.parseDouble(row[headerMap.get("Temperature")]),
        dtf.parseDateTime(row[headerMap.get("Date")] + " " + row[headerMap.get("Time")]).getMillis() ) );

}

}**

As it stands, I'm getting the following compile error:

error: constructor NumericImportPoint in class NumericImportPoint cannot be applied to given types;
new NumericImportPoint(
^
required: String,double,long,Map<String,String>
found: String,double,long
reason: actual and formal argument lists differ in length

So I'm thinking NumericImportPoint is not being mapped correctly. Here's what NumericImportPoint looks like for reference:

**package com.infiniteautomation.datafilesource.dataimage;

import java.util.Map;

import com.infiniteautomation.datafilesource.dataimage.ImportPoint;

public class NumericImportPoint extends ImportPoint{

    public final double value;

    public NumericImportPoint(String identifier, double value, long time, Map<String, String> extraParams) {
            super(identifier, time, extraParams);
            this.value = value;
    }

    @Override
    public DataType getDataType() {
            return DataType.NUMERIC;
    }

}**

What I'm trying to do is (a) get the original code to a point where it will compile and then (b) modify this code for my CSV file above.

A sample of the CSV file that the class within the documentation is designed for would be helpful also!

Thanks,
Brendan

[phildunlap]

I need to update that documentation! The fourth argument is a Map of a few traits you can specify for created points, if you use the data source to create the points.

To resolve, add a fourth argument to the constructor:

new HashMap<String, String>()

I will update the documents...

[phildunlap]

Here is the CSV that resulted in that example:

"Nr.","Date","Time","Meas.Unit","Value","Unit","Offset","Unit","Alpha","Unit","Temperature","Unit","ID"
0001,"12/19/2013","13:07","Extract",-0.0,"°Plato",,,,,20.1,"°C","FV6-01"
0002,"12/19/2013","13:08","Extract",-0.1,"°Plato",,,,,20.5,"°C","FV6-02"
0003,"12/19/2013","13:36","Extract",-0.0,"°Plato",,,,,20.7,"°C","FV7-01"
0004,"12/19/2013","13:38","Extract",-0.0,"°Plato",,,,,20.8,"°C","FV7-02"

[episensor]

Hi Phil,

That's great - thanks very much. I think I'm nearly there, but I'm not quite sure how to construct this - what I have is the following:

extends AbstractCSVDataSource {

DateTimeFormatter dtf = DateTimeFormat.forPattern("yyyy-MM-ddThh:mm:ss");
Map<String, Integer> headerMap = new HashMap<String, Integer>();
// new HashMap argument
@Override

public void importRow(String[] row, int rowNum) {

    if(rowNum == 0) for (int k = 0; k < row.length; ++k) { headerMap.put(row[k], k); }
    
    else this.parsedPoints.add(
                           
        new NumericImportPoint(
        row[headerMap.get("ExportID")],
        dtf.parseDateTime(row[headerMap.get("Date")]),
        Double.parseDouble(row[headerMap.get("Value")]) ) );
    
}

}

/* to parse the following CSV file:

ExportID, Date, Value
000D6F00010B760C_100333,2014-05-01T14:08:00,213855.4
000D6F0001A31EAF_100333,2014-05-01T14:08:00,348968.6
000D6F00010B75B8_100333,2014-05-01T14:09:00,347326.1
000D6F00010B7540_100333,2014-05-01T14:09:00,480619.3
000D6F00010B73A2_100333,2014-05-01T14:09:00,424276.6

*/

If you could nudge me in the right direction that would be greatly appreciated!

Best Regards,
Brendan

[phildunlap]

@episensor said:

Hi Phil,

That's great - thanks very much. I think I'm nearly there, but I'm not quite sure how to construct this - what I have is the following:

extends AbstractCSVDataSource {

DateTimeFormatter dtf = DateTimeFormat.forPattern("yyyy-MM-ddThh:mm:ss");
Map<String, Integer> headerMap = new HashMap<String, Integer>();
// new HashMap argument
@Override

public void importRow(String[] row, int rowNum) {

   if(rowNum == 0) for (int k = 0; k < row.length; ++k) { headerMap.put(row[k], k); }
   
   else this.parsedPoints.add(
                          
       new NumericImportPoint(
       row[headerMap.get("ExportID")],
       dtf.parseDateTime(row[headerMap.get("Date")]),
       Double.parseDouble(row[headerMap.get("Value")]),
       new HashMap<String, String>() ) );

}
}

/* to parse the following CSV file:

ExportID, Date, Value
000D6F00010B760C_100333,2014-05-01T14:08:00,213855.4
000D6F0001A31EAF_100333,2014-05-01T14:08:00,348968.6
000D6F00010B75B8_100333,2014-05-01T14:09:00,347326.1
000D6F00010B7540_100333,2014-05-01T14:09:00,480619.3
000D6F00010B73A2_100333,2014-05-01T14:09:00,424276.6

*/

If you could nudge me in the right direction that would be greatly appreciated!

Best Regards,
Brendan

I made the change to the constructor in the quote block that I think is missing.

[episensor]

Hi Phil, Thanks for the help - the compile worked with the following code in case it's of use to anyone else:

import java.util.HashMap;
import java.util.Map;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import com.infiniteautomation.datafilesource.contexts.AbstractCSVDataSource;
import com.infiniteautomation.datafilesource.dataimage.NumericImportPoint;

public class epiCSV

extends AbstractCSVDataSource {

DateTimeFormatter dtf = DateTimeFormat.forPattern("yyyy-MM-ddThh:mm:ss");
Map<String, Integer> headerMap = new HashMap<String, Integer>();
@Override

public void importRow(String[] row, int rowNum) {

if(rowNum == 0) for (int k = 0; k < row.length; ++k) { headerMap.put(row[k], k); }

else this.parsedPoints.add(

new NumericImportPoint(
row[headerMap.get("ExportID")],
Double.parseDouble(row[headerMap.get("Value")]),
dtf.parseDateTime(row[headerMap.get("Date")]).getMillis(),
new HashMap<String, String>() ) );

}
}

I copied the class into /web/modules/dataFile/web/templates/CSV/ and removed the .class extension, removed the placeholder file, saved the datasource, restarted etc. but I can't seem to get the 'template' to show up in the drop down list on the UI when adding a new data source (screenshot attached). Not sure if I'm missing another step?

Thanks,
Brendan

Attachment: download link

[phildunlap]

Why did you remove the .class extension? I believe that is the cause of it not being in the list.

[phildunlap]

Ah, that is mistaken in the documentation as well. I believe the .class is required.