Data Import Flexibility

---

TL;DR: The Data Import feature requires the next datetimestamp in the sheet/csv to be newer than the one before it, and that's not cool.

Alright, I believe I've found, in my opinion, a less-than-desirable behavior in the data import feature. I stumbled upon it while importing data from an old instance of Mango where for some reason it wouldn't show me "chart" info, and therefore wouldn't give me the option to download the data.

So naturally, I clicked on "Show xxx most recent records," and copied and pasted it into a spreadsheet that matched the format of a normal export.

I spent HOURS wondering what I had done wrong, for every time I imported, it would show me

Point values to be written           7218

Or some such nonsense. And it would NEVER go away until I rebooted. So, after much gnashing of teeth, I actually found one or two imports here and there among the objects I was trying to import from, and realized that sometimes it was importing the FIRST value listed, but not others. BUT WHY!?

The dates were from newest->oldest, not oldest->newest. Now, call me crazy, but if each line is individually evaluated, there should be no reason you can't import with ANY time in ANY cell order, but there it is. It seems to me that the Data Import will ONLY import data if the next date in the list is NEWER than the last, which makes absolutely no sense to me.

I have already imported all necessary data as of this moment, but I figured y'all might want to either fix it or maybe someone else has this issue and they can find the solution here.

Hi thewizardguy,

To be clear, you are using the /dataImport.shtm page?

There is no time ordering requirement. That module is open source, you can see what the controller would be doing right here: https://github.com/infiniteautomation/ma-modules-public/blob/main/Point Data Import/src/com/serotonin/m2m2/dataImport/DataImportController.java

I have tested that the order doesn't matter.

Perhaps there was an error in your log file?

I've run into errors, but at the moment I'm extraordinarily confident in this being the issue. Errors mostly prevent me from even importing. Yes, I'm using /dataImport.shtm.

One import with a time range that is jan-17->november-16 for instance will allow the import to occur, but show the dreaded "points to be written" forever, but a november-16->jan-17 import with the same format in every other way will import perfectly and write the objects within seconds.

The only thing in the log file is "Error during receive: NPDU [from=Address blahblahblah" all morning from 8am to now.

I just tested it again, and confirmed results. To be clear, the import APPEARS to work, but then it never writes and the "Point Values to be written" field in /internal/status.shtm never drops below the imported amount.

[1_1517512951906_mangoExampleBroken.xlsx](Uploading 100%) [0_1517512951906_mangoExampleWorking.xlsx](Uploading 100%)

how the hell do you upload files properly lol it doesnt look like its anything but text

@phildunlap said in Data Import Flexibility:

To be clear, you are using the /dataImport.shtm page?

I think that it would reduce confusion if the icon bar tooltip link to emport.shtm were changed from "Import/Export" to "Configuration Import/Export" to remind users that the "Data Import" link is elsewhere.

@pedro On mine it DOES say Configuration Import/Export for the tooltip on /emport.shtm (Mango 3.3)

@thewizardguy Glad to hear that was addressed. I'm still on 2.8.8, trying to work up the courage to upgrade.

@pedro If you're going to, I'd go all out, get a little NUC or something as a linux server, transfer the historical data/configs, don't try to just straight upgrade. Either that, or just get the Mango ES, it's been great for us. Just wire up some 24VDC power from a panel like a typical controller, but you can SSH into it. Viola!

In my experience, the windows version of Java doesn't allow mango to work as well as the linux implementation, and is far less stable (due to windows not being stable).

You can email me those data XLSX files (I thought you were using CSV by the OP) as I am unable to reproduce.

@phildunlap I used both CSV and Excel, I actually discovered the issue on the csv version, but it works the same for me either way. Emailing files now.