Processing files

psysak

Hey guys, I'll preface this by saying that this will be a totally internal application so as stupid as it sounds it should be fairly secure.
Is it possible to accept a file upload via just a web page being served by Mango, process it on the server and then allow the new file to be downloaded again? I just need to write a quick little csv file cleanup utility for a colleague which would basically take in a csv, run a regex against it to select and replace a couple of things and then offer it back to him. He's doing something manually in excel right now and a simple little thing like that would save them a bunch of time.

Jared Wiltshire

Personally, I would create a little dashboard to do what you need to do in his browser, you wouldn't need to upload the file to Mango at all. There's obviously a lot of ways to skin this cat and I'm sure my colleagues might chime in with another solution. You could for example upload the file to the file store via a dashboard and do the processing via a script run on the back-end.

Front-end only solution using Papa Parse (see their documentation for parsing / output options). You need to add this as a user module - https://help.infiniteautomation.com/getting-started-with-a-user-module/

define(['angular', 'require', 'papaparse'], function(angular, require, Papa) {
    'use strict';
    
    const userModule = angular.module('userModule', ['maUiApp']);
    
    class ProcessCsvController {
        fileSelected(event) {
            const files = event.target.files;
            if (files.length) {
                this.parseFile(files[0]);
            }
            event.target.value = null;
        }
        
        parseFile(file) {
            Papa.parse(file, {
            	complete: results => {
            		this.modifyRows(results.data);
            	}
            });
        }
        
        modifyRows(rows) {
            rows.forEach(row => {
                // whatever you want to do
                console.log(row);
            });
            
            this.downloadCsv(rows, 'myFile.csv');
        }
        
        downloadCsv(rows, filename) {
            const csvBlob = new Blob([Papa.unparse(rows)]);
            const url = URL.createObjectURL(csvBlob);
            try {
                const a = document.createElement('a');
                a.style.display = 'none';
                a.href = url;
                a.download = filename;
                document.body.appendChild(a);
                a.click();
                document.body.removeChild(a);
            } finally {
                URL.revokeObjectURL(url);
            }
        }
    }

    userModule.component('processCsv', {
        template: `<input type="file" accept="text/*,.csv,.txt" ma-change="$ctrl.fileSelected($event)">`,
        controller: ProcessCsvController
    });
    
    return userModule;
}); // define

All you have to do to use this is plonk it onto a page -

<process-csv></process-csv>

psysak

@jared-wiltshire COOL!! I'll have a look at that module, thank you so much!

phildunlap

I don't suppose he's using Linux / Mac, and/or Windows 10 with the bash command? If so, I'd almost certainly do it with some sed commands in a bash script.

If not, I agree with Jared that a frontend solution may be good. You could build your colleague a simple sanitize.html page with two textbox elements, and upon change in the input invoke a function to transform it to the output,

function transformInputCSV() {
  var input = document.getElementById("inputTextbox").value;
  var output = input.replace(/your regex here/, "You're replacement text");
//  ouput = output.replace(.....)
  document.getElementById("outputTextbox").value = output;
}

Here's the HTML for such a page, which he could open in a browser,

<html>
<head>
<script>
function transformInputCSV() {
  var input = document.getElementById("inputTextbox").value;
  var output = input.replace(/your regex here/, "Your replacement text");
//  output = output.replace(.....)
  document.getElementById("outputTextbox").value = output;
}
</script>
</head>
<body>
<div><textarea id="inputTextbox" onchange="transformInputCSV()"></textarea></div>
<div><textarea id="outputTextbox" ></textarea></div>
</body>
</html>

But it would be possible to tie Mango into things if there is a reason to. Have you checked out the API endpoints for the data file data source? One permits you to submit files for immediate processing, and there's no reason an importer could not simply place the post-processed file in the filestore for a user this user has access to.