ASCII file data source, reading linux log file events

[jeremyh]

Hi

I want to read a GPIO pin on a Raspberry Pi, but I didn't have any luck trying to read the device file (as per this this thread) so I thought I would try it another way.

I wrote a small script in python that reads the GPIO pin and then writes any events to a log file.
The format of my log file is like this:

2015-05-24 19:31:03,705 INFO CLOSED
2015-05-24 19:29:48,691 INFO OPEN
2015-05-24 19:29:52,053 INFO CLOSED
2015-05-24 19:29:53,094 INFO OPEN

Is there any way to get the ASCII file data source to read the 'CLOSED' and 'OPEN' events plus the timestamp?
I have tried, I think, dozens of combinations of regex expressions!

Cheers
Jeremy

[JoelHaggar]

I'm really not good with Regex but maybe one of our developers can answer next week. Here is an example file and data source that we use on the MangoES which should help. systemInfo.txt

ASCII Data Source example JSON: System Info Data Source.json

Let me know if that helps. Awhile ago Terry was telling me it would be fairly easy to make a data source to talk to the Raspberry Pi GPIO as there is an existing java library we could use. We might have to do that one of these days.

[jeremyh]

Hey Joel

Those links 404 for me. I'll have a look in one our our MangoES units though.

Regex does my head in.

@JoelHaggar said:

Let me know if that helps. Awhile ago Terry was telling me it would be fairly easy to make a data source to talk to the Raspberry Pi GPIO as there is an existing java library we could use. We might have to do that one of these days.

That would be really cool. I have a rPi at home that I am playing around with for various automation tasks.

FWIW you can also access the GPIO as a linux device/file. Not sure about setting outputs or reading ADC (haven't looked into it) but to read an input (pin 25) you just initialise it like this:

pi@rpi $ echo 25 > /sys/class/gpio/export

Then when you access this file:

/sys/class/gpio/gpio25/value

That file contains a single bit, either 0 or 1.

[jeremyh]

I had a look at the MangoES regex examples. Using regexr.com to test, I copied the regex expression format and ran it against my logfile:

Capture group 1 appears to contain 'INFO' and capture group 2 appears to contain the value ('OPEN').
I set up my point as follows:

This regex does capture the timestamp along with the point identifier and point value:

But even with my point setup as follows, I still get 'active source point has no value:

At this point I believe I'm doing everything correctly, so I'm a bit stuck.
If anyone else is able to take a crack at this I'd really appreciate the help!

[jeremyh]

OK, I appear to have gotten Mango reading the file and timestamp correctly.. It seems I just needed to select 'alphanumeric' not 'multistate'. However there is now an issue with repeated entries.

I'm not sure whether this is the expected behaviour of the module or not. If not I may be able to modify my script/log to suit.. At the moment it appends a new line at the bottom of the log every time the GPIO pin has a rising or falling edge.

When I create two lines in my log file like this:

Mango captures it like this:

'Justnow' and 'verysoon' are dummy values I made up. The timestamps are in the past and future respectively.

What I want is for Mango to just add history point for every line in the log file. Then Mango can read the file every minute and if my sensor was tripped 4 times, I'll get those 4 timestamped events. In other words, the Mango point history should have the same number of rows as my logfile, with the same timestamped values.

Is this possible or am I way off from what the module is designed to do?

[JoelHaggar]

This data source will not bring in history and will only look for a current value. If you could get your data into a csv format you could use the dataFile data source to import historical data.

If you do not have the NoSQL database then it's possible to have duplicate time stamps which is what you are seeing here.

[jeremyh]

@JoelHaggar said:

This data source will not bring in history and will only look for a current value. If you could get your data into a csv format you could use the dataFile data source to import historical data.

If you do not have the NoSQL database then it's possible to have duplicate time stamps which is what you are seeing here.

Hey Joel,

If I modify my python script so that the logfile is just overwriting the same single line (rather than amending a new line to the log), with the current timestamped state, would that give me the results that I expect? Or would I still get duplicate entries?

Cheers
Jeremy

[JoelHaggar]

I'm not sure as I've never used it this way but it probably depends on how you have the point set to log. If you have it set to log on change I would expect you would not get duplicates. You might see duplicates in the cache but only the change should be logged to the database.