So I wrote a very simple perl-script that automates this for me. It behaves like cat, but inserts empty lines between input lines proportional to the amount of time between them. The numbers of lines inserted in logarithmically proportional to the elapsed time: one line for the first second, a second line for the next two seconds, a third line for the next 4 seconds, …

$ tail -f /var/log/mail.log | logtail.pl | sed 's/ .*//'
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
 
 
 
 
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
 
 
 
 
2011-03-09T10:34:01+01:00
2011-03-09T10:34:01+01:00
 
 
 
 
 
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
 
 
 
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00

As soon as you download the data, the Log viewer is opened automatically presenting the full log. You can limit the graph to only one session (flight). The bottom shows the different datasets. Clicking their name adds them to the graph. You can zoom and pan by using the mouse; while hoovering the cursor, the bottom numbers show the numerical value of the different parameters at that instant in time.

You can save the data into a “csv file“. The term CSV is rather loosly interpreted, as the file contains some “comment lines” that are not in CSV-format. While the Log Viewer tool provides an export function for the graph data as well, I didn’t feel like exporting half a dozen of graphs for every flight by hand. So I stirred some perl magic and gnuplot wizardry together into a script that reads the CSV log file and produces a set of graphs for each session (flight) in the log:

Google pointed me to 010 Editor which supports binary templates, which is pretty much what I was looking for. However, this tool only runs on Windows and is commercial. Enough reason to look further.

That’s when I came across the Data::ParseBinary perl module, which is a true relief to use. It supports pretty much every thing you need to parse a binary file:

• Signed and unsigned integers
• Big and little endian
• 8, 16, 32 and 64 bit integers
• Bitfields
• Enum-types to specify your own names for values
• If-constructs: Fields are present or not depending on the value of another field

In short, an incredible tool!

Besides the standard password authentication, ssh also supports public key authentication. This key-based authentication has the added bonus of having per-key options:

• you can restrict the source IP from which this key may be used
• you can force a command to be executed instead of allowing the connecting side to specify one

Combining the power of these tools gives very fine grained control over the rsync process: you can create a “backup key” that only allows you to rsync from the server and only from a specified directory. Any other command besides rsync is rejected; rsync to the server or from another directory is also rejected. This script does it all, it’s part of the rsync package, but not installed by most distro’s. An example authorized_keys entry (with an abbreviated key):

command="/home/boss/niels/bin/rrsync.pl -ro /home" ssh-rsa AAAAB[...]fE+8QrME= 20090329 rsync key

Since this is electrical measuring equipment, the serial communication should be electrically insulated from the device. The VC-940 uses a photodiode. Since the cable connecting the multimeter to the computer only contains an optical receiver, it is impossible for the computer to communicate with the meter. This significally reduces the complexity of a protocol.

This optical part showed to be the first problem. When I insert my RS-232 tester when running the Windows application, I could see data passing; under MacOSX no data would pass.  Besides showing no data, the tester also indicated that the control lines were asserted differently. By asserting the control lines the same way under MacOSX, the problem was resolved. Apparently, the voltages on the control lines are used to power the optical receiver electronics.

The baudrate and other settings were found by simply trying different values. When receiving at 2400bps, 8 databits, no parity and 1 stopbit (2400 8N1 for short), measurements were recorded in the Windows application.

This perl-script opens the specified serial port, asserts the control lines as required (RTS logical 0, positive voltage; DTS logical 1, negative voltage) and outputs the data to stdout.

The message format

When observing the data flow, it was clear that the device sends out a byte-string of 11 bytes roughly every second:

B0B0B0B0B03131B0B50D8A
B0B0B0B0B03131B0B50D8A
B0B0B031B63131B0310D8A
B0B034B3323432B3310D8A
B0B03237B53432B3310D8A
...

Every message seems to be terminated by 0x0D8A. I used this to synchronize the input. The following table shows what I discovered for the other bits in the message. The nibbles (4 bit group) are numbered from left to right starting from 0.

Note that I didn’t figure out what every bit means…

The modes are listed below:

This perl-script parses the binary input (from stdin) to readable output:

2009-05-01-16-48-52 : 313834b5b3b0b3b0b0 :  184.53 mV
2009-05-01-16-48-53 : 3138343434b0b3b0b0 :  184.44 mV
2009-05-01-16-48-54 : 313834b534b0b3b0b0 :  184.54 mV
2009-05-01-16-48-55 : 313834b3b0b0b3b0b0 :  184.30 mV
2009-05-01-16-48-56 : 313834b037b0b3b0b0 :  184.07 mV
2009-05-01-16-48-57 : 3138b338b5b0b3b0b0 :  183.85 mV

The inverter (the device that converts DC into AC) is a SolarMax C-series. It has a 2-line LCD display that gives out some basic information: current, voltage, power; produced energy today, this month, this year, … This is very useful information, but is a bit hard to access. The instruction manual reveals that there is a computer interface available to read out its data. Naturally, I wanted to explore this!

The interface is physically an 8P8C (usually called RJ45). Electrically it’s a serial interface. The manual isn’t exactly clear whether it’s an RS-232 or an RS-485 interface. After some mailing and calling, the people at Sputnik Engineering just mailed us the pinout diagram and document (both in German); it appeared to be RS-232. Their website even has a free utility called MaxTalk to read out the data!

The MaxTalk program is very basic. It only reads out the data, but does not process it in any way. However, it showed that the PC communicated with the inverter, which is great news!

Reverse engineering the protocol

I wanted to get access to this data myself. This would allow me to run some statistics and/or graphing tools on them. I did not find any documentation on the protocol on the Internet, so I started to reverse engineer it myself.

I’m not sure of the legal aspect of this. Some facts to keep me safe: the software did not present a EULA; I only inspected the bytes traveling over the serial interface and did not touch the application in any way.

There is a wonderful tool called socat that allows me do some plumbing with a data stream: The MaxTalk application is running inside a VMware machine. The serial port of this machine is connected to a unix-domain socket by VMware itself. Socat connects this socket to the real serial interface and spits out a copy of the stream as a bonus. You can even bridge the serial line over TCP if you want to!

Since RS-232 is a fairly easy protocol, the easyest way is just to try out the different settings. I figured out that the SolarMax talked at 19200bps, using 8 databits, no parity and a single stop bit (19200 8N1 for short). The corresponding socat line thus became:

# socat -v /dev/ttyS1,b19200,cs8,parenb=0,cstopb=0 UNIX-CONNECT:/tmp/serial

Here is the (some of the) output I got. I re-assembled the fragments together to make it more readable:

<  {FB;05;4E|64:E1D;E11;E1h;E1m;E1M;E2D;E21;E2h;E2m;E2M;E3D;E31;E3h;E3m;E3M|1270}
>  {05;FB;7C|64:E1D=16;E11=8002;E1h=12;E1m=39;E1M=3;E2D=1C;E21=8004;E2h=E;E2m=21;
>  E2M=3;E3D=6;E31=8004;E3h=10;E3m=23;E3M=4|1C4E}
--
<  {FB;05;36|64:CAC;KHR;KDY;KMT;KYR;KT0;KLD;KLM;KLY|0D34}
>  {05;FB;59|64:CAC=1F3E;KHR=26D6;KDY=8E;KMT=8D;KYR=221;KT0=18C7;KLD=78;KLM=F8;
>  KLY=A8F|154C}
--
<  {FB;05;3E|64:ADR;DDY;DMT;DYR;PIN;SWV;THR;TMI;TYP;FRD;LAN|0FC4}
>  {05;FB;61|64:ADR=5;DDY=E;DMT=4;DYR=9;PIN=1428;SWV=7B;THR=13;TMI=A;TYP=BC2;
>  FRD=7D60905;LAN=1|178D}

The socat output shows that the messages are pure ASCII, which is a lot easier to figure out than raw binary.

From this point on, it’s just some trail-and-error to figure out what the message format looks like, how the responses are formatted and what they mean.

The message format

Each message has the following format:

• ‘{‘ : start of message indicator
• 2 characters source address in hex
• ‘;’
• 2 characters destination address in hex
• ‘;’
• 2 characters message length in hex
• ‘|64:’
• data
• ‘|’
• 4 characters checksum in hex
• ‘}’ : end of message indicator

The computer seems to have address 0xFB. The SolarMax has address 0×05, which is what we configured.

The length field is over the complete message, including the curly braces.

The checksum field is a standard 16-bit checksum over the body: the first character is the source address, the last character is the ‘|’ before the checksum.

In messages from computer to SolarMax, the data consists of one or more question-codes, separated by a ‘;’. The response contains the same codes, but includes the answers after an ‘=’ sign.

The codes

The meaning of the codes was largely educated guessing. Grab all known codes several times and compare what differs. Also compare this with what the Windows-interface shows.

This perl-script enumerates all codes that I found. I figured out the meaning of most of them.

# ./solarmax-test.pl /dev/ttyS1 5
Opening serial port /dev/ttyS1
Setting 19200 8N1
Reading all pending bytes
Address (ADR): 5
Type (TYP): 0xBC2
Software version (SWV): 12.3
Date day (DDY): 15
Date month (DMT): 4
Date year (DYR): 9
Time hours (THR): 18
Time minutes (TMI): 11
Error 1, number (E11): 32770
Error 1, day (E1D): 22
Error 1, month (E1M): 3
Error 1, hour (E1h): 18
Error 1, minute (E1m): 57
Error 2, number (E21): 32772
Error 2, day (E2D): 28
Error 2, month (E2M): 3
Error 2, hour (E2h): 14
Error 2, minute (E2m): 33
Error 3, number (E31): 32772
Error 3, day (E3D): 6
Error 3, month (E3M): 4
Error 3, hour (E3h): 16
Error 3, minute (E3m): 35
Operating hours (KHR): 9953
Energy today [Wh] (KDY): 12900
Energy yesterday [Wh] (KLD): 14200
Energy this month [kWh] (KMT): 154
Energy last monh [kWh] (KLM): 248
Energy this year [kWh] (KYR): 558
Energy last year [kWh] (KLY): 2703
Energy total [kWh] (KT0): 6356
Language (LAN): 1
DC voltage [mV] (UDC): 344700
AC voltage [mV] (UL1): 239400
DC current [mA] (IDC): 330
AC current [mA] (IL1): 520
AC power [mW] (PAC): 89000
Power installed [mW] (PIN): 2580000
AC power [%] (PRL): 3
??? Start ups ??? (CAC): 8004
??? (FRD): 0x7D60905
??? (SCD): 0x3C
??? (SE1): 0x0
??? (SE2): 0x0
??? (SPR): 0x1
#

The result

For those interested: the above graph (with the exception of the ‘Showing from [...] to [...]‘ lines) was generated using the following rrdtool command:

rrdtool --imgformat PNG --start "06:00" --end "21:00" --upper-limit 3100 --lower-limit 0 --rigid \
DEF:Pac=/var/lib/rrdtool/solarmax.1.rrd:Pac:AVERAGE \
DEF:Energy=/var/lib/rrdtool/solarmax.2.rrd:energy:AVERAGE \
VDEF:Pac_cur=Pac,LAST \
VDEF:Pac_avg=Pac,AVERAGE \
VDEF:Pac_peak=Pac,MAXIMUM \
CDEF:Pac_Wh=Pac,3600,/ \
VDEF:Eac=Pac_Wh,TOTAL \
VDEF:Eac2=Energy,TOTAL \
"AREA:Pac#0000ff:Pac    " \
"GPRINT:Pac_cur:%6.2lf%sW" \
"COMMENT:\\n" \
"HRULE:Pac_avg#bbbbff:Average" \
"GPRINT:Pac_avg:%6.2lf%sW" \
"COMMENT:\\n" \
"HRULE:Pac_peak#ff0000:Peak   " \
"GPRINT:Pac_peak:%6.2lf%sW" \
"COMMENT:\\n" \
"GPRINT:Eac:int(Pac)   %6.2lf%sWh\\n" \
"GPRINT:Eac2:meas       %6.2lf%sWh\\n" \
--vertical-label "Watt" --title "Power"

The total energy for the day is calculated in two ways: ‘int(Pac)’ uses the sampled power-gauge to calculate the total; ‘meas’ uses the energy-counter of the inverter. The latter one should be more accurate, but can’t be read out with as much precision.

Update 2009-10-09

Chris has made his own variant of the above Perl script:

Hi Niobos,

here we go… find attached to this mail my modification to
your script (a very helpful piece of code .

Since i am not the perl-crack i’ve not yet implemented the
command “SYS” and the translation of the returned value
into a readable Operation-State.

At the moment i don’t know how to convert the returned
value i.e. of “SYS=4E24,0″ which is 20004,0 in decimal
and means an “Operation at MPP”.

ThX…
Chris

Since the serial interface came with no documentation whatsoever, I had to figure everything out myself. The serial connection on the back of the weather station is a DE-9 connector; the manual specifies it’s an RS232 output. After some trail-and-error work, I figured out that the communication happened at 9600bps, 8 databits per word, no parity and no stop-bit. Socat was happy to give the hexdump using the following command:

socat -u /dev/ttyS0,b9600,cs8,parenb=0,cstopb=0,raw – | hexdump -C

The bursts of data come automatically, there seems to be no need to initialize the weather station. The messages are a few bytes long each.

The messages look fairly similar to the ones described in this document (or my mirrorred copy) for the WMR918 station. This perl-script parses the binary data from its standard input (which will probably come from a socat-pipe) and prints it in human readable form to stderr. It also tries to write the data to rrdtool for archiving, which is what I use to generate nice graphs.

The Skynet site contains content that is protected by Intellectual Property rights. This program however is for educational purposes only; in most countries this is allowed under “fair-use”. To ensure this (and to keep me out of legal trouble), I slowed the program down to an unusable speed.

To figure out how it works, start by trying “./grab.pl –help”; if that doesn’t get you the answer, feel free to browse through the code.

Caching

When experimenting with the grabber, I figured out that the Skynet page requires your browser to revalidate each and every page it serves. Using a standard Perl HTTP-caching module was not an option, since the server disallows caching.

I decided to write my own caching plugin for Perl’s LWP module. It uses the special protocol “cachedhttp://www.example.com” and is controlled through extra headers in the request. For more information use “perldoc cachedhttp.pm”.

The code

grab.pl: the grabber and parser itself

URI/cachedhttp.pm: helper module to define the cachedhttp:// URI
LWP/Protocol/cachedhttp.pm: the real work is done here

I hereby release this code as GPL.