So I wrote a very simple perl-script that automates this for me. It behaves like cat, but inserts empty lines between input lines proportional to the amount of time between them. The numbers of lines inserted in logarithmically proportional to the elapsed time: one line for the first second, a second line for the next two seconds, a third line for the next 4 seconds, …

$ tail -f /var/log/mail.log | logtail.pl | sed 's/ .*//'
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
 
 
 
 
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
 
 
 
 
2011-03-09T10:34:01+01:00
2011-03-09T10:34:01+01:00
 
 
 
 
 
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
 
 
 
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00

Installing

I again used the openwrt modules, nsupdate is contained within bind-client. There are, however, several dependencies:

• libbind9.so.40.0.3, libdns.so.43.0.0, libisc.so.41.1.0, libisccc.so.40.0.0, libisccfg.so.40.0.3, liblwres.so.40.0.0 (and symlinks) from bind-libs
• libcrypto.so.0.9.8 from libopenssl

These are some serious libraries, takeing up 2.7MB of free space…

Configuring

I tried to use SIG(0), but that failed. nsupdate complains about a missing symbol ‘flockfile’. So I settled for TSIG authentication. Since this is a post about dd-wrt, I’ll assume the sever is already set up and tested (this will guide you through if it’s not), so I’ll go straight to the config files:

/jffs/etc/ddns.key:

fqdn.of.key. 0huPr3nqFnxUETlrM/VxGg==

/jffs/etc/config/ddns-update.wanup:

#!/bin/sh

# wanup scripts seem to run without LD_LIBRARY_PATH set
export LD_LIBRARY_PATH='/lib:/usr/lib:/jffs/lib:/jffs/usr/lib:/jffs/usr/local/lib:/mmc/lib:/mmc/usr/lib:/opt/lib:/opt/usr/lib'

# wanup scripts have the IPLOCAL variable set, but cron does not
if [ -z "$IPLOCAL" ]; then
 IPLOCAL=`ip addr sh dev ppp0 | grep 'inet ' | cut '-d ' -f6`
fi

sleep 30 # wait for IPv6, DNS, … to stabilize

echo -e "server ddns.master.server.fqdn\nkey `cat /jffs/etc/ddns.key`\n
update delete fqdn.to.set A\nupdate delete fqdn.te.set TXT\n
update add fqdn.to.set 300 A $IPLOCAL\nupdate add fqdn.to.set 300 TXT `date "+%Y-%m-%d_%H:%M:%S"`\n
send" | /jffs/bin/nsupdate

I cut that last echo-line into pieces for readability, make sure that it’s one single line (from echo all the way to nsupdate).

I added the following line to the Additional cron jobs on the webinterface. Contrary to the dd-wrt wiki page, /jffs/etc/crontab does not seem to work. This will run the ddns-update script every hour, at 5 minutes past the hour:

5 * * * *  root  /jffs/etc/config/ddns-update.wanup

As soon as you download the data, the Log viewer is opened automatically presenting the full log. You can limit the graph to only one session (flight). The bottom shows the different datasets. Clicking their name adds them to the graph. You can zoom and pan by using the mouse; while hoovering the cursor, the bottom numbers show the numerical value of the different parameters at that instant in time.

You can save the data into a “csv file“. The term CSV is rather loosly interpreted, as the file contains some “comment lines” that are not in CSV-format. While the Log Viewer tool provides an export function for the graph data as well, I didn’t feel like exporting half a dozen of graphs for every flight by hand. So I stirred some perl magic and gnuplot wizardry together into a script that reads the CSV log file and produces a set of graphs for each session (flight) in the log:

When using SSH in a script, most pages tell you to use public keys. While this is an excellent idea, it’s sometimes just not possible due to policy. This Expect script fakes a regular username-password login

#!/usr/bin/expect -f

set target [lindex $argv 0]
set password [lindex $argv 1]
set command [lindex $argv 2]

spawn ssh $target $command

match_max 100000
# Look for passwod prompt
expect “*?assword:*”
# Send password aka $password
send — “$password\r”
# send blank line (\r) to make sure we get back to gui
send — “\r”
expect eof

This script can be run like this:

./ssh-passwd.ex root@192.0.2.1 password “ls /root”

$command | \
perl -pe '@now=localtime();printf "%04d-%02d-%02dT%02d:%02d:%02d ",$now[5]+1900,$now[4]+1,$now[3],$now[2],$now[1],$now[0];'

The perl command reads in every line, prints the current time in the default format (or in whatever format you specify), followed by the read line.

However, “ssh://” is not a standard registered protocol. I’d like Putty to handle this. Also, “telnet://” gets you the standard windows telnet client instead of putty. Putty can be called with command line arguments. Supplying the “telnet://” url as a parameter works, but “ssh://” does not.

Hence, I wrote a very small wrapper program to accept “ssh://” URL’s and convert them to Putty command line arguments:

• Source code in C: ssh-to-putty.c
• Compiled Windows executable: ssh-to-putty.exe
• Registry commands to set putty as telnet-handler: putty telnet url handler.reg
• Registry commands to set the wrapper as ssh-handler: putty ssh url handler.reg

Some notes:

• The registry commands assume Putty and the wrapper are installed in C:\Progs\SSH. If this is not the case, you need to change the .reg-files accordingly
• The wrapper-program assumes putty.exe to be in the same directory as itself

Google pointed me to 010 Editor which supports binary templates, which is pretty much what I was looking for. However, this tool only runs on Windows and is commercial. Enough reason to look further.

That’s when I came across the Data::ParseBinary perl module, which is a true relief to use. It supports pretty much every thing you need to parse a binary file:

• Signed and unsigned integers
• Big and little endian
• 8, 16, 32 and 64 bit integers
• Bitfields
• Enum-types to specify your own names for values
• If-constructs: Fields are present or not depending on the value of another field

In short, an incredible tool!

Besides the standard password authentication, ssh also supports public key authentication. This key-based authentication has the added bonus of having per-key options:

• you can restrict the source IP from which this key may be used
• you can force a command to be executed instead of allowing the connecting side to specify one

Combining the power of these tools gives very fine grained control over the rsync process: you can create a “backup key” that only allows you to rsync from the server and only from a specified directory. Any other command besides rsync is rejected; rsync to the server or from another directory is also rejected. This script does it all, it’s part of the rsync package, but not installed by most distro’s. An example authorized_keys entry (with an abbreviated key):

command="/home/boss/niels/bin/rrsync.pl -ro /home" ssh-rsa AAAAB[...]fE+8QrME= 20090329 rsync key

Since this is electrical measuring equipment, the serial communication should be electrically insulated from the device. The VC-940 uses a photodiode. Since the cable connecting the multimeter to the computer only contains an optical receiver, it is impossible for the computer to communicate with the meter. This significally reduces the complexity of a protocol.

This optical part showed to be the first problem. When I insert my RS-232 tester when running the Windows application, I could see data passing; under MacOSX no data would pass.  Besides showing no data, the tester also indicated that the control lines were asserted differently. By asserting the control lines the same way under MacOSX, the problem was resolved. Apparently, the voltages on the control lines are used to power the optical receiver electronics.

The baudrate and other settings were found by simply trying different values. When receiving at 2400bps, 8 databits, no parity and 1 stopbit (2400 8N1 for short), measurements were recorded in the Windows application.

This perl-script opens the specified serial port, asserts the control lines as required (RTS logical 0, positive voltage; DTS logical 1, negative voltage) and outputs the data to stdout.

The message format

When observing the data flow, it was clear that the device sends out a byte-string of 11 bytes roughly every second:

B0B0B0B0B03131B0B50D8A
B0B0B0B0B03131B0B50D8A
B0B0B031B63131B0310D8A
B0B034B3323432B3310D8A
B0B03237B53432B3310D8A
...

Every message seems to be terminated by 0x0D8A. I used this to synchronize the input. The following table shows what I discovered for the other bits in the message. The nibbles (4 bit group) are numbered from left to right starting from 0.

Note that I didn’t figure out what every bit means…

The modes are listed below:

This perl-script parses the binary input (from stdin) to readable output:

2009-05-01-16-48-52 : 313834b5b3b0b3b0b0 :  184.53 mV
2009-05-01-16-48-53 : 3138343434b0b3b0b0 :  184.44 mV
2009-05-01-16-48-54 : 313834b534b0b3b0b0 :  184.54 mV
2009-05-01-16-48-55 : 313834b3b0b0b3b0b0 :  184.30 mV
2009-05-01-16-48-56 : 313834b037b0b3b0b0 :  184.07 mV
2009-05-01-16-48-57 : 3138b338b5b0b3b0b0 :  183.85 mV

The inverter (the device that converts DC into AC) is a SolarMax C-series. It has a 2-line LCD display that gives out some basic information: current, voltage, power; produced energy today, this month, this year, … This is very useful information, but is a bit hard to access. The instruction manual reveals that there is a computer interface available to read out its data. Naturally, I wanted to explore this!

The interface is physically an 8P8C (usually called RJ45). Electrically it’s a serial interface. The manual isn’t exactly clear whether it’s an RS-232 or an RS-485 interface. After some mailing and calling, the people at Sputnik Engineering just mailed us the pinout diagram and document (both in German); it appeared to be RS-232. Their website even has a free utility called MaxTalk to read out the data!

The MaxTalk program is very basic. It only reads out the data, but does not process it in any way. However, it showed that the PC communicated with the inverter, which is great news!

Reverse engineering the protocol

I wanted to get access to this data myself. This would allow me to run some statistics and/or graphing tools on them. I did not find any documentation on the protocol on the Internet, so I started to reverse engineer it myself.

I’m not sure of the legal aspect of this. Some facts to keep me safe: the software did not present a EULA; I only inspected the bytes traveling over the serial interface and did not touch the application in any way.

There is a wonderful tool called socat that allows me do some plumbing with a data stream: The MaxTalk application is running inside a VMware machine. The serial port of this machine is connected to a unix-domain socket by VMware itself. Socat connects this socket to the real serial interface and spits out a copy of the stream as a bonus. You can even bridge the serial line over TCP if you want to!

Since RS-232 is a fairly easy protocol, the easyest way is just to try out the different settings. I figured out that the SolarMax talked at 19200bps, using 8 databits, no parity and a single stop bit (19200 8N1 for short). The corresponding socat line thus became:

# socat -v /dev/ttyS1,b19200,cs8,parenb=0,cstopb=0 UNIX-CONNECT:/tmp/serial

Here is the (some of the) output I got. I re-assembled the fragments together to make it more readable:

<  {FB;05;4E|64:E1D;E11;E1h;E1m;E1M;E2D;E21;E2h;E2m;E2M;E3D;E31;E3h;E3m;E3M|1270}
>  {05;FB;7C|64:E1D=16;E11=8002;E1h=12;E1m=39;E1M=3;E2D=1C;E21=8004;E2h=E;E2m=21;
>  E2M=3;E3D=6;E31=8004;E3h=10;E3m=23;E3M=4|1C4E}
--
<  {FB;05;36|64:CAC;KHR;KDY;KMT;KYR;KT0;KLD;KLM;KLY|0D34}
>  {05;FB;59|64:CAC=1F3E;KHR=26D6;KDY=8E;KMT=8D;KYR=221;KT0=18C7;KLD=78;KLM=F8;
>  KLY=A8F|154C}
--
<  {FB;05;3E|64:ADR;DDY;DMT;DYR;PIN;SWV;THR;TMI;TYP;FRD;LAN|0FC4}
>  {05;FB;61|64:ADR=5;DDY=E;DMT=4;DYR=9;PIN=1428;SWV=7B;THR=13;TMI=A;TYP=BC2;
>  FRD=7D60905;LAN=1|178D}

The socat output shows that the messages are pure ASCII, which is a lot easier to figure out than raw binary.

From this point on, it’s just some trail-and-error to figure out what the message format looks like, how the responses are formatted and what they mean.

The message format

Each message has the following format:

• ‘{‘ : start of message indicator
• 2 characters source address in hex
• ‘;’
• 2 characters destination address in hex
• ‘;’
• 2 characters message length in hex
• ‘|64:’
• data
• ‘|’
• 4 characters checksum in hex
• ‘}’ : end of message indicator

The computer seems to have address 0xFB. The SolarMax has address 0×05, which is what we configured.

The length field is over the complete message, including the curly braces.

The checksum field is a standard 16-bit checksum over the body: the first character is the source address, the last character is the ‘|’ before the checksum.

In messages from computer to SolarMax, the data consists of one or more question-codes, separated by a ‘;’. The response contains the same codes, but includes the answers after an ‘=’ sign.

The codes

The meaning of the codes was largely educated guessing. Grab all known codes several times and compare what differs. Also compare this with what the Windows-interface shows.

This perl-script enumerates all codes that I found. I figured out the meaning of most of them.

# ./solarmax-test.pl /dev/ttyS1 5
Opening serial port /dev/ttyS1
Setting 19200 8N1
Reading all pending bytes
Address (ADR): 5
Type (TYP): 0xBC2
Software version (SWV): 12.3
Date day (DDY): 15
Date month (DMT): 4
Date year (DYR): 9
Time hours (THR): 18
Time minutes (TMI): 11
Error 1, number (E11): 32770
Error 1, day (E1D): 22
Error 1, month (E1M): 3
Error 1, hour (E1h): 18
Error 1, minute (E1m): 57
Error 2, number (E21): 32772
Error 2, day (E2D): 28
Error 2, month (E2M): 3
Error 2, hour (E2h): 14
Error 2, minute (E2m): 33
Error 3, number (E31): 32772
Error 3, day (E3D): 6
Error 3, month (E3M): 4
Error 3, hour (E3h): 16
Error 3, minute (E3m): 35
Operating hours (KHR): 9953
Energy today [Wh] (KDY): 12900
Energy yesterday [Wh] (KLD): 14200
Energy this month [kWh] (KMT): 154
Energy last monh [kWh] (KLM): 248
Energy this year [kWh] (KYR): 558
Energy last year [kWh] (KLY): 2703
Energy total [kWh] (KT0): 6356
Language (LAN): 1
DC voltage [mV] (UDC): 344700
AC voltage [mV] (UL1): 239400
DC current [mA] (IDC): 330
AC current [mA] (IL1): 520
AC power [mW] (PAC): 89000
Power installed [mW] (PIN): 2580000
AC power [%] (PRL): 3
??? Start ups ??? (CAC): 8004
??? (FRD): 0x7D60905
??? (SCD): 0x3C
??? (SE1): 0x0
??? (SE2): 0x0
??? (SPR): 0x1
#

The result

For those interested: the above graph (with the exception of the ‘Showing from [...] to [...]‘ lines) was generated using the following rrdtool command:

rrdtool --imgformat PNG --start "06:00" --end "21:00" --upper-limit 3100 --lower-limit 0 --rigid \
DEF:Pac=/var/lib/rrdtool/solarmax.1.rrd:Pac:AVERAGE \
DEF:Energy=/var/lib/rrdtool/solarmax.2.rrd:energy:AVERAGE \
VDEF:Pac_cur=Pac,LAST \
VDEF:Pac_avg=Pac,AVERAGE \
VDEF:Pac_peak=Pac,MAXIMUM \
CDEF:Pac_Wh=Pac,3600,/ \
VDEF:Eac=Pac_Wh,TOTAL \
VDEF:Eac2=Energy,TOTAL \
"AREA:Pac#0000ff:Pac    " \
"GPRINT:Pac_cur:%6.2lf%sW" \
"COMMENT:\\n" \
"HRULE:Pac_avg#bbbbff:Average" \
"GPRINT:Pac_avg:%6.2lf%sW" \
"COMMENT:\\n" \
"HRULE:Pac_peak#ff0000:Peak   " \
"GPRINT:Pac_peak:%6.2lf%sW" \
"COMMENT:\\n" \
"GPRINT:Eac:int(Pac)   %6.2lf%sWh\\n" \
"GPRINT:Eac2:meas       %6.2lf%sWh\\n" \
--vertical-label "Watt" --title "Power"

The total energy for the day is calculated in two ways: ‘int(Pac)’ uses the sampled power-gauge to calculate the total; ‘meas’ uses the energy-counter of the inverter. The latter one should be more accurate, but can’t be read out with as much precision.

Update 2009-10-09

Chris has made his own variant of the above Perl script:

Hi Niobos,

here we go… find attached to this mail my modification to
your script (a very helpful piece of code .

Since i am not the perl-crack i’ve not yet implemented the
command “SYS” and the translation of the returned value
into a readable Operation-State.

At the moment i don’t know how to convert the returned
value i.e. of “SYS=4E24,0″ which is 20004,0 in decimal
and means an “Operation at MPP”.

ThX…
Chris