Since we currently already have an OpenVPN tunnel, I preferred to stay with OpenVPN rather than switch to a different stack. Turns out OpenVPN supports IPv6, but under a series of assumptions. Among others, you need OpenVPN 2.3 on both the client and the server to be able to use the new config directives such as route-ipv6. At this moment, 2.3 is still beta. While I enjoy beta-software on a daily basis, rolling it out as a production VPN seems a bit to risky.

To provide maximal compatibility, I decided to stay with OpenVPN 2.2 and use the tap-interface. I wrote my own up-scripts to take care of the required IPv6 configuration steps.

Communicating the parameters

OpenVPN has a setenv config directive, which sets additional environment variables for the up-script. Its brother setenv-safe (which prefixes all variables with OPENVPN_) can also be pushed from server to client, allowing arbitrary parameters to be communicated from server to client. I use these three:

push "setenv-safe IP6_PREFIX 2001:db8:0:1::/64"
push "setenv-safe IP6_GW 2001:db8:0:1::1"
push "setenv-safe IP6_ROUTES '2001:db8:0:0::/48 2001:db8:1234::/48'"

Configuring the client

The client now has all the information it needs to enable IPv6 on the tunnel-connection. It just needs to apply that knowledge. I wrote some very basic shell scripts to accomplish this. Basically, the script looks for the MAC-address of the (virtual) VPN interface, which it uses to form a SLAAC address together with the supplied prefix. At this moment, it does not check for duplicate IP addressing. Next, it configures the calculated IP onto the interface and adds the supplied routes to the routing table.

For those interested, I provide, without any warranty <insert legalese disclamer here>

• The MacOSX version, written in bash
• The Linux version, written in bash
• The Windows version, a batch-file and powershell hacked together

For the most part, this major changeover happened without as much of a hitch. In fact, if I hadn’t known it was World IPv6 day, I wouldn’t have noticed anything. But I’m not a normal web-user, so I did notice some issues.

Where it did went wrong

After some troubleshooting, they all boiled down to a single cause of oversight. They were not bugs or issues with IPv6 per se, just some “expected behavior” that we didn’t anticipate: IPv4-only VPNs.

Most servers in our datacenter are not publicly accessible; none of them are manageable over the public internet. In order to connect to them, you need a VPN connection. This serves multiple purposes: it secures all communication between client and server (so even plain-text http can be used securely to manage servers), it limits the number of users with access and most importantly (in the IPv4 world) it allows us to use RFC1918 addresses internally and still get the routing to work out. Technically it behaves an an extra (virtual) network card with a (virtual) cable connected straight to the datacenter. Additionally, some routes are configured automatically on the client to make sure traffic to the servers is sent over this “cable”.

We use two kinds of VPN-connections, but none of them was IPv6 enabled (i.e. could carry IPv6 data through the tunnel). Since by default client software prefers IPv6 connections, this caused the IPv6-internet connection to be preferred above the IPv4-VPN connection. Obviously, the firewall at the datacenter didn’t agree and refused access.

The solution was fairly obvious to state (enable IPv6 through the tunnels) but difficult to implement. In fact, I have not been able to get it to work well enough to install it on someone else’s computer.

The attempts

IPsec in transport mode

The “natural” solution would be to secure the IPv6 packets with IPsec, preferably in transport mode, between the client and the firewall. Since there are no NAT-issues, tunnel mode is not required.

However, I was not able to get this to work, even in manual keying mode (i.e. without ISAKMP). I couldn’t get setkey to accept the src-dst parameter in the SPD:

# setkey -c
spdadd 2001:db8:0:1::1 2001:db8:1:1::3 any -P fwd ipsec esp/transport/2001:db8:1:0:2-2001:db8:1:1::3/require;
^D
# setkey -DP
2001:db8:0:1::1[any] 2001:db8:1:1::3[any] any
 fwd prio def ipsec
 esp/transport//require
 created: Jun 14 12:13:53 2011  lastused:                    
 lifetime: 0(s) validtime: 0(s)
 spid=1641 seq=1 pid=10485
 refcnt=1

This seems to be a Linux issue (Ubuntu 10.04 LTS with kernel 2.6.32-28-generic and ipsec-tools 0.7.1), since this does work on MacOSX (10.6.7).

IPsec tunnel mode

Since I’m not entirely sure that what I tried above (transport mode) is even supposed to work, I also tried tunnel mode. This worked, but is a pain to configure. I only tried manual keying, but using racoon to do username/password authentication will be even harder to explain to users…

The Mac built-in VPN client only supports “Cisco IPsec“. This uses the mode configuration stage to communicate the set of “networks” to tunnel (i.e. the SPD). However, according to racoon.conf man-page, I can only push IPv4 networks in the split_network directive.

OpenVPN with tun driver

According to the OpenVPN FAQ, IPv6 is only supported if the underlying TUN-driver supports it. The tuntaposx-page does not mention IPv6 at all and hasn’t been updated for almost 2 years, so this seems unlikely to work.

Also, OpenVPN does not provide configuration directives to push IPv6 routes over from server to client.

OpenVPN with tap driver

Even when using the TAP-driver and router advertisements, MacOSX does not seem to like enabling IPv6… Even after manually enabling it, MacOSX still doesn’t pick up its SLAAC address:

# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 ether 7e:95:80:00:90:0e
 inet 192.0.2.10 netmask 0xffffff00 broadcast 10.90.9.255
 open (pid 3847)

# ip6config start-v6 tap0
Starting IPv6 on tap0.

# sleep 60 # Wait for Router advertisement to show up

# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 ether 7e:95:80:00:90:0e
 inet 192.0.2.10 netmask 0xffffff00 broadcast 10.90.9.255
 inet6 fe80::7c95:80ff:fe00:900e%tap0 prefixlen 64 tentative scopeid 0xa
 open (pid 3847)

And this still doesn’t allow me to push IPv6 routes to the clients upon connecting.

The conclusion

IPv6 is very stable and capable, but there are certain network-issues where there is still some work to do. If you happen to know a VPN-solution which supports IPv6 and works on Windows, linux and Mac, please let me know!

Edit: I worked out my own solution.

So I wrote a very simple perl-script that automates this for me. It behaves like cat, but inserts empty lines between input lines proportional to the amount of time between them. The numbers of lines inserted in logarithmically proportional to the elapsed time: one line for the first second, a second line for the next two seconds, a third line for the next 4 seconds, …

$ tail -f /var/log/mail.log | logtail.pl | sed 's/ .*//'
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
2011-03-09T10:33:02+01:00
 
 
 
 
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
2011-03-09T10:33:36+01:00
 
 
 
 
2011-03-09T10:34:01+01:00
2011-03-09T10:34:01+01:00
 
 
 
 
 
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
2011-03-09T10:34:36+01:00
 
 
 
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00
2011-03-09T10:34:48+01:00

I chose to leave smtpd_client_restrictions, smtpd_helo_restrictions and smtpd_sender_restrictions blank and do all the checks in smtpd_recipient_restrictions. While it’s possible to reject messages earlier, this setup gives more info in the logs for rejected messages. These are the filters that I apply.

First, the local originated mail is allowed:

• permit_mynetworks: Allow local originated mail

Mail servers introduce themselves with their hostname. RFC 5321 requires that all hostnames are fully qualified.

• reject_non_fqdn_hostname: Non fully qualified hostnames names are rejected.
• reject_invalid_hostname: Hostnames which have an invalid syntax are rejected as well.

The next phase in the SMTP conversation is identifying the sender. SMTP has an “envelope sender”. This is the address where bounces are returned to. Usually, this is the same as the “From” field, but this is not required. If my mailserver is to accept responsibility to deliver the mail, it should have a way to contact the sender. If the sender address is not usable, we can’t bounce if needed. Don’t accept responsibility in this case.

• reject_non_fqdn_sender
• reject_unknown_sender_domain

The same applies for the recipients:

• reject_non_fqdn_recipient
• reject_unknown_recipient_domain

The next statement is very important: don’t become an open relay. Only accept mail for the domains you’re actually responsible for:

• reject_unauth_destination

The previous tests were local and fast. The mail-server can verify them with minimal effort. But this is not enough to fight spam. Greylisting in particular is very effective. Greylisting will cause every mail to be initially rejected with a temporary failure. RFC 5321 requires sending mail-servers to retry this delivery. On a second attempt, the message is accepted.

• check_policy_service inet:127.0.0.1:10023: The greylisting server listens on this port and keeps the database of seen mails

SPF is another anti-spam technique. This verifies that the sending mail-server is actually allowed to send mail from this email-address. However, this causes problems with standard forwarding of emails, so I don’t use it to reject messages, but I do log the result

• warn_if_reject, check_policy_service unix:private/policy-spf

As a final test, several blacklists are checked. If the sending mail-server is listed as a known spammer, the mail is rejected.

• reject_rbl_client bl.spamcop.net
• reject_rbl_client sbl-xbl.spamhaus.org
• reject_rbl_client dnsbl.sorbs.net

All previous configuration will either accept the mail for delivery, or reject the mail. It will not silently drop mail, which is a very important thing in my opinion. If you really want to, you can chain spamassassin to the end.

echo 3 > /proc/sys/vm/drop_caches

Writing 1 only clears the pagecache; 2 clears the dentries and inodes; 3 clears all.

I tried to configure TBF on my Ubuntu 9.10 VMware box, but I got disappointing results: I only got around 100kbps, no matter what I configured. I tried the same setup on a physical server, with the same results.

After a full afternoon of experimenting and Googling, I was able to solve the issue by disabling offloading:

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

# ethtool -K eth0 sg off
# ethtool -K eth0 tso off
# ethtool -K eth0 gso off

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off

$command | \
perl -pe '@now=localtime();printf "%04d-%02d-%02dT%02d:%02d:%02d ",$now[5]+1900,$now[4]+1,$now[3],$now[2],$now[1],$now[0];'

The perl command reads in every line, prints the current time in the default format (or in whatever format you specify), followed by the read line.

When using SSL-secured connections, this doesn’t work anymore. The problem is similar to the situation in HTTP1.0: The server needs to know to which SSL-host the connection is addressed. SNI introduces a similar solution: It specifies an extension to the SSL negotiation to indicate which server the client wishes to access.

This extension opens up a small security hole: Since you send the server the expected hostname before the SSL negotiation starts, the server could, theoretically, generate a matching certificate on-the-fly. This certificate will probably be rejected by your browser, since it won’t be signed with a trusted root certificate. Another consideration is that the hostname you are connecting to is sent in plain text, before the secure channel is set up. However, a reverse lookup of the destination IP will usually return the same information.

Since SNI is an extension, support isn’t automatic. Ubuntu 9.10 (Karmic) supports SNI in apache out of the box. Paul Kehrer’s blog has the full story, but it’s as simple as it seems:

NameVirtualHost *:443
<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    DocumentRoot /my/doc/root
    ServerName mydomain.com
    SSLEngine On
    SSLCertificateFile /path/to/domain.crt
    SSLCertificateKeyFile /path/to/domain.key
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    DocumentRoot /my/doc/root
    ServerName mydomain2.com
    SSLEngine On
    SSLCertificateFile /path/to/domain2.crt
    SSLCertificateKeyFile /path/to/domain2.key
</VirtualHost>

On the browser side, Wikipedia has a list of compatible browsers. Most notably:

• IE7 under XP does NOT support SNI
• Mobile Safari on iPhone (3.0.2) does NOT support SNI

Non-SNI compatible browsers will get the SSL certificate of the first vhost, which is probably wrong. You can require SNI-support by setting

SSLStrictSNIVHostCheck on

This will return a 403 to non-compliant browsers.

Update 2010-07-01

I just upgraded to iOS 4, which seems to have SNI-support (and IPv6 support as well)!

Obviously, I should use tee to send the output to both less and the file, but I regularly forget this. That where this hint comes in: you can save the current less-buffer to file!

In short, to save the buffer that is being displayed by a session of `less’, use its pipe-to-shell-command capability by scrolling to the top of the file and press `|’ followed by `$’ as well as entering `tee DESTINATION_FILE’ when prompted for the shell command.

• BASEDIR=/whatever
• cp $DEBs $BASEDIR/.
• mkdir -p $BASEDIR/dists/$DIST/$COMPONENT/binary-amd64
• cd $BASEDIR
• apt-ftparchive packages . > dists/$DIST/$COMPONENT/binary-amd64/Packages
• echo “deb file:$BASEDIR $DIST $COMPONENT” >> /etc/apt/sources.list
• apt-get update

$BASEDIR is whatever directory you have space in (preferably an empty directory); $DIST and $COMPONENT are “jaunty-backports” and “main” in my setup, but you can vary.

You also might need to add the following line to /etc/apt/apt.conf to get rid of the “untrusted source” warning:

APT::Get::AllowUnauthenticated “true”;