For the most part, this major changeover happened without as much of a hitch. In fact, if I hadn’t known it was World IPv6 day, I wouldn’t have noticed anything. But I’m not a normal web-user, so I did notice some issues.

Where it did went wrong

After some troubleshooting, they all boiled down to a single cause of oversight. They were not bugs or issues with IPv6 per se, just some “expected behavior” that we didn’t anticipate: IPv4-only VPNs.

Most servers in our datacenter are not publicly accessible; none of them are manageable over the public internet. In order to connect to them, you need a VPN connection. This serves multiple purposes: it secures all communication between client and server (so even plain-text http can be used securely to manage servers), it limits the number of users with access and most importantly (in the IPv4 world) it allows us to use RFC1918 addresses internally and still get the routing to work out. Technically it behaves an an extra (virtual) network card with a (virtual) cable connected straight to the datacenter. Additionally, some routes are configured automatically on the client to make sure traffic to the servers is sent over this “cable”.

We use two kinds of VPN-connections, but none of them was IPv6 enabled (i.e. could carry IPv6 data through the tunnel). Since by default client software prefers IPv6 connections, this caused the IPv6-internet connection to be preferred above the IPv4-VPN connection. Obviously, the firewall at the datacenter didn’t agree and refused access.

The solution was fairly obvious to state (enable IPv6 through the tunnels) but difficult to implement. In fact, I have not been able to get it to work well enough to install it on someone else’s computer.

The attempts

IPsec in transport mode

The “natural” solution would be to secure the IPv6 packets with IPsec, preferably in transport mode, between the client and the firewall. Since there are no NAT-issues, tunnel mode is not required.

However, I was not able to get this to work, even in manual keying mode (i.e. without ISAKMP). I couldn’t get setkey to accept the src-dst parameter in the SPD:

# setkey -c
spdadd 2001:db8:0:1::1 2001:db8:1:1::3 any -P fwd ipsec esp/transport/2001:db8:1:0:2-2001:db8:1:1::3/require;
^D
# setkey -DP
2001:db8:0:1::1[any] 2001:db8:1:1::3[any] any
 fwd prio def ipsec
 esp/transport//require
 created: Jun 14 12:13:53 2011  lastused:                    
 lifetime: 0(s) validtime: 0(s)
 spid=1641 seq=1 pid=10485
 refcnt=1

This seems to be a Linux issue (Ubuntu 10.04 LTS with kernel 2.6.32-28-generic and ipsec-tools 0.7.1), since this does work on MacOSX (10.6.7).

IPsec tunnel mode

Since I’m not entirely sure that what I tried above (transport mode) is even supposed to work, I also tried tunnel mode. This worked, but is a pain to configure. I only tried manual keying, but using racoon to do username/password authentication will be even harder to explain to users…

The Mac built-in VPN client only supports “Cisco IPsec“. This uses the mode configuration stage to communicate the set of “networks” to tunnel (i.e. the SPD). However, according to racoon.conf man-page, I can only push IPv4 networks in the split_network directive.

OpenVPN with tun driver

According to the OpenVPN FAQ, IPv6 is only supported if the underlying TUN-driver supports it. The tuntaposx-page does not mention IPv6 at all and hasn’t been updated for almost 2 years, so this seems unlikely to work.

Also, OpenVPN does not provide configuration directives to push IPv6 routes over from server to client.

OpenVPN with tap driver

Even when using the TAP-driver and router advertisements, MacOSX does not seem to like enabling IPv6… Even after manually enabling it, MacOSX still doesn’t pick up its SLAAC address:

# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 ether 7e:95:80:00:90:0e
 inet 192.0.2.10 netmask 0xffffff00 broadcast 10.90.9.255
 open (pid 3847)

# ip6config start-v6 tap0
Starting IPv6 on tap0.

# sleep 60 # Wait for Router advertisement to show up

# ifconfig tap0
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 ether 7e:95:80:00:90:0e
 inet 192.0.2.10 netmask 0xffffff00 broadcast 10.90.9.255
 inet6 fe80::7c95:80ff:fe00:900e%tap0 prefixlen 64 tentative scopeid 0xa
 open (pid 3847)

And this still doesn’t allow me to push IPv6 routes to the clients upon connecting.

The conclusion

IPv6 is very stable and capable, but there are certain network-issues where there is still some work to do. If you happen to know a VPN-solution which supports IPv6 and works on Windows, linux and Mac, please let me know!

Edit: I worked out my own solution.

So I wanted my own router to establish the PPPoE connection and obtain a public IPv4 address. It seems that even here the default setup of the b-box2 is good: PPPoE passthrough is enabled by default. This technique allows you to establish a PPPoE connection from the “LAN”-side of the router, effectively bypassing it. There are numerous posts how to reconfigure the b-box into RFC 1483 bridge-mode, but this is not required! So just plugging in my router and configuring PPPoE on it was enough!

There are some finishing touches however that can be learned from the bridge-posts:

• I disabled my PPPoE connection on the b-box itself. I will not be using it, so there is no need to waste a public IPv4 address here.
  In the webinterface (http://192.168.1.1/ by default) go to Advanced Settings – Network Interfaces. Open the Wan PPPoE and Disable it, confirm by clicking OK.
• I also disabled the built-in WiFi access point, since I have my own 802.11n access point right next to it. This is even officially documented.
  Advanced Settings – Wireless and click Deactivate.

The default Linksys firmware is actually not bad, but dd-wrt just offers a ton more features: Multiple SSIDs, IPv6 support (including Sixxs tunneling), WAN volume monitoring, custom firewalling, traffic shaping, … So I decided to void my warranty and put my router on steroids! Mandatory note: this may very well turn your router in to a very expensive brick.

The initial flash

After reading through the dd-wrt forum (most notably these three posts) and the wiki page, I learned a few things:

• Apparently, this router has its reset button wired to the wrong GPIO pin. Therefor, the 30/30/30 reset DOES NOT WORK on this router! There is an alternative: use at least version 13493, power down the router, push and hold the WPS button (on top), power up the router, hold the WPS button for 10-12 more seconds, then release.
• The latest recommended firmware is BrainSlayer’s 14929

This is the procedure I followed, with success, starting from Linksys version v1.0.03 (build 010Jul 24, 2009):

1. Download the tailored build for the WRT320N (for the freaks, my binary MD5s to e1d7edd368bf5259c18a0874c5e761db).
2. Connect via wired ethernet to the router. That way, you can see the link going up/down.
3. In the Linksys firmware, upload this file.
4. Wait 5 very long minutes.
5. Configure yourself a static IP in the 192.168.0.0/24 network (I use 192.168.0.8)
6. Direct your browser to http://192.168.0.1/
7. Set a temporary password
8. Wait 1 minute
9. Reset the router: Power down, push & hold WPS button, power up, keep holding for 11 seconds, release.
10. Close & reopen your browser to flush all cached pages and credentials
11. Direct your browser to http://192.168.0.1/
12. Enjoy

The upgrade

After the initial flash, you can upgrade to any regular version, but keep in mind that this unit requires a 2.6 kernel. I choose the 14929-mini version (md5 af9ab2ff822ab69d26fa7308d47ad05a), not because it provided all I need (it doesn’t support IPv6 for example), but because it leaves the most free space for me to fiddle with.

To switch versions, I always follow this overly cautious procedure:

1. Reset to defaults: power down, push & hold WPS button, power up, keep holding for 11 seconds, release
2. Make sure your IP is in the correct range (192.168.0.0/24)
3. Set a temporary password
4. Upload the new firmware
5. Wait until the browser is again at the “Set password” page
6. Set temporary password
7. Reset to defaults again

The settings

Most configuration is fairly straightforward with the GUI. But setting up a second, routed SSID needed a little non-intuitive work:

• Go to Wireless -> Basic Settings and add a new Virtual Interface
  • Leave Network Configuration to Bridged
• Go to Setup -> Networking
  • Under Create Bridge, click Add
  • Name the new bridge “br1″ and disable STP
  • Apply Settings
  • Now add the desired IP and Subnet mask for this brigde-port
  • Apply Settings again
  • Click Add under Assign to bridge
  • Now assign the wl0.1 interface to this newly created bridge br1
• Optionally Add a DHCP range for br1. In this case, you need to use DNSmasq as DHCP-server.

Some guides tell you to configure it in Unbridged mode. Using Bridged mode gives the potential advantage that you can link a wired port to this IP-range easily.

Now you can easily firewall between the two WLANs by putting iptables-lines in the startup script.

I tried to configure TBF on my Ubuntu 9.10 VMware box, but I got disappointing results: I only got around 100kbps, no matter what I configured. I tried the same setup on a physical server, with the same results.

After a full afternoon of experimenting and Googling, I was able to solve the issue by disabling offloading:

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

# ethtool -K eth0 sg off
# ethtool -K eth0 tso off
# ethtool -K eth0 gso off

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off

100BASE-TX requires UTP cables of Cat5 or higher and utilizes 1 pair for Tx and 1 pair for Rx.

Note that by using the 4B5B encoding, it is no longer necessary to define a bit-order. The transmit-bit-order is specified in the 4B5B code.

The start of a new frame is indicated by a preamble, just as in 10BASE-T. The preamble used is slightly different: 0xJK 55 55 55 55 55 D5. J and K are special symbols that have their own 5bit-symbol in the 4B5B coding. By using a separate symbol for control purposes, it becomes easier to recover from errors: a data symbol can never be misunderstood as a control symbol or the other way around.

Similarly, the end of a frame is indicated with 0xTR. T and R also have a special 5-bit symbol.

Bits to scrambled bits

To comply to the FCC regulations, the resulting bitstream is scrambled. This operation gives the bit-stream more desirable characteristics, especially for cross-talk.

The scrambler used is a simple linear feedback shift register, 11bits long, with taps at the 11th and 9th bit (x¹¹+x⁹+1). This outputs a 2047bit long pseudo-random bit sequence. For those interested, this textfile shows the output.

The scrambling process itself is a traditional XOR operation: the 4B5B output is XORed bit-by-bit with the scrambler output.

Note that 100BASE-FX (over fiber) does not scramble its bits prior to transmission.

Idle

A 10BASE-T connection is silent between frames (apart from a LIT pulse); 100BASE-T in contrast sends IDLE-symbols. An Idle-symbol is a special 5-bit code (0b11111) different from all data-symbols. Note that this symbol is also scrambled.

A receiver can use this information to synchronize its descrambler: Since the output bit-stream are all 1s, the scrambled output contains the raw scramble-code, inverted. A receiver simply needs to copy and invert 11 bits into its LSFR-scrambler to synchronize.

Bits to voltage

The (scrambled) bit-stream is encoded with a MLT-3 encoding. This Multi-Level Transmit method uses 3 voltage levels to encode a bit, thereby reducing the required bandwidth.

It acquires this goal by cycling through 4 states: 0V, +1V, 0V, -1V. A 1 input-bit changes the output-state; a 0 input-bit results in no output change. The highest attainable frequency is thus 31.25MHz (remember that there are 125M symbol bits to transmit each second after 4B5B encoding).

Additional things

This presentation contains a nice overview of 100BASE-T.

Auto-negotiate allows a device to negotiate its speed and duplex settings with the other side. This is especially useful for multi-speed devices such as 10/100-base-T adapters. The previous spec for auto-negotiate was open for interpretation, leading to incompatible devices. Since 1998, the debatable section was removed from the IEEE 802.3 standard.

Auto-negotiate (ANeg) works by sending a series of short pulses, +1V high, 100ns long. This signal is chosen to stay compatible with 10BASE-T’s LIT (see my post on 10BASE-T for details). Instead of a single pulse, ANeg sends a series of pulses, called the Fast Link Pulse (FLP).

These pulses encode a 16-bit word, which indicates the capabilities of the transmitting device: 10 and/or 100 Mbps, half and/or full duplex. Note that it is normal to support all lower standards as well, however this is not a requirement: A device can advertise that it only supports 100BASE-TX – Full Duplex. This is not the same as setting 100-full-fixed. The former case works in 100-full, but communicates this information to the other side of the link; the latter works in 100-full, but does not inform the other side.

Analysis

The addition of a scrambler significantly increases the unpredictability. This scrambler is not synchronized to frame boundaries, not even to frame bytes. This makes it essentially impossible to construct a frame with a known wire-level signal. Still, I gave it a try:

• If the 4B5B encoded output of the frame matches the output of the scrambler, the resulting (scrambled) bitstream will be a stream of 0′s. MLT-3 encodes this as “no transition”. This will give the receiver-clock a hard time.
  After some searching, the bytes 0xB9 20 98 6F C3 result in 50 bits of symbols that match a part of the scrambling-code. Since the scrambler is not synchronized, you need a bit of luck to allign them.
  By filling a 1280B frame with this bit-pattern, you have 256 chances to get them alligned: a little more than 12.5% chance.

Note that, although I try to be fairly correct in this explanation, I’m still cutting some corners. The one and only true explanation is in the full IEEE802.3 document.

Bytes to bits

The first step in this process is called serialization. Computers usually work with a bunch of bits at a time (typically 32 or 64). While it is possible to transmit a bunch of bits at the same time as well, 10base-T only transmits a single bit at a time. Thus, we need to convert a series of bytes into a series of bits. The 10base-T standard sends out the bits of each byte with the least-significant bit first: i.e. when transmitting 0×33 over the wire, the first signal will be a 1.

To notify the other end of the line that a frame is about to be sent, every transmission starts with a preamble. In 10base-T this is a series of 64 bits, alternating 1 and 0, ending with two consecutive 1s: 0×55 55 55 55 55 55 55 D5. This last bytes is also called the start-of-frame indicator.

After the preamble is sent, the receiver has synchronized its clock to the incomming bitstream and is ready to accept the data. This data always starts with destination MAC address and ends with the FCS.

Bits to voltage

This bitstream needs to be converted into voltages and currents. Also, the other side should be able to determine where one bit ends and where the next bits starts. 10base-T uses Manchester encoding to mark the beginning and ending of bits. The Manchester code converts every bit into a transition: a 1-bit is a transition from low to high; the 0-bit is a transition from high to low. This ensures that the clocking-circuit on the receiving end will be able to see every bit. Note that this encoding may introduce extra transitions: two consecutive 1-bits need an additional high-to-low transition in between them.

The transmitter will put a +1V differential signal onto the TD-lines to indicate a high-state; it will put a -1V differential signal onto the lines to indicate a low state. Note that this voltage will be lower on the receiving end due to cable losses.

When the last bit is transmitted, the transmitter will keep the lines in a high state for approximately 2-bit-times (depending on the value of the last bit). After that, the line goes into idle with 0V.

Additional things

To detect the presence of a link partner, 10base-T specifies a Link Integrity Test (LIT) signal; nowadays usually referred to as the Normal Link Pulse (NLP). This is a short, 100ns, positive signal generated every 16±8ms. This is the signal that brings up the green led on most Ethernet ports.

Analysis

The resulting signal is fairly stable in its characteristics:

• It is impossible to introduce a DC-offset since the Manchester encoding delivers an equal amount of 1s and 0s, independent of the data pattern
• The highest frequency is achieved by sending a long run of 0×00 or 0xFF. This results in a 10MHz signal on the line
• The lowest frequency is achieved by sending a long run of alternating bits 0×55. This results in a 5MHz signal on the line

Links

http://www.fpga4fun.com/10BASE-T1.html – FPGA project supporting 10BASE-T

The FCS used by Ethernet is a Cyclic Redunancy Check. The CRC algorithm has some very nice features for this application:

• It is very efficiently implemented in hardware using shift registers
• If sutably designed, it be tuned to detect pretty much every random alteration in the data. Note that the alterations should be random. CRCs are very weak hash functions in the cryptographic sense: it is very easy to generate a message with a predetermined CRC.

How CRCs work

CRCs are much like the remainder of a division. The message is the numerator; the denominator is chosen but fixed number. Here is an example with message “123456789″ and denominator 3:

123456789 / 3 = 41152263 with remainder 0

Transmitted message: 123456789; 0

If the message (or the checksum) somehow got corrupted, the receiver will notice:

Received message:123446789; 0

123446789 / 3 = 41148926 with remainder 2

A real CRC works similar, but has some differences:

• The message is a bunch of bits instead of a number
• The result of the division is not needed, only the remainder is needed; this allows some optimizations in the algorithm
• Computers love to work in binary, not decimal. In binary, you can define another type of division which has very similar properties, but is much faster to calculate

This page (local copy) has a very thorough explanation on what a CRC exactly is, including the different algorithms to calculate them. You can try the different algorithms on this page.

The Ethernet CRC

Ethernet uses a 32 bit long CRC with polynomial 0x04C11DB7.

Depending on your implementation, you can either initialize the CRC-register to 0x46AF6449 and feed in the unmodified data; or you can initialize the register to 0×00000000 and complement the first 32 bits of the data.

Since Ethernet transmits bytes with their least-significant bit first, the CRC uses this same, reflected, order: the first bits of byte 0×33 are two 1s.

After all data-bits are passed through the CRC, push an additional 32 bits of zero’s. This assures that the last 32bits of the message are pushed completely through the register. Finally, complement the content of the register, i.e. XOR it with 0xFFFFFFFF.

The register now contains the CRC in order ff you want to transmit the CRC right away: high polynomial coeficient first. In case you want to add it to your data, remember to reflect each byte from its least-significant-bit-first order to the normal most-significant-bit first order.

This page lists a full ethernet frame, including CRC (correctly reflected from the actual register).

• Security: You don’t want the carrier-network to see your data
• IP conflicts and/or routing issues: You want to carry private IP addresses (eg 10.0.0.0/8 in IPv4) across a public segment (eg the Internet)
• Workarounds: for whatever reason you whised that there was a connection between A and B

Technically, a tunnel consists of taking a bunch of bytes and re-packaging it. The well knows GRE-tunnel takes IP packets and encapsulates them in another IP packet. The outer IP-layer is used to carry the packet across to the other endpoint, where the inner IP-layer reappears. Other tunnels operate at different layers. SSH for example encapsulates TCP segments inside its TCP-connection; SSL-based VPN encapsulate IP packets inside their TLS-session.

Especially in the third case, workarounds, it would be very practical to be able to build a layer 2 tunnel: transport raw ethernet frames, including IEEE802.1q VLAN tags. This is called the L2TP, Layer 2 tunneling protocol. The current version (3) is defined in RFC 3931.

The Cisco website has a complete page on L2TP, in all its variants (Frame relay over MPLS, HDLC over ATM, …). I only wanted to do ethernet over IP. This is the (partial) router configuration I used. Items in bold need to be adapted on the receiving end.

pseudowire-class PW_TEST
 encapsulation l2tpv3
 protocol none
 ip local interface FastEthernet0/1
!
interface FastEthernet0/0
 no ip address
 xconnect 10.0.0.3 1 encapsulation l2tpv3 manual pw-class PW_TEST
  l2tp id 2 3
!
interface FastEthernet0/1
 ip address 10.0.0.2 255.255.255.

This is all that is needed to tunnel every frame of Fa0/0 to 10.0.0.3 (and back). I pinged through this tunnel using both the native VLAN and a tagged vlan. Here is the resulting PCAP file. Wireshark shows the tunneled ethernet frame: