Tunnelling Ethernet over IP for FreeBSD

Tunnelling Ethernet over IP for FreeBSD

This document is largely based upon the work done by Husni.

It only provides some enhancements regarding automatic start-up and some fine details.

This document explains the configurations and procedures to enable Ethernet over IP tunneling on FreeBSD. I succesfully performed the test on a FreeBSD 4.10-RELEASE. I make no claim that it will work on other releases. I also tested it on FreeBSD 4.7-RELEASE.

Objective

The objective of this set-up is to join two Ethernet segments S1 and S2 through the Internet and bridges, as if S1 and S2 were one single Ethernet segment.

The situation can arrise where both segments are too far away to be connected by a cable (or even optic fiber). Segments need not to be connected to Internet, though an Internet connection should be available at close distance.

The solution consist of encapsulating Ethernet frames into IP packets, those packets being transmitted through Internet and decapsulated at the other end. This kind of implementation is called tunnelling Ethernet over IP.

Tunnelling Ethernet over IP

In this setting, we use a software bridge that runs on FreeBSD.

Configuration Environment

Hardware

i386 (Pentium III)
32 MB RAM
2 NICs (RealTek, 3Com, etc.)

Operating system: FreeBSD 4.10

generic kernel
no IP forwarding
kernel security low

Modules (default modules)

netgraph
if_tap
ng_ether
ng_bridge
ng_socket

Software

vtun (installed from the ports collection)

Installation

In the following part, we will use the following conventions:

vtun is a client server protocol, we will call the server vtun-server and the client vtun-client
each box has 2 network interfaces of type 3Com 905
the interface xl0 is called the outside interface (0 is close to o-utside), it's the interface that is connected to the Internet
the interface xl1 is called the inside interface (1 looks like i-nside), it is the interface that connects to the ethernet segment we want to prolongate

Prepare two FreeBSD boxes, vtun-server and vtun-client.
Assign an IP to the outside interface of the machines. Being the server, vtun-server will probably have a static IP, while vtun-client can do with a dynamic IP.
It is a good idea to install sudo from the security packages (from CD).
For those who like it, install emacs from the editor packages (from CD).
The file /etc/rc.conf will look like:

inetd_enable="NO" sendmail_enable="NONE" sshd_enable="YES" usbd_enable="NO" #kern_securelevel="2" kern_securelevel_enable="NO" syslogd_flags="-ss" # Flags to syslogd (if enabled). ifconfig_xl0="inet 203.159.32.5 netmask 255.255.255.248" defaultrouter="203.159.32.3" # cs router ifconfig_xl1="up" hostname="vtun-server.xxx"

on vtun-server

sshd_enable="YES" usbd_enable="NO" sendmail_enable="NONE" inetd_enable="NO" ifconfig_xl0="DHCP" ifconfig_xl1="up" hostname="vtun-client.xxx" #kern_securelevel="2" kern_securelevel_enable="NO" syslogd_flags="-ss" # Flags to syslogd (if enabled).

on vtun-client
Note that:

We disabled inetd, sendmail (totally), the USB ports
We have syslogd running without any socket
We bring up the inside interface

Install vtun on both machines, from the ports collection.
cd /usr/ports/net/vtun
make && make install
To save some network bandwidth, it is a good practice to install vtun on one machine first, then to copy /usr/ports/distfiles from that machine to the other before making vtun. That way, the distributions will not be downloaded again from the Internet.

Edit vtun configuration on the machines.
Comment some lines of /usr/local/etc/vtund.conf in the lion session part. The differences would be like below:

vtun-server%diff vtund.conf vtund.conf.orig 304c304 < # ifconfig "%% 10.1.0.1 netmask 255.255.255.0"; ---<br> > ifconfig "%% 10.1.0.1 netmask 255.255.255.0"; 307c307 < # route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2"; --- > route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2"; 310c310 < # firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ"; --- > firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ"; 321c321 < # firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ"; --- > firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";

The modifications of file /usr/local/etc/vtund.conf

On vtun-server modify line 423, change the IP address that will be used by the tunnel end to avoid duplication of IP address. The IP address should read 10.1.0.3

423c423 < ifconfig "%% 10.1.0.3 netmask 255.255.255.0"; ---<br> > ifconfig "%% 10.1.0.2 netmask 255.255.255.0";

The modifications of file /usr/local/etc/vtund.conf on vtun-server

Configure the boot loader to automatically load the necessary modules at boot time.
Once the kernel has been set into a secure mode, some modules will not load anymore, so it is safe to load them since the boot time.
On both machines, edit /boot/loader.conf to look like:

# -- sysinstall generated deltas -- # userconfig_script_load="YES" netgraph_load="YES" if_tap_load="YES" ng_ether_load="YES" ng_bridge_load="YES" ng_socket_load="YES"

The /boot/loader.conf file

Install the netgraph bridge on both machines.
cd /etc
cp /usr/share/examples/netgrap/ether.bridge .
chmod u+x ether.bridge
Then edit the file /etc/ether.bridge to include the lines:

The modifications of file /etc/ether.bridge

Install the starter for /etc/ether.bridge on both machines.
The netgraph bridge can only be started when the interface tap1 is up. And the interface tap1 comes up only when a vtun connection has been established. To avoid manual starting of netgraph bridge the following script will test for the interface tap1 and then only start the bridge.
Copy that script in /etc/start-bridge, make it executable:
chmod u+x /etc/start-bridge

#!/bin/sh res=`/sbin/ifconfig tap1 2>&1`; while [ "$res" = "ifconfig: interface tap1 does not exist" ]; do # echo tap1 is down res=`/sbin/ifconfig tap1 2>&1`; sleep 1 done # echo $res #echo tap1 is up /etc/ether.bridge start /sbin/sysctl kern.securelevel=2

The script /etc/start-bridge

Note that this script will also put the kernel in its higher security mode. It is valid because at that point the tunnel is up and working.

Edit the /etc/rc.local on both machines.
On vtun-server it will start vtun in server mode and wait for connections, it will also initiate the netgraph bridge:

The file /etc/rc.local on vtun-server
On vtun-client it connects the the vtun server mode and initiates the netgraph bridge:

The file /etc/rc.local on vtun-client
Note we use the option -p so vtun client automatically reconnects to the server, whenever the server goes down or the connectivity is lost. This option allows to start the client before the server; the connection will be made as soon as the server is ready and visible.
203.159.32.5 is the IP address of vtun-server.
Reboot both machines.
Once all the cables are connected, the tunnel should be established automatically.

Make sure the tunnel is connected.

vtun-server<on>45: ifconfig tap1 tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 inet 10.1.0.2 netmask 0xffffff00 broadcast 10.1.0.255 inet6 fe80::2bd:85ff:fe02:1%tap1 prefixlen 64 scopeid 0x8 ether 00:bd:85:02:00:01 Opened by PID 116 vtun-server<on>46: netstat -a |grep ESTABLISHED tcp4 0 0 203.159.32.5.commplex- 203.159.31.7.2800 ESTABLISHED vtun-server<on>47: grep promiscuous /var/log/messages Aug 6 13:23:46 vtun-server /kernel: tap1: promiscuous mode enabled Aug 6 13:23:46 vtun-server /kernel: xl1: promiscuous mode enabled vtun-server<on>48: sysctl -a|grep secure kern.securelevel: 2 vtun-server<on>49: ps auwx|grep vtun root 116 0.0 2.3 2128 1376 ?? S< 4:54PM 0:15.17 vtund: lion ether tap1 (vtund) root 97 0.0 1.9 2128 1096 ?? Is 4:54PM 0:00.00 vtund: waiting for connections on port 5000 (vtund)

Some tests to be sure the tunnel is working

Note:

The command ifconfig shows that the interface tap1 is up and running.
The command netstat shows that there is an established vtun connection; vtun port is commplex-main or port 5000.
A look at /var/log/messages shows that the interfaces xl1 and tap1 are in promiscuous mode, netgraph bridge has started.
In a same way, the kernel has been changed to extreme security mode kern.securelevel=2. This is done only after netgraph bridge has started.
Lastly, ps shows the vtun server waiting and one vtun connection ongoing. On the vtun-client machine, we will not see vtun waiting for connections.

Check that packets are crossing the tunnel.
To do that, we attach one end of the tunnel to the segment it should tunnel and we check that we see some traffic at the other end of the tunnel.
In our example (see below) we attached the inside interface of vtun-server to the hub H1, then on vtun-client, we check that we receive packed on the interface xl1:

vtun-client<on>37: sudo tcpdump -i xl0 Password: tcpdump: WARNING: xl0: no IPv4 address assigned tcpdump: listening on xl1 13:02:29.936925 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:29.964519 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:29.982184 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:30.013232 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:30.028988 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:30.053034 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408 13:02:30.078648 sfc-imlsdds.ai3.net.1580 > 224.227.127.211.56952: udp 1408

Check the tunnel with tcpdump on vtun-client

Setting up security.
Security was not considered in this framework, but IP firewall should be used to restrict access to the vtun server. Similarily, access to the vtun client should be restricted. Lastly, encryption shold be enabled on vtun tunnel.

Setting for the workshop

Tunnel topology for SOI Asia workshop

The tunnel is established between the hubs H1 and H2. H1 is directly connected to the UDLR receiver Sony Box, while H2 is in the classroom. The machine connected to H2 gets a dynamic IP address so it becomes the vtun client, while the machine connected to H1 gets a static IP and is configured as vtun-server.

Change since Frebruary 2003 setting is with the outside interface of vtun-server. It is not connected to AI3 subnet anymore, but directly to the border of AI3 and AIT AS's. That way we save one hop (and one IP from AI3 subnet).