This document explains the configurations and procedures to enable Ethernet
over IP tunneling on FreeBSD. I succesfully performed the test on a FreeBSD
4.10-RELEASE. I make no claim that it will work on other releases. I also tested
it on FreeBSD 4.7-RELEASE.
The objective of this set-up is to join two Ethernet segments S1 and S2 through
the Internet and bridges, as if S1 and S2 were one single Ethernet segment.
The situation can arrise where both segments are too far away to be connected
by a cable (or even optic fiber). Segments need not to be connected to Internet,
though an Internet connection should be available at close distance.
The solution consist of encapsulating Ethernet frames into IP packets, those
packets being transmitted through Internet and decapsulated at the other end.
This kind of implementation is called tunnelling Ethernet over IP.
Tunnelling Ethernet over IP
In this setting, we use a software bridge that runs on FreeBSD.
- i386 (Pentium III)
- 32 MB RAM
- 2 NICs (RealTek, 3Com, etc.)
- Operating system: FreeBSD 4.10
- generic kernel
- no IP forwarding
- kernel security low
- Modules (default modules)
- vtun (installed from the ports collection)
In the following part, we will use the following conventions:
- vtun is a client server protocol, we will call the server vtun-server
and the client vtun-client
- each box has 2 network interfaces of type 3Com 905
- the interface xl0 is called the outside interface (0 is close to
o-utside), it's the interface that is connected to the Internet
- the interface xl1 is called the inside interface (1 looks like
i-nside), it is the interface that connects to the ethernet segment we want
- Prepare two FreeBSD boxes, vtun-server and vtun-client.
Assign an IP to the outside interface of the machines. Being the server, vtun-server
will probably have a static IP, while vtun-client can do with a dynamic
It is a good idea to install sudo from the security packages (from CD).
For those who like it, install emacs from the editor packages (from CD).
The file /etc/rc.conf will look like:
syslogd_flags="-ss" # Flags to syslogd (if enabled).
ifconfig_xl0="inet 22.214.171.124 netmask 255.255.255.248"
defaultrouter="126.96.36.199" # cs router
syslogd_flags="-ss" # Flags to syslogd (if enabled).
- We disabled inetd, sendmail (totally), the USB ports
- We have syslogd running without any socket
- We bring up the inside interface
- Install vtun on both machines, from the ports collection.
make && make install
To save some network bandwidth, it is a good practice to install vtun
on one machine first, then to copy /usr/ports/distfiles from that
machine to the other before making vtun. That way, the distributions
will not be downloaded again from the Internet.
- Edit vtun configuration on the machines.
Comment some lines of /usr/local/etc/vtund.conf in the lion
session part. The differences would be like below:
The modifications of file /usr/local/etc/vtund.conf
vtun-server%diff vtund.conf vtund.conf.orig
< # ifconfig "%% 10.1.0.1 netmask 255.255.255.0";
> ifconfig "%% 10.1.0.1 netmask 255.255.255.0";
< # route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2";
> route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2";
< # firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";
> firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";
< # firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";
> firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";
On vtun-server modify line 423, change the IP address that will be used by the
tunnel end to avoid duplication of IP address.
The IP address should read 10.1.0.3
The modifications of file /usr/local/etc/vtund.conf on vtun-server
< ifconfig "%% 10.1.0.3 netmask 255.255.255.0";
> ifconfig "%% 10.1.0.2 netmask 255.255.255.0";
- Configure the boot loader to automatically load the necessary modules at
Once the kernel has been set into a secure mode, some modules will not load anymore, so it is safe to load them since the boot
On both machines, edit /boot/loader.conf to look like:
The /boot/loader.conf file
# -- sysinstall generated deltas -- #
- Install the netgraph bridge on both machines.
cp /usr/share/examples/netgrap/ether.bridge .
chmod u+x ether.bridge
Then edit the file /etc/ether.bridge to include the lines:
The modifications of file /etc/ether.bridge
- Install the starter for /etc/ether.bridge on both machines.
The netgraph bridge can only be started when the interface tap1 is up. And the interface tap1
comes up only when a vtun connection has been established. To avoid manual starting of netgraph bridge
the following script will test for the interface tap1 and then only start the bridge.
Copy that script in /etc/start-bridge, make it executable:
chmod u+x /etc/start-bridge
The script /etc/start-bridge
res=`/sbin/ifconfig tap1 2>&1`;
while [ "$res" = "ifconfig: interface tap1 does not exist" ]; do
# echo tap1 is down
res=`/sbin/ifconfig tap1 2>&1`;
# echo $res
#echo tap1 is up
Note that this script will also put the kernel in its higher security mode. It is valid because at that
point the tunnel is up and working.
- Edit the /etc/rc.local on both machines.
On vtun-server it will start vtun in server mode and wait for connections, it will also initiate the
The file /etc/rc.local on vtun-server
/usr/local/sbin/vtund -f /usr/local/etc/vtund.conf -s
On vtun-client it connects the the vtun server mode and initiates the
The file /etc/rc.local on vtun-client
/usr/local/sbin/vtund -p -f /usr/local/etc/vtund.conf lion 188.8.131.52
Note we use the option -p so vtun client automatically reconnects to the server,
whenever the server goes down or the connectivity is lost. This option allows to start the client before the server;
the connection will be made as soon as the server is ready and visible.
184.108.40.206 is the IP address
- Reboot both machines.
Once all the cables are connected, the tunnel should be established automatically.
- Make sure the tunnel is connected.
Some tests to be sure the tunnel is working
vtun-server45: ifconfig tap1
tap1: flags=8943 mtu 1500
inet 10.1.0.2 netmask 0xffffff00 broadcast 10.1.0.255
inet6 fe80::2bd:85ff:fe02:1%tap1 prefixlen 64 scopeid 0x8
Opened by PID 116
vtun-server46: netstat -a |grep ESTABLISHED
tcp4 0 0 220.127.116.11.commplex- 18.104.22.168.2800 ESTABLISHED
vtun-server47: grep promiscuous /var/log/messages
Aug 6 13:23:46 vtun-server /kernel: tap1: promiscuous mode enabled
Aug 6 13:23:46 vtun-server /kernel: xl1: promiscuous mode enabled
vtun-server48: sysctl -a|grep secure
vtun-server49: ps auwx|grep vtun
root 116 0.0 2.3 2128 1376 ?? S< 4:54PM 0:15.17
vtund: lion ether tap1 (vtund)
root 97 0.0 1.9 2128 1096 ?? Is 4:54PM 0:00.00
vtund: waiting for connections on port 5000 (vtund)
- The command ifconfig shows that the interface tap1 is up and running.
- The command netstat shows that there is an established vtun connection; vtun port is commplex-main or
- A look at /var/log/messages shows that the interfaces xl1 and tap1 are in
promiscuous mode, netgraph bridge has started.
- In a same way, the kernel has been changed to extreme security mode kern.securelevel=2.
This is done only after netgraph bridge has started.
- Lastly, ps shows the vtun server waiting and one vtun connection ongoing. On
the vtun-client machine, we will not see vtun waiting for connections.
- Check that packets are crossing the tunnel.
To do that, we attach one end of the tunnel to the segment it should tunnel and we check that we see some traffic at the other end of the tunnel.
In our example (see below) we attached the inside interface
of vtun-server to the hub H1, then on vtun-client,
we check that we receive packed on the interface xl1:
Check the tunnel with tcpdump on vtun-client
vtun-client37: sudo tcpdump -i xl0
tcpdump: WARNING: xl0: no IPv4 address assigned
tcpdump: listening on xl1
13:02:29.936925 sfc-imlsdds.ai3.net.1580 > 22.214.171.124.56952: udp 1408
13:02:29.964519 sfc-imlsdds.ai3.net.1580 > 126.96.36.199.56952: udp 1408
13:02:29.982184 sfc-imlsdds.ai3.net.1580 > 188.8.131.52.56952: udp 1408
13:02:30.013232 sfc-imlsdds.ai3.net.1580 > 184.108.40.206.56952: udp 1408
13:02:30.028988 sfc-imlsdds.ai3.net.1580 > 220.127.116.11.56952: udp 1408
13:02:30.053034 sfc-imlsdds.ai3.net.1580 > 18.104.22.168.56952: udp 1408
13:02:30.078648 sfc-imlsdds.ai3.net.1580 > 22.214.171.124.56952: udp 1408
- Setting up security.
Security was not considered in this framework, but IP firewall should be
used to restrict access to the vtun server. Similarily, access to the
vtun client should be restricted. Lastly, encryption shold be enabled on vtun tunnel.
Setting for the workshop
Tunnel topology for SOI Asia workshop
The tunnel is established between the hubs H1 and H2.
H1 is directly connected to the UDLR receiver Sony Box, while H2
is in the classroom. The machine connected to H2 gets a dynamic IP
address so it becomes the vtun client, while the machine connected
to H1 gets a static IP and is configured as vtun-server.
Change since Frebruary 2003 setting is with the outside interface
of vtun-server. It is not connected to AI3 subnet anymore, but directly to the
border of AI3 and AIT AS's. That way we save one hop (and one IP from AI3 subnet).