Since I first write my first attempt at trying to get VLAN support working under Xen, I’ve received some reports for people stating that it doesn’t work as expected. And they are right.
At the end of the first article, I pointed out I was having problems with UDP traffic. In turn, it was worse than I ever expected, since it was affecting DNS name resolution, DHCP services and other services running as inside a domainU. This is the reason why I rethought the implementation and now have it working on a production machine acting, among as other things, as a DHCP server and DNS server.
In this second try I decided not to mess around with Xen’s default
network configuration, so please undo all the changes you did so you end
end up with a pristine Xen configuration. In this new scenario all the
native traffic (tagged an untagged Ethernet frames) is being captured by
Xen’s switch, xenbr0, and sent to the right network
interface. If the traffic being received is a 802.11q tagged frame, the
target will receive it tagged and thus will have to implement measures
to untag and process it accordingly.
Introduction
So, let’s say we have the following logical network topology and virtual machines:
|
LAN
|
-------------------+----------------------------------
| | |
| peth0 ---- xen-br0 |
| | |
| ----------------------------- |
| | | |
| vif0.0 vif1.0 |
| | | |
| | +--------------+------------
| | | |
| | | ------------+------------
| | | | | |
| eth0 | | eth0 |
| | | | | |
| -------+------- | | ------+------- |
| | | | | | | |
| eth0.1000 eth0.10 | | eth0.2000 eth0.10 |
| | | | | | | |
| VLAN 1000 VLAN 10 | | VLAN 2000 VLAN 10 |
| | | | | | | |
| www ssh | | ftp ssh |
| | | |
| Domain0 | | DomainU |
--------------------------- -------------------------
The Xen’s switch configuration can be seen with the following command:
root@xen:~# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no peth0
vif0.0
vif1.0
For each domain — this includes domain0 or any domainU — there is a vif|X|.|Y| interface attached to Xen’s bridge xen-br0, where |X|
is the domain ID (0 for domain0 and a monotonically increasing number
for every domainU). Then, we have every network interface card inside
the domain, in the form of eth|Y|. Thus, if a domainU with ID #3 defines two network interfaces, eth0 and eth1, there will two corresponding virtual network interfaces in domain0, named vif3.0 and vif3.1.
Instead of trying to export VLAN interfaces to one or more domainUs, we export the whole, native (tagged or not) network interface to the domainU and, inside this domainU, we can configure VLAN subinterfaces if needed.
Sample scenario
Let’s say we want to offer the following services per VLAN:
- WWW server on VLAN 1000
- FTP server on VLAN 2000
- SSH access to administer the WWW sever, reachable only through the VLAN 10
- SSH server to administer the FTP server, reachable only through the VLAN 10
But we also want to partition the physical machine in two, so domain0 serves WWW traffic while domainU servers FTP traffic:
| WWW | FTP | SSH | |
|---|---|---|---|
| domain0 | VLAN 1000 | - | VLAN 10 |
| domainU | - | VLAN 2000 | VLAN 10 |
Thus, we need the following VLAN subinterfaces:
eth0.10andeth0.1000on domain0eth0.10andeth0.2000on domainU
Configuring VLAN subinterfaces in domainU is straight forward. However, it’s a little bit more difficult for domain0.
Configuring VLAN subinterfaces for domain0
First of all, make sure you are using bridging for your Xen configuration. Make sure the following line is uncommented in /etc/xen/xend-config.sxp:
(network-script network-bridge)
And comment any other network-script configuration lines, like:
(network-script network-nat)
or
(network-script network-route)
It seems we can’t bring up VLAN subinterfaces before Xen’s network
script is fired up since Xen’s network scripts perform some black magic
on the network interfaces, mainly renaming eth0 to peth0 and bringing up a dummy interface named eth0. Any subinterface related to the original eth0 seems to stop working after the renaming takes place.
Thus, I coded up an init script used to bring up the VLAN subinterfaces that gets invoked just after Xen’s network script has finished. Note that it’s targeted for RedHat-based distributions:
#!/bin/sh
#
# Init file for Network-VLAN
# STARTS AFTER XEN (which is S50 and K01)
#
# chkconfig: 2345 51 89
# description: VLAN networking
. /etc/init.d/functions
case "$1" in
start)
echo -n $"Configuring VLAN interfaces:"
if [ ! -f /var/lock/subsys/network-vlan ]; then
(
modprobe 8021q || exit 1
vconfig add eth0 10 || exit 2
ifconfig eth0.10 up 10.0.0.1 netmask 255.0.0.0 || exit 3
vconfig add eth0 1000 || exit 2
ifconfig eth0.1000 up 11.0.0.1 netmask 255.0.0.0 || exit 3
) > /dev/null 2>&1
RETVAL=$?
[ "$RETVAL" = 0 ] && ( success ;\\
touch /var/lock/subsys/network-vlan ) || failure
fi
echo
;;
stop)
echo -n $"Unconfiguring VLAN interfaces:"
if [ -f /var/lock/subsys/network-vlan ]; then
(
ifconfig eth0.10 down && vconfig rem eth0.10 ;
ifconfig eth0.1000 down && vconfig rem eth0.1000
) > /dev/null 2>&1
RETVAL=$?
[ "$RETVAL" = 0 ] && ( rm -f /var/lock/subsys/network-vlan ;\\
success ) || failure
fi
echo
esac
Save this script as /etc/init.d/network-vlan, then run:
chmod +x /etc/init.d/network-vlan
chkconfig --add /etc/init.d/network-vlan
The script runs just after Xen’s init script has renamed the real
Ethernet interface and has brought up a dummy interface called eth0. Then, the network-vlan
script brings up two VLAN subinterfaces, one for VLAN 10 and another
one for VLAN 1000, and then assigns each one its own IP address.
Additionally, these are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
Note that eth0 in this context refers to the real
Ethernet interface, since Xen’s init script has not been ran yet. I
didn’t configure any IP address for this interface since I only want to
process tagged traffic. Beware that on many switches — i.e., Cisco 2960
and 3560 —, VLAN1 is, by default, the native VLAN and traffic on the
native VLAN doesn’t get tagged.
Configuring VLAN subinterfaces for domainU
These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
I didn’t configure any IP address for this interface since I only want to process tagged traffic. Read the note above on untagged frames and native VLANs.
These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.10:
DEVICE=eth0.10
BOOTPROTO=static
IPADDR=10.0.0.2
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes
These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.2000:
DEVICE=eth0.2000
BOOTPROTO=static
IPADDR=12.0.0.1
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes
Bonding
For those who desire to use bonding, it seems some tweaking of the networking scripts is required. I recommend them to look at this post on .
Conclusion
I’m sure there are better ways to configure VLAN subinterfaces in domain0, but it was in a hurry and couldn’t find of a better way to get it done.
If anyone out there has a different way of achieving this, please let me know 