XenNetworking

XenNetworking

目录

Virtual Ethernet interfaces

Xen creates, by default, seven pair of "connected virtual ethernet interfaces" for use by dom0. Think of them as two ethernet interfaces connected by an internal crossover ethernet cable. veth0 is connected to vif0.0, veth1 is connected to vif0.1, etc, up to veth7 -> vif0.7. You can use them by configuring IP and MAC addresses on the veth# end, then attaching the vif0.# end to a bridge.

Diagram of Physical and Logical network cards:

Every time you create a running domU instance, it is assigned a new domain id number. You don't get to pick the number, sorry. The first domU will be id #1. The second one started will be #2, even if #1 isn't running anymore.

For each new domU, Xen creates a new pair of "connected virtual ethernet interfaces", with one end in domU and the other in dom0. For linux domU's, the device name it sees is named eth0. The other end of that virtual ethernet interface pair exists within dom0 as interface vif.0. For example, domU #5's eth0 is attached to vif5.0. If you create multiple network interfaces for a domU, it's ends will be eth0, eth1, etc, whereas the dom0 end will be vif.0, vif.1, etc.

Logical network cards connected between dom0 and dom1:

When a domU is shutdown, the virtual ethernet interfaces for it are deleted.

MAC addresses

Virtualised network interfaces in domains are given Ethernet MAC addresses. By default xend will select a random address, this will differ between instantiations of the domain. If it is required to have a fixed MAC address for a domain (e.g. for using with DHCP) then this can be configured using the mac= option to the vif configuration directive (e.g. vif = ['mac=aa:00:00:00:00:11']).

When choosing MAC addresses to use, ensure you choose a unicast address. That is, one with the low bit of the first octet set to zero. For example, an address starting aa: is OK but ab: is not. It is best to keep to the range of addresses declared to be "locally assigned" (rather than allocated globally to hardware vendors). These have the second lowest bit set to one in the first octet. For example, aa: is OK, a8: isn't.

In summary, an address of the following form should be OK:

XY:XX:XX:XX:XX:XX

where X is any hexadecimal digit, and Y is one of 2, 6, A or E.

It's recommended to use a MAC address inside the range 00:16:3e:xx:xx:xx. This address range is reserved for use by Xen.

Bridging

Illustration on network-bridge and vif-bridge:

The default Xen configuration uses bridging within domain 0 to allow all domains to appear on the network as individual hosts. If extensive use of iptables is made in domain 0 (e.g. a firewall) then this can affect bridging because bridged packets pass through the PREROUTING, FORWARD and POSTROUTING iptables chains. This means that packets being bridged between guest domains and the external network will need to be permitted to pass those chains. The most likely problem is the FORWARD chain being configured to DROP or REJECT packets (this is different from IP forwarding in the kernel).

iptable FORWARDing can be disabled for all packets; to prevent the dom0 from acting as an IP router: echo 0 > /proc/sys/net/ipv4/ip_forward.

A slightly more secure method is to allowing packet forwarding (at the iptables level) between the external physical interface and the vifs for the guests. For a machine with a single ethernet card this would be:

iptables -A FORWARD -m physdev --physdev-in eth0 --physdev-out '!' eth0  -j ACCEPT
iptables -A FORWARD -m physdev --physdev-out eth0 --physdev-in '!' eth0  -j ACCEPT

(needs the ipt_physdev [aka xt_physdev] module to be available).

The ebtables project has an interesting document on the interaction of bridging and iptables.

Packet flow in bridging

()

Packet arrives at hardware, is handled by dom0 Ethernet driver and appears on peth0. peth0 is bound to to the bridge, so its passed to the bridge from there. This step is run on Ethernet level, no IP addresses are set on peth0 or bridge.

Now the bridge distributes the packet, just like a switch would. Filtering at this stage would be possible with .

Now there's a number of vifX.Y connected to the bridge, it decides where to put the packet based on the receiver's MAC.

The vif interface puts the packet into Xen, which then puts the packet back to the domain the vif leads to (its also done that way for dom0, hence the vif0.0->(v)eth0 pair).

The target device in the dom0/domU finally has an IP address, you can apply iptables filtering here.

network-bridge

When xend starts up, it runs the network-bridge script, which:

1. creates a new bridge named xenbr0

2. "real" ethernet interface eth0 is brought down

3. the IP and MAC addresses of eth0 are copied to virtual network interface veth0

4. real interface eth0 is renamed peth0

5. virtual interface veth0 is renamed eth0

6. peth0 and vif0.0 are attached to bridge xenbr0

7. the bridge, peth0, eth0 and vif0.0 are brought up

It is good to have the physical interface and the dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does not affect the traffic to the domUs (just for protecting dom0 alone).

vif-bridge

When a domU starts up, xend (running in dom0) runs the vif-bridge script, which:

1. attaches vif.0 to xenbr0

2. vif.0 is brought up

Additional Notes

• you can change the bridge name from xenbr0 using:

  (network-script 'network-bridge bridge=mybridge')
  

  in xend-config.sxp and rebooting or restarting xend

• remember to configure the bridge to attach to in the domU's config file using:

  vif=[ 'bridge=mybridge' ]
  

  or perhaps something like:

  vif=[ 'mac=00:16:3e:01:01:01,bridge=mybridge' ]
  

• you can create multiple network interfaces, and attach them to different bridges using:

  vif=[ 'mac=00:16:3e:70:01:01,bridge=br0', 'mac=00:16:3e:70:02:01,bridge=br1' ]
  

• if you want to use multiple bridges, you must create them yourself, either manually, or via your own startup script, or via a custom script to replace network-bridge. For example:

  $ cd /etc/xen/scripts
  $ cp network-bridge network-custom
  $ cp vif-bridge vif-custom
  $ vi /etc/xen/xend-config.sxp
          (network-script network-custom)
          (vif-script vif-custom)
  $ vi network-custom
          # whatever you want
  

• before you connect a physical interface to a bridge, remember to reset it's mac and turn arp off. For example:

  # ip link set eth1 down
  # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off
  # brctl addif br1 eth1
  # ip link set eth1 up
  

• With Xen 3.0 the best method for additional bridges is to use the default Xen scripts with a slight modification. Following the . For example in a two bridge network with eth0 and eth1. Create /etc/xen/scripts/my-network-script with

  #!/bin/sh
  dir=$(dirname "$0")
  "$dir/network-bridge" "$@" vifnum=0
  "$dir/network-bridge" "$@" vifnum=1
  

• With Xen 3.2.1 (tested on Debian Etch 4.0r3), here is a script example that creates two virtual interfaces corresponding to the 2 physical network interfaces

  # xemacs /etc/xen/scripts/network-bridge-wrapper
  #!/bin/sh
  /etc/xen/scripts/network-bridge $1 netdev=eth0
  /etc/xen/scripts/network-bridge $1 netdev=eth1
  

The $1 will use the argument of xend (in the /etc/xen/xend-config.sxp configuration file). If there is a default physical network interface, the standard network-bridge script of Xen will create a vif for this interface, and not the other ones also.

• (Additional note by steve_from_moreover - May be stating the obvious but remember to do - chmod 755 /etc/xen/scripts/my-network-script or when you reboot it will silently not be able to run this script).
• On SuSE Linux (at least), each interface requires an ifcfg script in /etc/sysconfig/network, e.g. /etc/sysconfig/network/ifcfg-eth1. Otherwise, network-bridge will create the bridge with no interfaces attached.
• Then change /etc/xen/xend-config.sxp with the following (network-script my-network-script).
• The same principle can apply to networks without a physical ethernet device. Use a dummy interface with

  "$dir/network-bridge" "$@" vifnum=2 netdev=dummy0
  

Links

Some relevant topics from the mailing list:

• eth0 IP in dom0 2005/01/14

• Bridging vs. Routing 2005/01/13

• Bridging vs. Routing 2004/07/18

• An attempt to explain Xen networking

• (with bridging)

• (with bridging)

• You can also have a look to:

Routing

This section applies only if you choose to use network-route and vif-route instead of network-bridge and vif-bridge.

Illustration on network-route and vif-route:

Routing creates a point-to-point link between dom0 and each domU. Routes to each domU are added to dom0's routing table, so domU must have a known (static) IP. DHCP doesn't work, because the route won't be created, and the DHCP offer won't arrive.

network-route

When xend starts up, it runs network-route which:

1. enables ip forwarding within dom0

vif-route

When domU starts up, xend runs (within dom0) vif-route which:

1. copies the ip address from eth0 to vif.0

2. brings up vif.0

3. adds host static route for domU's ip address specified in domU config file, pointing at interface vif.0

Virtual Network

The virtual network is currently a non-standard configuration.

Illustration of virtual network setups:

The virtual network configuration places all domU on a shared virtual network with dom0. This allows domU to use a DHCP server provided by dom0, without allowing DHCP requests from domU to escape onto the physical network. (As far as I can tell, vif0.0 and dummy0 are not strictly required.)

Links

• My (sapphirecat) virtual network and DHCP setup:

• Another virtual network setup:

Interface names

The default configuration for Xen systems is to use bridging. When xend starts it creates a bridge called xen-br0. xend takes the IP address etc. from eth0 and assigns to xen-br0 (as dom0's interface onto the bridge). So dom0's own external-facing interface is now xen-br0 - make sure any firewall config reflects this.

VLANs

Multiple tagged VLANs can be supported by configuring 802.1Q VLAN support into domain 0. A local interface in dom0 is needed for each desired VLAN although it need not have an IP address in dom0. A bridge can be set up for each VLAN, and guests can then connect the the appropriate bridge.

My () prefered method is to define the bridge as an interface which is not brought up automatically (e.g. for Debian /etc/network/interfaces, with no "auto" entry):

iface xen-br293 inet manual
    up vconfig add eth0 293
    up /etc/xen/scripts/network start netdev=eth0.293 bridge=xen-br293 antispoof=no
    up /sbin/ifconfig eth0.293 up
    down /etc/xen/scripts/network stop netdev=eth0.293 bridge=xen-br293 antispoof=no
    down vconfig rem eth0.293

I then add an init.d script to bring the interface up between xend and xendomains starting.

Yet Anther ASCII Graphics Description of Xen Networking

Xen 3.1- Networking

Notes:

• vif0.0 is absolute different from vif1.0. vif0.0 is created by netloop, while vif1.0 is created by netback.

Xen 3.2+ Networking

Notes:

• eth0 and eth1 are the bridge names, which is a bit confusing.

Alternative Xen Networking Architecture

Notes:

• based on xen 3.3, change bridge name to xenbr0, xenbr1, ...
• xenbrX has an active address, which is used by dom0 to communicate with outside.

Xen Networking with vlan

Notes:

• With this configuration, are completely unaware of the fact that they are utilizing a VLAN, all the work is done within the bridges in Dom0.

• Dom0 is aware of the traffic within the VLAN, because it has an active address on the xenbrX interfaces. To prevent it, don't give the xenbrX an active address, but configure a extra interface for management.
• There are two things may need to be configured:

  • If your ethernet card does not natively support VLAN tags, you will have to set the maximum MTU to 1496 to make room for the tag. With command:

     # ifconfig eth0 mtu 1496
    

  • With the bridged to VLAN interfaces, some optimizations need to be disabled or tcp and udp connections will fail. This is done by disabling transmit checksum offloading:

     # ethtool -K eth0 tx off
    

• Need further test in the production environment.

Xen Networking with bonding

Notes:

• Need some more destructive testing.

Xen Networking with vlan on bonding

Reference

注：该文章的译文为http://www.cublog.cn/u2/72217/showart_1154322.html