分类： LINUX
2010-06-01 10:13:25
Summary: Despite the most resilient plans, systems are prone to failure. This article gives you some guidance for system failures, including where to look and how to interpret the problems, and offers some answers on fixes, all within the VMware ESX framework.
The VMware ESX server allows several instances of similar or dissimilar operating systems to run as virtual machines on a single server, so consolidating application workloads is simple and fast. But despite the best, most comprehensive plans, systems can fail.
To aid in troubleshooting, you can categorize problems when a VMware ESX server fails in several ways, depending on the failure. The most common method is to split the categories into a four-way matrix with server and virtual machine problems on one axis and network and storage on the other axis.
Additionally, another broad area of potential trouble is the MUI (Management User Interface), which occasionally encounters problems.
When a failure occurs, the first step in diagnosing it is to collect diagnostic data -- once you collect it, you can analyze that data for the cause of the failure. The next few sections show you how to collect the data, where to look for information, and how to interpret it.
The first critical piece of data to collect is the output file generated from the /usr/bin/vm-support script. This created file is placed in the current directory and is called esx-XXXX-XX-XX.XXXX.tgz (replace the Xs with date/pid information: for example, esx-2005-01-04.27059.tgz).
The /usr/bin/vm-support script is regularly updated by VMware. To collect the most accurate information, download and install the latest version. In addition, if you are experiencing problems with VirtualCenter, the VirtualCenter logs need to be collected (diagnosis on this problem is beyond the scope of this article). For the latest versions of each of these, see Resources.
Once collected, you can transfer vm-support output (in binary mode) to the appropriate support personnel for diagnosis. To extract the file on a Linux-based system, issue the following command: tar zxvf esx-XXXX-XX-XX.XXXX.tgz.
Let's take a system-wide look at how the hardware on the system is configured and allocated. You can do this by either using command-line utilities or looking at the output from the /usr/bin/vm-support file.
The output from the /usr/bin/vm-support gzipped tar file contains the following directories once extracted:
etc/ proc/ root/ tmp/ usr/ var/ and possibly 'home' or 'vpx' depending on the location of the .vmx config files. |
To get a total picture of the layout of the ESX server, start in the tmp directory. The PCI bus devices on the system are in the file tmp/lspci.1.*.txt. You can get the same output from running the /sbin/lspci command. Listing 2 shows an example output listing.
# /sbin/lspci 00:00.0 Host bridge: ServerWorks: Unknown device 0014 (rev 33) 00:00.1 Host bridge: ServerWorks: Unknown device 0014 00:00.2 Host bridge: ServerWorks: Unknown device 0014 00:01.0 VGA compatible controller: ATI Technologies Inc Radeon VE QY 00:0f.0 ISA bridge: ServerWorks: Unknown device 0203 (rev a0) 00:0f.1 IDE interface: ServerWorks: Unknown device 0213 (rev a0) 00:0f.2 USB Controller: ServerWorks: Unknown device 0221 (rev 05) 00:0f.3 Host bridge: ServerWorks: Unknown device 0227 00:10.0 Host bridge: ServerWorks: Unknown device 0101 (rev 05) 00:10.2 Host bridge: ServerWorks: Unknown device 0101 (rev 05) 01:01.0 Ethernet controller: Intel Corporation 8254NXX Gigabit Ethernet Controller (rev 04) 01:02.0 Ethernet controller: Intel Corporation 8254NXX Gigabit Ethernet Controller (rev 04) 02:01.0 Ethernet controller: Intel Corporation 8254NXX Gigabit Ethernet Controller (rev 03) 02:01.1 Ethernet controller: Intel Corporation 8254NXX Gigabit Ethernet Controller (rev 03) 02:03.0 Fiber Channel: QLogic Corp QLA231x/2340 (rev 02) 02:03.1 Fiber Channel: QLogic Corp QLA231x/2340 (rev 02) |
Listing 2 gives you a compendium of all the PCI controllers in the machine. The left column tells you the bus, slot, and function of the card. For instance, the Ethernet controller with specification 02:01.0 tells you that the card is in bus #02, slot #01, with function #0. Because there is another Ethernet controller with the same bus and slot number (and a different function), this is a two-port controller.
Now that we know what is in the machine, we need to see which of these resources are allocated to the console and which ones are allocated to the virtual machines. For this, take a look at the tmp/vmkchdev.*.txt file or use the command /usr/sbin/vmkchdev whose output is shown in Listing 3.
Listing 3. Output from vmkchdev
# /usr/sbin/vmkchdev -L 000:00.0 1166:0014 0000:0000 console PCI device 1166:0014 (ServerWorks) 000:00.1 1166:0014 0000:0000 console PCI device 1166:0014 (ServerWorks) 000:00.2 1166:0014 0000:0000 console PCI device 1166:0014 (ServerWorks) 000:01.0 1002:5159 8086:34b1 console PCI device 1002:5159 (ATI Technologies Inc) 000:15.0 1166:0203 8086:34b1 console PCI device 1166:0203 (ServerWorks) 000:15.1 1166:0213 8086:34b1 console PCI device 1166:0213 (ServerWorks) 000:15.2 1166:0221 8086:34b1 console PCI device 1166:0221 (ServerWorks) 000:15.3 1166:0227 8086:34b1 console PCI device 1166:0227 (ServerWorks) 000:16.0 1166:0101 0000:0000 console PCI device 1166:0101 (ServerWorks) 000:16.2 1166:0101 0000:0000 console PCI device 1166:0101 (ServerWorks) 001:01.0 8086:1028 8086:34b1 vmkernel vmnic3 PCI device 8086:1028 (Intel Corporation) 001:02.0 8086:1028 8086:34b1 vmkernel vmnic0 PCI device 8086:1028 (Intel Corporation) 002:01.0 8086:107b 8086:34b1 vmkernel vmnic1 PCI device 8086:107b (Intel Corporation) 002:01.1 8086:107b 8086:34b1 vmkernel vmnic2 PCI device 8086:107b (Intel Corporation) 002:03.0 1077:2312 1014:027d vmkernel vmhba0 PCI device 1077:2312 (Q Logic) 002:03.1 1077:2312 1014:027d vmkernel vmhba1 PCI device 1077:2312 (Q Logic) |
This output tells you which devices are allocated to the vmkernel and which ones the console owns. Items that say console are allocated to the console OS; all others are allocated to the virtual machines.
You can match devices with the lspci output in the left column. You can also find some of the same information in the etc/vmware/hwconfig file. The hwconfig file also tells you which devices are shared between the console and the virtual machines.
Once you know what cards are in the machine and how they are allocated, you need to ensure the right drivers are being loaded. In the tmp/modules.*.txt file, you can see which drivers the console OS is using (see Listing 4). You can obtain the same output with the /sbin/lsmod command.
# /sbin/lsmod Module Size Used by Tainted: PF vmxnet_console 13212 1 vmnixmod 177056 3 [vmxnet_console] e1000 68456 0 (unused) usb-storage 20028 0 mousedev 3936 0 (unused) keybdev 1696 0 (unused) hid 17728 0 (unused) input 3488 0 [mousedev keybdev hid] usb-ohci 17600 0 (unused) usbcore 50112 1 [usb-storage hid usb-ohci] |
You can find parameter settings for the modules loaded in the console OS in the /etc/modules.conf file.
You also need to ensure the right modules are loaded for the virtual machines. This information is kept in the tmp/vmkmod.*.txt file (see Listing 5). You can also find this information with the /usr/sbin/vmkload command.
# /usr/sbin/vmkload_mod -l Name R/O Addr Length R/W Addr Length ID Loaded vmklinux 0x4de000 0xf000 0x12438f8 0x53000 1 Yes nfshaper 0x4ed000 0x1000 0x129b160 0x1000 2 Yes e1000 0x4ee000 0xf000 0x129c168 0x6000 3 Yes qla2300_604 0x4fd000 0x19000 0x12fe008 0x22000 4 Yes bond 0x516000 0x2000 0x1574b80 0x2000 5 Yes |
To see what options are passed to the modules for the virtual machines, you need to look in the file tmp/vmkpcidivy.vmkmod.*.txt (see Listing 6).
vmklinux linux nfshaper.o nfshaper e1000.o nic qla2300_604.o fc qlloop_down_time=90 qlport_down_retry=10 |
Various parameters for the fibre storage may be required based on the type of storage connected. You will need to ensure you have the right settings recommended by the storage vendor.
To see the options available for a given module, you can again use the /usr/sbin/vmkload_mod command (see Listing 7).
# vmkload_mod -s mptscsi Using /usr/lib/vmware/vmkmod/mptscsi.o mptscsih string PortIo int, description "[0]=Use mmap, 1=Use port io" |
Many of the system storage problems are caused by misconfiguration or by problems external to the ESX server. You can solve most of the FAStT misconfigurations by reading the IBM Redbook Implementing VMware ESX Server 2.1 with IBM TotalStorage FAStT (see Resources for a link).
Another cause of storage issues can be compatibility problems. Following the System, I/O, SAN, and Backup Compatabilty Guides (see Resources for a link) can help you solve these problems.
Once it is set up correctly, the configuration should look similar to Figure 1.
In most cases, you should see four paths to each LUN (logical unit number, which is useful if you're running applications in your virtual machines that need visibility to the physical characteristics of the storage device), except for local LUNs and those in which failover is not critical. You can view confirmation of a typical layout in the tmp/vmkmultipath.*.txt file or by issuing the vmkmultipath command (seeListing 8).
# /usr/sbin/vmkmultipath -q
Disk and multipath information follows:
Disk vmhba0:0:0 (225,278 MB) has 4 paths. Policy is mru.
vmhba0:0:0 on (active, preferred)
vmhba0:1:0 on
vmhba1:0:0 on
vmhba1:1:0 on
|
If the active path is not the same as the preferred, most likely there is a cabling, zoning, or hardware problem. Typical messages for those kind of failures in the var/log/vmkernel can include those shown in Listing 9.
Jan 6 10:21:36 vmware01-ss vmkernel: 0:00:00:57.966 cpu1:132) WARNING: SCSI: 2046: Manual switchover to path vmhba3:0:5 begins. Jan 6 10:21:36 vmware01-ss vmkernel: 0:00:00:57.966 cpu1:132) SCSI: 2050: Changing active path to vmhba3:0:5 Jan 6 10:21:36 vmware01-ss vmkernel: 0:00:00:57.967 cpu1:132) WARNING: SCSI: 1683: Did not switchover to vmhba3:0:5. Check Unit Ready Command returned READY instead of NOT READY for standby controller . Jan 6 10:21:36 vmware01-ss vmkernel: 0:00:00:57.967 cpu1:132) WARNING: SCSI: 2089: Manual switchover to vmhba3:0:5 completed successfully. |
Once you understand which network devices are allocated to the virtual machines, you must next understand how they are being used. For this information, you need to look at etc/vmware/hwconfig and etc/vmware/netmap.conf. The netmap.conf file tells you the name of the internal ESX switches and what devices they are using (as shown in Listing 10).
# cat netmap.conf network0.name = "External Linux Net" network0.device = "vmnic0" network1.name = "Internal Windows Net" network1.device = "vmnet_0" network2.name = "Public Web access" network2.device = "bond0" |
This example tells you there are two virtual switches on the server, their names (.name), and which devices they are using (.device). If the device is not a real device (vmnic or bond), then no outbound adapters are assigned to the virtual switch.
The etc/vmware/hwconfig file contains other valuable information if the outbound adapter is a bond (see Listing 11).
nicteam.vmnic0.team = "bond0" nicteam.vmnic1.team = "bond0" |
These two nics are bonded together as a single NIC.
You can look in several places for diagnostic information on the network. The first location is in proc/vmware/net. You should see subdirectories for each NIC and bond on the system. An example from proc/vmware/net/vmnic0/config is shown in Listing 12.
VLanHwTxAccel Yes VLanHwRxAccel Yes VLanSwTagging Yes PromiscuousAllowed No InterruptClustering No Link state: Up Speed: 1000 Mbps, full duplex Queue: Running PCI (bus:slot.func): 1:0.0 Minimum Capabilities 0x0 Device Capabilities 0x74b Maximum Capabilities 0x76b NICTeamingMaster: bond0 TeamFailoverBeacon: Off Interrupt vector 0x69 DebugSocket Closed |
This tells you that the link is up and available. A down state would indicate a possible bad cable or network switch port. Within this same directory, you can see a stats file that lists various network statistics.
Another source of information is var/log/vmkernel and var/log/messages. If you are seeing a lot of the following types of messages (see Listing 13), the NICs may be having trouble negotiating speed/duplex with the switch.
Feb 2 12:48:23 SYNVM7 kernel: bcm5700: eth0 NIC Link is DOWN Feb 2 12:48:23 SYNVM7 kernel: bcm5700: eth0 NIC Link is UP, 1000 Mbps full duplex |
In some cases (particularly with Broadcom NICs and Cisco switches), you may have to hardcode the speed/duplex on the card and on the switch end. This issue is supposed to be resolved with ESX V2.5, and customers should set this up as auto-negotiate in this release. Do the hardcoding using the MUI for the virtual machines and in /etc/modules.conf for the console OS Ethernet.
Another source of corroboration for network issues is in proc/net/nicinfo. Files in this directory contain failure counts for issues on the network. Non-zero lines for any of the statistics other than Rx_Packets, Tx_Packets, Rx_Bytes, or Tx_Bytes indicate problems on the external network or possibly with the NIC itself.
If there are disk issues that appear to be limited to a single virtual machine, the first place to look is in the log file for the virtual machine. These are in the same location as the .vmx files: etc/vmare/vm-list.
Looking at one such type of failure in a .vmx file, you might see the following:
Feb 02 11:39:07: vmx| DiskVmnixSetupSCSIDevices: failed to get handle for SCSI Device 0 Feb 02 11:39:07: vmx| Msg_Post: Error Feb 02 11:39:07: vmx| [msg.diskVmnix.scsiopenfailed] Unable to open scsi target VMFS-SAN1:mywindowsmachine.vmdk: Device or resource busy(2) Feb 02 11:39:07: vmx| [msg.vmxlsilogic.poweronFailed] Feb 02 11:39:07: vmx| Failed to configure scsi0. Feb 02 11:39:07: vmx| ---------------------------------------- Feb 02 11:39:07: vmx| POST(no connection): Unable to open scsi target VMFS-SAN1: mywindowsmachine.vmdk: Device or resource busy(2) Feb 02 11:39:07: vmx| Feb 02 11:39:07: vmx| Failed to configure scsi0. Feb 02 11:39:07: vmx| Feb 02 11:39:07: vmx| Module VmxLSILogic power on failed. |
This can indicate that the storage device has a problem or that some other virtual machine is using the disk file. You can look at the process table listing to determine who might be using a given file. For instance, you might see the following from "ps -efwww" (or from the tmp/ps.*.txt file):
root 2751 0.0 1.7 404252 6612 ? S< 12:29 0:00 vmware-mks -A 11 -D 13 -S -L /tmp/vmware-root-2750.log -P 2750 -g -@ vm=374709cf368cf239; gui=false; vmdbMemMapHandle=0x4; vmdbMemMapSize=0x400000; useSELinux=false -C /home/vmware/mywindowsmachine/mywindowsmachine.vmx |
If a process is hanging around that is holding onto the file, it will prevent the virtual machine from being able to get the lock on the file. If the storage is shared, you need to make sure the virtual machine isn't running on another ESX server.
Usually virtual machine networking issues are related to misconfiguration of the system or possibly a speed/duplex issue. For example, a network problem may exist on one of the NICs within a bond the virtual machine is on.
On occasion, you may see problems with the driver within the virtual machine. A reinstall of the VMware tools fixes this problem. You can test this by switching from a vlance to a vmxnet driver or vice versa in the MUI.
Another issue may arise if a NIC is removed or added to the system. This can lead to device slippage (for instance, if vmnic1 becomes vmnic0). In this case, you should edit the configuration for the VM to ensure that it is pointing to the correct vmnic or bond and is on the right network. Another method of troubleshooting this problem is to change the NIC to the console OS, bring up the interface, and ensure you can ping other IPs on that network.
In the event of a sudden system shutdown or panic, the MUI may have trouble starting when booting up because the MUI doesn't have the proper time to clean up leftover lock files when a system fails.
Other MUI failures can include the MUI dying or problems where it gets out of sync with the command line. The quickest resolution is to try restarting the MUI. This will not affect the virtual machines in any way. To do this, issue the following service command (see Listing 16):
# service httpd.vmware restart Shutting down http.vmware: [ OK ] Starting httpd.vmware: [ OK ] |
Intermittent problems with the MUI may be related to speed/duplex problems on the console OS. Check the settings in /etc/modules.conf.
As a last resort, you can reinstall the MUI. The rpm exists on the installation CD and is typically of the form VMware-mui-2.1.2-9638.rpm (the version should match the version of VMware on the system -- version 9638 may differ depending on the release of ESX being run). You can remove the old MUI with rpm -e VMware-mui-2.1.2-9638 and install the new one with rpm -i VMware-mui-2.1.2-9638.rpm.
The techniques and examples in this article should get you started in locating and resolving problems with your VMware system: the systems storage and network, the virtual machine storage and network, and the MUI.
Greg Lindley is the team lead for VMware ESX product support within IBM and has been supporting the VMware product for the better part of two years. In addition, he has been supporting Linux for more than five years and UNIX for more than fifteen. You can contact Greg at .
Chinese version（中文版本）： http://www.ibm.com/developerworks/cn/linux/l-vmware/index.html
