Clock in a Linux Guest Runs More Slowly or Quickly

Clock in a Linux Guest Runs More Slowly or Quickly Than Real Time

Details

Why does the clock in my Linux guest consistently run more slowly or more quickly than real world time?

Solution

Linux guest operating systems keep time by counting timer interrupts. Unpatched 2.4 and earlier kernels program the virtual system timer to request clock interrupts at 100Hz (100 interrupts per second). 2.6 kernels, on the other hand, request interrupts at 1000Hz — ten times as often. Some 2.4 kernels modified by distribution vendors to contain 2.6 features also request 1000Hz interrupts, or in some cases, interrupts at other rates, such as 512Hz.

Furthermore, an SMP-capable Linux kernel requests additional timer interrupts from the virtual local APIC timer. An SMP-capable kernel running on a one-CPU system generates twice as many total timer interrupts as the corresponding UP kernel, while such a kernel running on a two-CPU system requests three times as many. In general, an SMP-capable kernel running on CPUs requests times as many interrupts per second as a UP kernel. For example, an unmodified 2.6 Linux kernel running on a two-CPU virtual machine requests a total of 3000 clock interrupts per second.

When a guest asks for more than 1000 clock interrupts per second, it can be difficult for the virtual machine to keep up, especially if other applications are running on the host at the same time. This can cause the clock in the guest operating system to fall so far behind real time that it is unable to catch up. The overhead of delivering so many virtual clock interrupts can also hurt guest performance and increase host CPU consumption.

It can also be difficult for the guest operating system to field 1000 clock interrupts per second. Even on real hardware, clock interrupts are sometimes lost because the operating system is busy for more than 1 millsecond and another clock interrupt comes in before the previous one was handled. Linux 2.6 contains code to detect such "lost ticks" and correct for them. Unfortunately, this code can trigger the correction spuriously in some cases, resulting in the Linux clock running more quickly than real time. This problem happens more often in a virtual machine than on real hardware, and can cause noticeable time gains. In some cases, the guest clock has been observed to run more than 10% more quickly than real time.

Workarounds

If your Linux 2.6 guest's clock is running too quickly, it indicates a problem with lost tick correction. Apply one of the workarounds in the section "Preventing the Clock from Running Too Quickly."

If your guest's clock is running too slowly, it indicates that the host's real timer interrupt rate can't keep up with the guest's virtual timer interrupt rate. There are two approaches to dealing with the problem: either decrease the guest's rate or increase the host's rate. Apply one of the workarounds in the section "Preventing the Clock from Running Too Slowly." Also apply one of the workarounds for running too quickly, because correcting the first problem often reveals the second one.

In both cases, also make sure that VMware Tools is installed in your guest, that time synchronization is enabled and that you are not running any other clock synchronization software in the guest at the same time (such as ntpd).

If your host uses power management features (such as Intel SpeedStep, or AMD PowerNow or Cool'n'Quiet) that vary the processor speed, see .

Preventing the Clock from Running Too Quickly

There are two kernel options that help with the guest kernel's over-correction for lost ticks:

• Add the clock=pit boot option to your guest's kernel command line in the /etc/lilo.conf or /boot/grub/grub.conf file.

  The following example shows the syntax for LILO:

  image=/boot/vmlinuz
  label="linux"
  root=/dev/hda1
  initrd=/boot/initrd.img
  append="resume=/dev/hda6 splash=silent clock=pit"
  read-only
  
  (Remember to run /sbin/lilo after editing lilo.conf, so that your edits take effect.)

  Here is an example of the syntax for GRUB:

  title Fedora Core (2.6.9-1.667)
  root (hd0,0)
  kernel /vmlinuz-2.6.9-1.667 ro root=/dev/hda2 clock=pit

  Adding this boot option disables the kernel's correction for lost ticks, so be sure to also install VMware Tools and turn on time synchronization. The latter prevents the guest clock from losing time over the long term due to lost ticks.

  For additional information about working with boot loaders, see your Linux distribution's documentation.

• As an alternative, especially if you are unable to use VMware Tools, you can instead give the kernel command line option

    clock=pmtmr 

  With this option, the kernel corrects more properly for lost ticks, but occasionally overcorrects and ends up gaining time slowly. This option is the default for most 2.6 kernels, but some distributions may patch their kernels to change the default. In SuSE SLES 9 kernels, the default is clock=tsc. The code enabled by the tsc setting severely overcorrects for lost ticks when used in a virtual machine and tends to gain time rapidly.

Preventing the Clock from Running Too Slowly

One approach to a slow guest clock is to reduce the guest timer interrupt rate.

• In a one-CPU virtual machine, add the following kernel command line parameters to the guest:

    nosmp noapic nolapic 

  Kernel command line parameters are specified in the /etc/lilo.conf or /boot/grub/grub.conf file, depending on your choice of boot loader.

  Here is an example for LILO:

  image=/boot/vmlinuz
  label="linux"
  root=/dev/hda1
  initrd=/boot/initrd.img
  append="resume=/dev/hda6 splash=silent nosmp noapic nolapic"
  read-only
  
  (Remember to run /sbin/lilo after editing lilo.conf, so that your edits take effect.)
  
  And for GRUB:
  
  title Red Hat Linux (2.4.20-28.9)
  root (hd0,0)
  kernel /vmlinuz-2.4.20-28.9 ro root=/dev/hda2 clock=pit nosmp noapic nolapic
  
  You can even specify the entries to keep the clock from running too slowly and too quickly together. The entries together for LILO look like this:
  
  image=/boot/vmlinuz label="linux"
  root=/dev/hda1 initrd=/boot/initrd.img
  append="resume=/dev/hda6 splash=silent clock=pit nosmp noapic nolapic"
  read-only
  
  And for GRUB:
  
  title Red Hat Linux (2.4.20-28.9)
  root (hd0,0)
  kernel /vmlinuz-2.4.20-28.9 ro root=/dev/hda2 clock=pit nosmp noapic nolapic
  
  For additional information about working with boot loaders, see your Linux distribution's documentation.

• SUSE LINUX 9.0 Professional Edition, although its kernel is 2.4-based, includes a patch that raises the clock rate to 1000Hz. This patch is enabled by the kernel parameter desktop. SUSE installations pass this parameter to the kernel by default. To remove the parameter from your guest operating system, follow these steps:
  1. Edit your /etc/lilo.conf or /boot/grub/grub.conf file.
  2. Delete all instances of the word desktop.
  3. Exit the editor, saving your changes.
  4. If you are using LILO, run /sbin/lilo.
  5. Reboot the guest.
     
     
• In standard 2.6 kernels, the timer interrupt rate is fixed at kernel compile time and cannot be changed by command line parameters. You can, however, recompile your kernel with a lower timer interrupt rate. 100Hz is adequate for most applications in a Linux guest. See the documentation for your Linux distribution for detailed instructions on how to build and run a custom kernel. Before recompiling the guest kernel, locate the following line in /usr/src/linux-2.6/include/asm-i386/param.h:

    #define HZ 1000 

  Change the value of HZ to 100:

    #define HZ 100 

A different way to deal with a slow guest clock is to increase the host timer interrupt rate. Current versions of VMware products automatically increase the host's timer interrupt rate if needed, up to the maximum permitted by the host operating system. In some cases, though, you can increase this maximum.

• On Windows hosts, 1000Hz is an absolute maximum.
  
  
• On most Linux hosts, VMware is able to increase the timer interrupt rate to 8192Hz by requesting additional interrupts from the /dev/rtc device. However, on a few systems this device may not be configured. On 64-bit systems running Linux 2.4 kernels, the device cannot provide interrupts; on some outdated versions of VMware products, only one virtual machine can use /dev/rtc at a time. To deal with such issues, see .
  
  
• For ESX Server, 1000Hz is the default maximum, but you can increase the rate using the technique described at .