Why does the clock in my Linux guest consistently run more slowly or more quickly than real world time?
Linux guest operating systems keep time by counting timer
interrupts. Unpatched 2.4 and earlier kernels program the virtual
system timer to request clock interrupts at 100Hz (100 interrupts per
second). 2.6 kernels, on the other hand, request interrupts at 1000Hz —
ten times as often. Some 2.4 kernels modified by distribution vendors
to contain 2.6 features also request 1000Hz interrupts, or in some
cases, interrupts at other rates, such as 512Hz.
Furthermore, an SMP-capable Linux kernel requests additional timer
interrupts from the virtual local APIC timer. An SMP-capable kernel
running on a one-CPU system generates twice as many total timer
interrupts as the corresponding UP kernel, while such a kernel running
on a two-CPU system requests three times as many. In general, an
SMP-capable kernel running on CPUs requests times
as many interrupts per second as a UP kernel. For example, an
unmodified 2.6 Linux kernel running on a two-CPU virtual machine
requests a total of 3000 clock interrupts per second.
When a guest asks for more than 1000 clock interrupts per second, it
can be difficult for the virtual machine to keep up, especially if
other applications are running on the host at the same time. This can
cause the clock in the guest operating system to fall so far behind
real time that it is unable to catch up. The overhead of delivering so
many virtual clock interrupts can also hurt guest performance and
increase host CPU consumption.
It can also be difficult for the guest operating system to field
1000 clock interrupts per second. Even on real hardware, clock
interrupts are sometimes lost because the operating system is busy for
more than 1 millisecond and another clock interrupt comes in before the
previous one was handled. Linux 2.6 contains code to detect such "lost
ticks" and correct for them. Unfortunately, this code can trigger the
correction spuriously in some cases, resulting in the Linux clock
running more quickly than real time. This problem happens more often in
a virtual machine than on real hardware, and can cause noticeable time
gains. In some cases, the guest clock has been observed to run more
than 10% more quickly than real time.
Workarounds
If your Linux 2.6 guest's clock is running too quickly, it
indicates a problem with lost tick correction. Apply one of the
workarounds in the section "Preventing the Clock from Running Too
Quickly."
Note: The overcorrection for lost ticks has been fixed in Linux kernel 2.6.18. Upgrading to 2.6.18 or later also solves this problem.
If your guest's clock is running too slowly, it indicates that the
host's real timer interrupt rate can't keep up with the guest's virtual
timer interrupt rate. There are two approaches to dealing with the
problem: either decrease the guest's rate or increase the host's rate.
Apply one of the workarounds in the section "Preventing the Clock from
Running Too Slowly." Also apply one of the workarounds for running too
quickly, because correcting the first problem often reveals the second
one.
In both cases, also make sure that VMware Tools is installed in your
guest, that time synchronization is enabled and that you are not
running any other clock synchronization software in the guest at the
same time (such as ntpd).
If your host uses power management features (such as Intel
SpeedStep, or AMD PowerNow or Cool'n'Quiet) that vary the processor
speed, see .
Preventing the Clock from Running Too Quickly
32-bit Systems
For 32-bit systems, there are two kernel options that help with the guest kernel's over-correction for lost ticks:
- Add the clock=pit boot option to your guest's kernel command line in the /etc/lilo.conf or /boot/grub/grub.conf file.
The following example shows the syntax for LILO:
image=/boot/vmlinuz
label="linux"
root=/dev/hda1
initrd=/boot/initrd.img
append="resume=/dev/hda6 splash=silent clock=pit"
read-only
(Remember to run /sbin/lilo after editing lilo.conf, so that your edits take effect.)
Here is an example of the syntax for GRUB:
title Fedora Core (2.6.9-1.667)
root (hd0,0)
kernel /vmlinuz-2.6.9-1.667 ro root=/dev/hda2 clock=pit
Adding this boot option disables the kernel's correction for lost
ticks, so be sure to also install VMware Tools and turn on time
synchronization. The latter prevents the guest clock from losing time
over the long term due to lost ticks.
For additional information about working with boot loaders, see your Linux distribution's documentation.
- As an alternative, especially if you are unable to use VMware Tools, you can instead give the kernel command line option
clock=pmtmr
With this option, the kernel corrects more properly for lost
ticks, but occasionally overcorrects and ends up gaining time slowly.
This option is the default for most 2.6 kernels, but some distributions
may patch their kernels to change the default. In SuSE SLES 9 kernels,
the default is clock=tsc. The code enabled by the tsc setting severely overcorrects for lost ticks when used in a virtual machine and tends to gain time rapidly.
Preventing the Clock from Running Too Slowly
One approach to a slow guest clock is to reduce the guest timer interrupt rate.
- In a one-CPU virtual machine, add the following kernel command line parameters to the guest:
nosmp noapic nolapic
Kernel command line parameters are specified in the /etc/lilo.conf or /boot/grub/grub.conf file, depending on your choice of boot loader.
Here is an example for LILO:
image=/boot/vmlinuz
label="linux"
root=/dev/hda1
initrd=/boot/initrd.img
append="resume=/dev/hda6 splash=silent nosmp noapic nolapic"
read-only
(Remember to run /sbin/lilo after editing lilo.conf, so that your edits take effect.)
And for GRUB:
title Red Hat Linux (2.4.20-28.9)
root (hd0,0)
kernel /vmlinuz-2.4.20-28.9 ro root=/dev/hda2 nosmp noapic nolapic
You
can even specify the entries to keep the clock from running too slowly
and too quickly together. The entries together for LILO look like this:
image=/boot/vmlinuz label="linux"
root=/dev/hda1 initrd=/boot/initrd.img
append="resume=/dev/hda6 splash=silent clock=pit nosmp noapic nolapic"
read-only
And for GRUB:
title Red Hat Linux (2.4.20-28.9)
root (hd0,0)
kernel /vmlinuz-2.4.20-28.9 ro root=/dev/hda2 clock=pit nosmp noapic nolapic
For additional information about working with boot loaders, see your Linux distribution's documentation.
- SUSE LINUX 9.0 Professional Edition, although its kernel is
2.4-based, includes a patch that raises the clock rate to 1000Hz. This
patch is enabled by the kernel parameter desktop. SUSE
installations pass this parameter to the kernel by default. To remove
the parameter from your guest operating system, follow these steps:
-
Edit your /etc/lilo.conf or /boot/grub/grub.conf file.
- Delete all instances of the word desktop.
- Exit the editor, saving your changes.
- If you are using LILO, run /sbin/lilo.
-
Reboot the guest.
- In standard 2.6 kernels, the timer interrupt rate is fixed at
kernel compile time and cannot be changed by command line parameters.
You can, however, recompile your kernel with a lower timer interrupt
rate. 100Hz is adequate for most applications in a Linux guest. See the
documentation for your Linux distribution for detailed instructions on
how to build and run a custom kernel. Before recompiling the guest
kernel, locate the following line in /usr/src/linux-2.6/include/asm-i386/param.h:
#define HZ 1000
Change the value of HZ to 100: #define HZ 100
A different way to deal with a slow guest clock is to increase the
host timer interrupt rate. Current versions of VMware products
automatically increase the host's timer interrupt rate if needed, up to
the maximum permitted by the host operating system. In some cases,
though, you can increase this maximum.
-
On Windows hosts, 1000Hz is an absolute maximum.
-
On most Linux hosts, VMware is able to increase the timer interrupt rate to 8192Hz by requesting additional interrupts from the /dev/rtc
device. However, on a few systems this device may not be configured. On
64-bit systems running Linux 2.4 kernels, the device cannot provide
interrupts; on some outdated versions of VMware products, only one
virtual machine can use /dev/rtc at a time. To deal with such issues, see .
- For ESX Server, 1000Hz is the default maximum, but you can increase the rate using the technique described at .