MySQL and the Leap Second, High CPU and the Fix

ETA: For those wanting to know the root cause, see the comments.

So, MySQL seemed to be faring just fine during and after the leap second. There were no big problems or catastrophic failures. However, we noticed that MySQL (and Java) servers* had a huge spike in CPU – which was odd, because nothing out of the ordinary was running. We were able to fix the problem quite easily. Here’s the graph, and the fix:

Note that the CPU spikes starting at 17:00 Pacific, which is midnight UTC. We fixed it around 19:35 Pacific.

The fix is quite simple – simply set the date. Alternatively, you can restart the machine, which also works. Restarting MySQL (or Java, or whatever) does NOT fix the problem. We put the following into puppet to run on all our machines:

$ cat files/bin/leap-second.sh 
#!/bin/bash
# this is a quick-fix to the 6/30/12 leap second bug

if [ ! -f /tmp/leapsecond_2012_06_30 ]
then
/etc/init.d/ntpd stop; date -s "`date`" && /bin/touch /tmp/leapsecond_2012_06_30
fi

We ran this in puppet, which we set to run the command on the next puppet run, and puppet would also restart ntp (the way we have it configured; your mileage may vary).

* and a bunch of our IT guys have personal, at-home minecraft servers, which were also similarly affected.

 

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/