Incremental Backups
Incremental backups, as described earlier in this chapter, are a good way to keep your system
backups up-to-date. For example, you can take nightly backups of only those files that
changed in the last 24 hours, weekly backups of all files that changed in the last week, and
monthly backups of the entire system.
You can create incremental backups using the tools mentioned previously: tar, gzip, cpio, and
so on. The first step in creating an incremental backup is to produce a list of files that changed
since a certain amount of time ago. You can do this easily with the find command.2 If you use
a special backup program, you will most likely not have to do this, but set some option
somewhere that you want to do an incremental backup.
For example, to produce a list of all files that were modified in the last 24 hours, we can use
the command:
find / -mtime -1 \! -type d -print > /tmp/filelist.daily
The first argument to find is the directory to start from — here, /, the root directory.
The -mtime -1 option tells find to locate all files that changed in the last 24 hours.
The \! -type d is complicated (and optional), but it cuts some unnecessary stuff from your
output. It tells find to exclude directories from the resulting file list. The ! is a negation
operator (meaning here, "exclude files of type d"), but put a backslash in front of it because
otherwise the shell interprets it as a special character.
Note that if you use find in this way, it traverses all mounted filesystems. If you have a CDROM
mounted, for example, find attempts to locate all files on the CD-ROM as well (which
you probably do not wish to backup). The -prune option can be used to exclude certain
directories from the walk that find performs across the system; or, you can use find multiple
times with a first argument other than /. See the manual page for find(1) for details.
Now you have produced a list of files to back up. Previously, when using tar, we have
specified the files to archive on the command line. However, this list of files may be too long
for a single command line (which is usually limited to around 2048 characters), and the list
itself is contained within a file.
You can use the -T option with tar to specify a file containing a list of files for tar to back up.
In order to use this option, you have to use an alternate syntax to tar in which all options are
specified explicitly with dashes. For example, to back up the files listed in /tmp/filelist.daily to
the device /dev/qft0, use the command:
tar -cv -T /tmp/filelist.daily -f /dev/qft0
You can now write a short shell script that automatically produces the list of files and backs
them up using tar. You can use cron to execute the script nightly at a certain time; all you
have to do is make sure there's a tape in the drive. You can write similar scripts for your
weekly and monthly backups. cron is covered in the next section.
To do it once a week on Monday, restore the third field to an asterisk but specify either 1 or
mon as the fifth field:
0 1 * * mon find /tmp -atime 3 -exec ls -l {} \;
To get even more sophisticated, there are ways to specify multiple times in each field. Here, a
comma means "run on the 1st and 15th day" of each month:
0 1 1,15 * * find /tmp -atime 3 -exec ls -l {} \;
while a hyphen means "run every day from the 1st through the 15th, inclusive":
0 1 1-15 * * find /tmp -atime 3 -exec ls -l {} \;
and a slash followed by a 5 means "run every fifth day" which comes out to the 1st, 6th, 11th,
and so on:
0 1 */5 * * find /tmp -atime 3 -exec ls -l {} \;
In this entry, we redirect the standard output, but allow the standard error to be sent as a mail
message. This can be a nice feature because we'll get a mail message if anything goes wrong.
If you want to make sure you don't receive mail under any circumstances, redirect both the
standard output and the standard error to a file:
0 1 * * * find /tmp -atime 3 -exec ls -l {} \; >> /home/mdw/log 2>&1
What to Do in an Emergency
It's not difficult to make a simple mistake as root that can cause real problems on your
system, such as not being able to log in or losing important files. This is especially true for
novice system administrators who are beginning to explore the system. Nearly all new system
admins learn their lessons the hard way, by being forced to recover from a real emergency. In
this section, we'll give you some hints about what to do when the inevitable happens.
You should always be aware of preventive measures that reduce the impact of such
emergencies. For example, take backups of all important system files, if not the entire system.
If you happen to have a Linux distribution on CD-ROM, the CD-ROM itself acts as a
wonderful backup for most files (as long as you have a way to access the CD-ROM in a tight
situation — more on this later). Backups are vital to recovering from many problems; don't let
the many weeks of hard work configuring your Linux system go to waste.
Also, be sure to keep notes on your system configuration, such as your partition table entries,
partition sizes and types, and filesystems. If you were to trash your partition table somehow,
fixing the problem might be a simple matter of rerunning fdisk, but this helps only as long as
you can remember what your partition table used to look like. (True story: one of the authors
once created this problem by booting a blank floppy, and had no record of the partition table
contents. Needless to say, some guesswork was necessary to restore the partition table to its
previous state!)
Of course, for any of these measures to work, you'll need a way to boot the system and access
your files, or recover from backups, in an emergency. This is best accomplished with an
"emergency disk," or "root disk." Such a disk contains a small root filesystem with the basics
required to run a Linux system from floppy — just the essential commands and system files,
as well as tools to repair problems. You use such a disk by booting a kernel from another
floppy (see Section 5.2.1 in Chapter 5) and telling the kernel to use the emergency disk as the
root filesystem.
Most distributions of Linux include such a boot/root floppy combination as the original
installation floppies. The installation disks usually contain a small Linux system that can be
used to install the software as well as perform basic system maintenance. Some systems
include both the kernel and root filesystem on one floppy, but this severely limits the number
of files that can be stored on the emergency disk. How useful these disks are as a maintenance
tool depends on whether they contain the tools (such as fsck, fdisk, a small editor such as vi,
and so on) necessary for problem recovery. Some distributions have such an elaborate
installation process that the installation floppies don't have room for much else.
At any rate, you can create such a root floppy yourself. Being able to do this from scratch
requires an intimate knowledge of what's required to boot and use a Linux system, and exactly
what can be trimmed down and cut out. For example, you could dispose of the startup
programs init, getty, and login, as long as you know how to rig things so that the kernel starts
a shell on the console instead of using a real boot procedure. (One way to do this is to have
/etc/init be a symbolic link to /sbin/bash, all on the floppy filesystem.)
While we can't cover all the details here, the first step in creating an emergency floppy is to
use mkfs to create a filesystem on a floppy (see the section Section 6.1.4 in Chapter 6). You
then mount the floppy and place on it whatever files you'll need, including appropriate entries
in /dev (most of which you can copy from /dev on your hard-drive root filesystem). You'll
also need a boot floppy, which merely contains a kernel. The kernel should have its root
device set to /dev/fd0, using rdev. This is covered in Section 5.2.1 in Chapter 5. You'll also
have to decide whether you want the root floppy filesystem loaded into a ramdisk (which you
can set using rdev as well). If you have more than 4 MB of RAM, this is a good idea because
it can free up the floppy drive to be used for, say, mounting another floppy containing
additional tools. If you have two floppy drives, you can do this without using a ramdisk.
If you feel that setting up an emergency floppy is too hard for you now after reading all this,
you might also want to try some of the scripts available that do it for you (e.g., tomsrtbt at
). But whatever you do, be sure to try the emergency floppy before
disaster happens!
At any rate, the best place to start is your installation floppies. If those floppies don't contain
all the tools you need, create a filesystem on a separate floppy and place the missing programs
on it. If you load the root filesystem from floppy into a ramdisk, or have a second floppy
drive, you can mount the other floppy to access your maintenance tools.
What tools do you need? In the following sections, we'll talk about common emergencies and
how to recover from them; this should guide you as to what programs are required for various
situations. It is best if the tools you put on that floppy are statically linked in order to avoid
problems with shared libraries not being available at emergency time.
Repairing Filesystems
It is possible to corrupt a filesystem so that it cannot be mounted. This is usually the result of
damage to the filesystem's superblock, which stores information about the filesystem as a
whole. If the superblock is corrupted, the system won't be able to access the filesystem at all,
and any attempt to mount it will fail (probably with an error to the effect of "can't read
superblock")
Because of the importance of the superblock, the filesystem keeps backup copies of it at
intervals on the filesystem. Second Extended filesystems are divided into "block groups,"
where each group has, by default, 8192 blocks. Therefore, there are backup copies of the
superblock at block offsets 8193, 16385 (that's 8192 x 2 + 1), 24577, and so on. If you use the
ext2 filesystem, check that the filesystem has 8192-block groups with the following
command:
dumpe2fs device | more
(Of course, this works only when the master superblock is intact.) This command will print a
great deal of information about the filesystem, and you should see something like:
Blocks per group: 8192
If another offset is given, use it for computing offsets to the superblock copies, as mentioned
earlier.
If you can't mount a filesystem because of superblock problems, chances are that fsck (or
e2fsck) will fail as well. You can tell e2fsck to use one of the superblock copies, instead, to
repair the filesystem. The command is:
e2fsck -f -b offset device
where offset is the block offset to a superblock copy; usually, this is 8193. The -f switch is
used to force a check of the filesystem; when using superblock backups, the filesystem may
appear "clean," in which case no check is needed. -f overrides this. For example, to repair the
filesystem on /dev/hda2 with a bad superblock, we can say:
e2fsck -f -b 8193 /dev/hda2
Superblock copies save the day. The previous commands can be executed from an emergency
floppy system and will hopefully allow you to mount your filesystems again.
Recently, so-called journalling filesystems have been introduced in most Linux distributions.
Examples of these are the ext3 filesystem, the Reiser filesystem, and the jfs filesystem. These
are less prone to filesystem corruption because they keep a log (the "journal") of all changes
made. Chances are thatwith these filesystems, you will never need to use any of the
techniques described here.
Restoring Files from Backup
If you have deleted important system files, it might be necessary to restore backups while
booting from an emergency disk. For this reason, it's important to be sure your emergency
disk has the tools you need to restore backups; this includes programs such as tar and gzip, as
well as the drivers necessary to access the backup device. For instance, if your backups are
made using the floppy tape device driver, be sure that the ftape module and insmod command
are available on your emergency disk. See Section 7.5 in Chapter 7 for more about this.
All that's required to restore backups to your hard-drive filesystems is to mount those
filesystems, as described earlier, and unpack the contents of the archives over those
filesystems (using the appropriate tar and gzip commands, for example; see Section 8.1 earlier
in this chapter). Remember that every time you restore a backup you will be overwriting other
system files; be sure you're doing everything correctly so that you don't make the situation
worse. With most archiving programs, you can extract individual files from the archive.
Likewise, if you want to use your original CD-ROM to restore files, be sure the kernel used
on your emergency disks has the drivers necessary to access the CD-ROM drive. You can
then mount the CD-ROM (remember the mount flags -r -t iso9660) and copy files from there.
The filesystems on your emergency disks should also contain important system files; if you
have deleted one of these from your system, it's easy to copy the lost file from the emergency
disk to your hard-drive filesystem.
