Learn The dd Command

This document is made to be easily skimmed. The latest addition was on 04-01-08, "Make an AES encrypted loop device". This is comprehensive documentation for one of the most useful linux/UNIX Windows commands-dd. It is a bitstream duplicator for duplicateing data. If you have a question, post it.

Windows users will find help about 50 lines down from here.

First Time visitors please leave a reply.

duplicateing smaller partition, or drive to larger partition, or drive; or vice versa:

Code:

rsync -avH mount_point/* other_mount_point/

Example: you want to duplicate the root directory tree to another drive, but the other drive is larger. If you use dd, you will get a that is smaller then the larger destination drive. In this case you want to duplicate files, not the file system. Format and mount the destination drive.

Code:

rsync -avH  /* /mnt/destination_drive/

will duplicate all the files as files. Rsync preserves the target file system. You need to run:

Code:

grub-install
update-grub

from a the rescue menu of an installation CD/DVD for the target to become bootable. If the target was bootable previously, it remains bootable.

Brief notes for Microsoft Windows XP and Windows Vista users

There is no excuse for not using dd for all your drive cloning, backup, upgrading, and restore tasks. Boot a Windows XP machine with a Knoppix CD; A live CD operating system that is self contained, relying only on the CD, not the . You can download Knoppix, burn the iso image file to a CD, boot with it, and clone drives. Drives are described to the dd command using device files. When you boot into Knoppix Linux, open a root shell. It's in the penguin menu. Parallel IDE (80 conductor grey ribbon cable), uses the first drive (master) on channel 0 (The first channel): /dev/hda. The second drive (slave) on IDE channel 0: /dev/hdb. SATA are /dev/sda and /dev/sdb. The root shell is a command interpretor.

Code:

fdisk -l /dev/hda

The partitions on the first drive.

Code:

man fdisk

The manual page for fdisk. Parted to make partitions

Code:

man parted

Dd command examples will be found in the remainder of the post. If one has trouble, ask. Performance of Knoppix Linux does not indicate performance of Linux. Knoppix Linux runs on a (1/1000 the speed of a HDD). Linux is high performance, giving power users the speed and flexibility to do High Performance Computing. The Knoppix CD contains over 1 GB of software, in compressed format. Feel free to read the entire post.
[color="red"]Dd is not presently able to clone Microsoft Windows Vista OEM Partitions.
End Windows Section

Begin Linux dd
The basic command structure is as follows:

Code:

dd if= of= bs=("USUALLY" some power of 2, not less than 512 bytes(ie, 512, 1024, 2048, 4096, 8192, 16384[/b], but can be ANY reasonable number.) skip= seek= conv=.

Source is the data being read. Target is where the data gets written.

Warning!! If you reverse the source and target, you can wipe out a lot of data. This feature has inspired the nickname "dd" Data Destroyer.
Warning!! Caution should be observed when using dd to duplicate encrypted partitions.

Examples: duplicate one hard to another hard disk:

Code:

dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=notrunc,noerror

sda2 and sdb2 are partitions. You want to duplicate sda2 to sdb2. If sdb2 doesn't exist, dd will start at the beginning of the disk, and create it. Be careful with order of if and of. You can write a blank disk to a good disk if you get confused. If you duplicate a smaller partition to a larger one the larger one, using dd, the larger one will now be formatted the same as the smaller one, and there won't be any space left on the drive. The way around this is to use

Code:

rsync

, as described in the beginning of the post
Make an iso image of a CD:

Code:

dd if=/dev/hdc of=/home/sam/myCD.iso bs=2048 conv=sync,notrunc

This duplicates sector for sector. MyCD.iso will be a hard disk image file of the CD. You can mount the image:

Code:

mkdir /mnt/myCD
mount -o loop /home/sam/myCD.iso /mnt/myCD
CD /mnt/myCD
ls

This will display the top level of the CD.
duplicate a floppy disk:

Code:

dd if=/dev/fd0 of=/home/sam/floppy.image bs=2x80x18b conv=notrunc

OR

Code:

dd if=/dev/fd0 of=/home/sam/floppy.image conv=notrunc

The 18b specifies 18 sectors of 512 bytes. 2x multiplies the sector size by the number of heads. 80x specifies cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to

Code:

/home/sam/floppy.image

. Thanks to Sam Cheeseman for documenting this unusual, but highly cool CHS method of specifying the block size (bs=) parameter of dd.
This makes a hard drive image of the floppy, with bootable info intact. The second example uses default bs=512: 1.44 MB floppy sector size. If you're concerned about spies with superconducting quantum-interference detectors, you can always add a "for" loop for US Government DoD approved secure hard disk erasure. Copy and paste the following two lines into a text editor.

Code:

#!/bin/bash 
for n in `seq 7`; do dd if=/dev/urandom of=/dev/sda bs=8b conv=notrunc; done

Save the file as 'swqid'.

Code:

chmod a+x swqid

Don't run the program until you are want to wipe the drive.
To make a bootable : Download 50 MB Debian based distro here:

Plug in the thumb drive into a USB port. Do:

Code:

dmesg | tail

Look where the new drive is, sdb1, or something similar. Do:

Code:

dd if=/home/sam/insert.iso of=/dev/sdb ibs=4b obs=1b conv=notrunc,noerror

Set the BIOS to USB boot, and boot.
duplicate the MBR and boot sector of a floppy to hard drive image:

Code:

dd if=/dev/fd0 of=/home/sam/MBRboot.image bs=512 count=2

This duplicates the first 2 sectors of the floppy.
Cloning an entire :

Code:

dd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror

/dev/sda is the source. /dev/sdb is the target. Do not reverse the intended source and target. It happens. Notrunc means 'do not truncate the output file'. Noerror means to keep going if there is an error. Normally dd stops at any error.
Duplicate MBR only:

Code:

dd if=/dev/sda of=/home/sam/MBR.image bs=446 count=1

This will duplicate the first 446 bytes of the hard drive to a file. If you haven't already guessed, reversing the objects of if and of, on the dd command line, reverses the direction of the write.
Wipe a hard drive: (Boot from a live CD distro to do this.)
is a good boot CD. The helix boot environment contains the DoD version of dd called dcfldd. It works the same way, but is has a .

Code:

dcfldd if=/dev/zero of=/dev/sda conv=notrunc

Beware, dcfldd does not observe the 'seek=' parameter.
This is useful for making the drive almost like new. Most drives have 0xffh written to every byte, from the factory.
Overwrite all the free space on a partition (deleted files you don't want recovered)

Code:

dd if=/dev/urandom of=/home/sam/fileconsumingallfreespace.file

When dd says no room left on device, all the free space has been overwritten with random characters. Then, delete the big file with

Code:

rm

.
:

Code:

dd if=/proc/kcore | hexdump -C | less

use PgUp, PgDn, up arrow, down arrow to navigate in less. Less is my favorite editor, except you can't edit with it.
Filesystems:

Code:

dd if=/proc/filesystems | hexdump -C | less

Modules:

Code:

dd if=/proc/kallsyms | hexdump -C | less

Interrupt table:

Code:

dd if=/proc/interrupts | hexdump -C | less

Uptime (seconds):

Code:

dd if=/proc/uptime | hexdump -C | less

Partitions (in KB):

Code:

dd if=/proc/partitions | hexdump -C | less

Memory stats:

Code:

dd if=/proc/meminfo | hexdump -C | less

I put two identical drives in every one of my machines. Before I do anything that most probably spells disaster, like an untested command line in a root shell, that contains 'xargs rm -rf', I do:

Code:

dcfldd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror

and duplicate my present working /dev/sda drive system to the /dev/sdb drive. If I wreck the installation on sda, I boot with the helix CD and:

Code:

dcfldd if=/dev/sdb of=/dev/sda bs=4096 conv=notrunc,noerror

and I get everything back exactly the same as before whatever daring maneuver I was trying didn't work. You can really, really learn this way, because you can't wreck what you have an exact duplicate of. You also might consider making the root partition separate from /home, and make /home big enough to hold the root partition, plus more. Then you can do:

Code:

dd if=/dev/sda2 (root) of /home/sam/root.img bs=4096 conv=notrunc,noerror

To make a backup of root, and :

Code:

dd if /home/sam/root.img of=/dev/sda2 (root) bs=4096 conv=notrunc,noerror

To write the image of root back to the root partition if you messed up and can't launch the , or edited /etc/fstab and can't figure out what you did wrong. It only takes a few minutes to restore a 15 GB root partition from an image file.



How to make a swap file, or another swapfile on a running system:

Code:

dd if=/dev/zero of=/swapspace bs=4k count=250000
mkswap /swapspace
swapon /swapspace

This can solve out of memory issues due to memory leaks on servers that cannot easily be rebooted.

How to pick proper block size:

Code:

dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file
dd if=/dev/zero bs=2048 count=500000 of=/home/sam/1Gb.file
dd if=/dev/zero bs=4096 count=250000 of=/home/sam/1Gb.file
dd if=/dev/zero bs=8192 count=125000 of=/home/sam/1Gb.file

This method can also be used as a drive benchmark, to find strengths and weaknesses in hard drives:
Read:

Code:

dd if=/home/sam/1Gb.file bs=64k | dd of=/dev/null

Write:

Code:

dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file

When dd finishes it outputs (total size)/(total time). You get the idea.
Play with 'bs=' and 'count=', always having them multiply out to the same toal size. You can calculate bytes/second like this: 1Gb/total seconds = Gb/s. You can get more realistic results using a 3Gb file.

Rejuvenate a hard drive
To cure input/output errors experienced when using dd. Over time the data on a drive, especially a drive that hasn't been used for a year or two, grows into larger magnetic flux points than were originally recorded. It becomes more difficult for the drive heads to decipher these magnetic flux points. This results in I/O errors. Sometimes sector 1 goes bad, resulting in a useless drive. Try:

Code:

dd if=/dev/sda of=/dev/sda

to rejuvenate the drive. Rewrites all the data on the drive in nice tight magnetic patterns that can then be read properly. The procedure is safe and economical.

Make a file of 100 random bytes:

Code:

dd if=/dev/urandom of=/home/sam/myrandom bs=100 count=1

/dev/random produces only as many random bits as the entropy pool contains. This yields quality randomness for cryptographic keys. If more random bytes are required, the process stops until the entropy pool is refilled (waggling your mouse helps). /dev/urandom does not have this restriction. If the user demands more bits than are currently in the entropy pool, it produces them using a pseudo random number generator. Here, /dev/urandom is the Linux random byte device. Myrandom is a file.

Randomize data over a file before deleting it:

Code:

ls -l

to find filesize.
In this case it is 3769

Code:

ls -l afile -rw------- ... 3769 Nov 2 13:41 

Code:

dd if=/dev/urandom of=afile bs=3769 count=1 conv=notrunc

duplicate a disk partition to a file on a different partition.

Warning!! Do not duplicate a partition to the same partition.

Code:

dd if=/dev/sdb2 of=/home/sam/partition.image bs=4096 conv=notrunc,noerror

This will make a file that is an exact duplicate of the sdb2 partition. You can substitue hdb, sda, hda, etc ... OR

Code:

dd if=/dev/sdb2 ibs=4096 | gzip > partition.image.gz conv=noerror

Makes a gzipped archive of the entire partition. To restore use:

Code:

 dd if=partition.image.gz | gunzip | dd of=/dev/sdb2

For bzip2 (slower,smaller), substitute bzip2 and bunzip2, and name the file

Code:

< filename >.bz2

Restore a disk partition from an image file.

Code:

dd if=/home/sam/partition.image of=/dev/sdb2 bs=4096 conv=notrunc,noerror

Convert a file to uppercase:

Code:

dd if=filename of=filename conv=ucase

Make a ramdrive:
The Linux kernel makes a number a you can make into ramdrives. You have to populate the drive with zeroes like so:

Code:

dd if=/dev/zero of=/dev/ram7 bs=1k count=16384

Populates a 16 MB ramdisk.

Code:

mke2fs -m0 /dev/ram7 4096

puts a file system on the ramdisk, turning it into a ramdrive. Watch this puppy smoke.

Code:

debian:/home/sam # hdparm -t /dev/ram7
/dev/ram7:
Timing buffered disk reads:   16 MB in  0.02 seconds = 913.92 MB/sec

You only need to do the timing once, because it's cool. Make the drive again, because hdparm is a little hard on ramdrives. You can mount the ramdrive with:

Code:

mkdir /mnt/mem
mount /dev/ram7 /mnt/mem

Now you can use the drive like a hard drive. This is particularly superb for working on large documents or programming. You can duplicate the large file or to the ramdrive, which on my machine is at least 27 times as fast as /dev/sda, and every time you save the huge document, or need to do a compile, it's like your machine is running on nitromethane. The only drawback is data security. The ramdrive is volatile. If you lose power, or lock up, the data on the ramdrive is lost. Use a reliable machine during clear skies if you use a ramdrive.

Duplicate to a file:

Code:

dd if=/dev/mem of=/home/sam/mem.bin bs=1024

The device

Code:

/dev/mem

is your system memory. You can actually duplicate any block or character device to a file using dd. Memory capture on a fast system, with bs=1024 takes about 60 seconds, a 120 GB HDD about an hour, a CD to hard drive about 10 minutes, a floppy to a hard drive about 2 minutes. With dd, your floppy drive images will not change. If you have a bootable DOS diskette, and you save it to your HDD as an image file, when you restore that image to another floppy it will be bootable.

Dd will print to the terminal window if you omit the

Code:

of=/dev/output

part.

Code:

dd if=/home/sam/myfile

will print the file myfile to the terminal window.

To search the :

Code:

dd if=/dev/mem | hexdump -C | grep 'some-string-of-words-in-the-file-you-forgot-to-save-before-the-power-failed'

If you need to cover your tracks quickly, put the following commands in a script to overwrite system ram with zeroes. Don't try this for fun.

Code:

mkdir /mnt/mem
mount -t ramfs /dev/mem /mnt/mem
dd if=/dev/zero > /mnt/mem/bigfile.file

This will overwrite all unprotected memory structures with zeroes, and freeze the machine so you have to reboot (Caution, this also prevents committment of the file system journal, and could trash the file system).

You can get arrested in 17 states for doing this next thing. Make an AES encrypted loop device:

Code:

dd if=/dev/urandom of=/home/sam/aes-drv bs=16065b count=100
modprobe loop
modprobe cryptoloop
modprobe aes
losetup -e aes /dev/loop1 ./aes-drv
password:
mkreiserfs /dev/loop1
mkdir /aes
mount -o loop,encryption=aes,acl ./aes-drv /aes
password:
mv /home/sam/porno /aes

to get the porno on the aes drive image.

Code:

umount /aes
losetup -d /dev/loop1
rmmod aes 
rmmod cryptoloop
rmmod loop

to make 'aes-drv' look like a 400 MB file of random bytes. Every time the lo interface is configured using losetup, according to the above, and the file 'aes-drv' is mounted, as above, the porno stash will be accessible in /aes/porno. You don't need to repeat the dd command, OR, the format with reiserfs, OR, the mv command. You only do those steps once. If you forget the password, there is no way to recover it besides guessing. Once the password is set, it can't be changed. To change the password, make a new file with the desired password, and move everything from the old file to the new file. Acl is a good mount option, because it allows use of acls. Otherwise your stuck with u,g,o and rwx.

If you are curious about what might be on you , or what an MBR looks like, or maybe what is at the very end of your disk:

Code:

dd if=/dev/sda count=1 | hexdump -C

Will show you sector 1, or the MBR. The bootstrap code and partition table are in the MBR.
To see the end of the disk you have to know the total number of sectors, and the MAS must be set equal to the MNA. The helix CD has a utility to set this correctly. In the dd command, your skip value will be one less than MNA of the disk. For a 120 GB Seagate

Code:

dd if=/dev/sda of=home/sam/myfile skip=234441646 bs=512

,
So this reads sector for sector, and writes the last sector to myfile. Even with LBA addressing, disks still secretly are read in sectors, cylinders, and heads.
There are 63 sectors per track, and 255 heads per cylinder. There is a total cylinder count. 512_bytes/sector*63_sectors/track*255heads=16065*512bytes/cylinder=8,225,280_bytes/cylinder. 63_sectors/track*255_heads=sectors/cylinder. With 234441647 total sectors, and 16065 sectors per cylinder, you get some trailing sectors which do not make up an entire cylinder: 14593.317584812_cylinders/drive. This leaves 5102 sectors which cannot be partitioned, because to be in a partition you have to be a whole cylinder. It's like having part of a person. That doesn't really count as a person. These become surplus sectors after the last partition. You can't ordinarily read past the last partition. But dd can. It's a good idea to check for anything writing to surplus sectors. For our GB drive, 234,441,647_sectors/drive - 5102_surplus_sectors = 234,436,545 partitionable sectors.

Code:

dd if=/dev/sda of=/home/sam/myfile skip=234436545

writes the last 5102 sectors to myfile. Launch midnight commander (mc) to view the file. If there is something in there, you do not need it for anything. In this case you would write over it with random characters:

Code:

dd if=/dev/urandom of=/dev/sda bs=512 seek=234436545

Will overwrite the 5102 surplus sectors on our 120 GB Seagate drive.

Block size:
One cylinder in LBA mode = 255_heads*63_sectors/track=16065_sectors=16065*512_bytes=8,225,280_bytes. The b means '* 512'. 32130b represents a two cylinder block size. Cylinder block size always works to cover every sector in a partition, because partitions are made of a whole number of cylinders. One cylinder is 8,225,280 bytes. If you want to check out some random area of the disk:

Code:

dd if=/dev/sda of=/home/sam/myfile bs=4096 skip=2000 count=1000

Will give you 8,000 sectors in myfile, after the first 16,000 sectors. You can open that file with a , edit some of it, and write the edited part back to disk:

Code:

dd if=/home/sam/myfile of=/dev/sda bs=4096 seek=2000 count=1000

Image a partition to another machine:
On source machine:

Code:

dd if=/dev/hda bs=16065b | netcat < targethost-IP > 1234

On target machine:

Code:

netcat -l -p 1234 | dd of=/dev/hdc bs=16065b

Variations on target machine:

Code:

netcat -l -p 1234 | bzip2 > partition.img

makes a compressed image file using bzip2 compression.

Code:

netcat -l -p 1234 | gzip > partition.img

makes a compressed image file using gzip compression. I back up a 100 GB lappy disk on a desktop drive, over a , and the 100 GB compresses to about 4.0 GB. Most of the drive is empty, so it's mostly zeroes. Repetitive zeroes compress well.
Alert!! Don't hit enter yet. Hit enter on the target machine. THEN hit enter on the source machine.

Netcat is a program, available by default, on most linux installations. It's a networking swiss army knife. In the preceding example, netcat and dd are piped to one another. One of the functions of the linux kernel is to make pipes. The pipe character looks like two little lines on top of one another, both vertical. Here is how this command behaves: This byte size is a cylinder. bs=16065b equals one cylinder on an LBA drive. The dd command is piped to netcat, which takes as its arguments the of the target(like 192.168.0.1, or any IP address with an open port) and what port you want to use(1234).



CONTINUES...SEE NEXT POST
dd will not duplicate or erase an HPA, OR, host protected area. Dd will erase a disk completely, but not as well as using the hardware secure erase, security erase unit command
Dd need not be black boxed like other inexpensive forensic :

For a low cost bootable CD based professional ghosting solution, that supports all operating systems and file systems:

Variation of dd for data rescue off defective media:
http://www.garloff.de/kurt/linux/ddrescue/
Department of Defense implementation of dd:

Sdd is useful when input block size is different than output block size, and will succeed in some instances where dd fails:

This is one of the best links I haven't written about dd:


Dd is like Symantec Norton Ghost, Acronis True Image, Symantec Drive Image. You can perform disk drive backup, restore, imaging, disk image, cloning, clone, drive cloning, transfer image, transfer data, clone to another drive or clone to another machine, move to a new hard drive, clone Windows XP, clone Windows, transfer Windows, hard drive upgrade, duplicate a boot drive, duplicate a bootable drive, upgrade your operating system hard drive, Tired of reinstalling WinXP Windows XP?