EXT3 Recovery - Score one for the fleshlings

I just got my blog back after wrestling with a hosed ext3 partition, and winning. Long story short, I put some bad ram into Adrock which quickly caused a kernel panic. Upon removing the bad RAM and rebooting /var was toast, the journal was toast and it looked like the superblocks were a bit charred as well. I had to dig for a while to get this problem solved... so here's the info on the failure and the fix.

/var was unmountable.
I removed /var from fstab and booted the machine on a /var dir off the root so I could work on it remotely.

Here's what happened next. I fsck a variety of ways, fsck.ext3 etc.. each reported the same:
fsck /dev/ida/c0d0p7
fsck 1.27 (8-Mar-2002)
e2fsck 1.27 (8-Mar-2002)
Group descriptors look bad... trying backup blocks...
fsck.ext2: Invalid argument while checking ext3 journal for /var

Tried downgrading to ext2 to bypass the hosed journal issues:

tune2fs -O ^has_journal /dev/ida/c0d0p7
tune2fs 1.27 (8-Mar-2002)
tune2fs: Invalid argument while reading journal inode

No good. Tried mounting as ext2 read only:

mount -t ext2 /dev/ida/c0d0p7 /mnt -o ro
mount: wrong fs type, bad option, bad superblock on /dev/ida/c0d0p7,
or too many mounted file systems

Nada. Tried some more stuff... tune2fs, and debugfs to try to fix / disable the journal, then decided I'd better back up the partition... should have done that first! ;P

e2fsck -c /dev/ida/c0d0p7
e2fsck 1.27 (8-Mar-2002)
Group descriptors look bad... trying backup blocks...
e2fsck: Invalid argument while checking ext3 journal for /var
[root@adrock init.d]# tune2fs -j /dev/ida/c0d0p7
tune2fs 1.27 (8-Mar-2002)
The filesystem already has a journal.
[root@adrock init.d]# debugfs -w /dev/ida/c0d0p7
debugfs 1.27 (8-Mar-2002)
/dev/ida/c0d0p7: Can't read an inode bitmap while reading inode bitmap
debugfs: quit
[root@adrock init.d]# dd if=/dev/ida/c0d0p7 of=/big/var.bak.dd
530368+0 records in

Still hosed, but now I have a backup of the burnination. And now for the FIX!!!

mke2fs -S /dev/ida/c0d0p7
mke2fs 1.27 (8-Mar-2002)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
66528 inodes, 265184 blocks
13259 blocks (5.00%) reserved for the super user
First data block=1
33 block groups
8192 blocks per group, 8192 fragments per group
2016 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185

Writing superblocks and filesystem accounting information: done

That did it! mke2fs re-wrote all of the superblocks and group descriptors and I was able to mount in RO mode and copy off the data. After that I was able to fsck and remount the original parition with no problems!

I found some great info on the RedHat ext3 mailing list archive which led me to this fix. Big thanks to the guys on that list