use kgdb with vmware

from: 

KGDB is a set of kernel patches that allows a second machine running a debugger (gdb) to control the target machine over a serial port. This page describes:

• compiling and running KGDB-enabled kernels, for versions  and .
• establishing a serial connection between gdb and a virtual machine running under  or 

Compiling and running a 2.4 kernel with KGDB

1. Download the kgdb 1.9 kernel patch for 2.4.23 from: linux-2.4.23-kgdb-1.9.patch
2. Apply the patch: cd /usr/src/linux-2.4.23 patch -p1 < /tmp/linux-2.4.23-kgdb-1.9.patch
3. make menuconfig, and under "Kernel Hacking" enable "KGDB" and "KGDB Thread Analysis"
4. build your kernel. (since it's 2.4, that means make dep and make bzImage)
5. copy the kernel (bzImage) to the target virtual machine, or alternately copy vmlinux, the kernel executable, to the debugger machine.
6. add another lilo entry for the debug kernel, and append the argument "gdb" to get it to stop in the debugger during boot. I added the following lines to my lilo.conf file:prompt timeout=100 image=/vmlinuz.kgdb label=KGDB root=/dev/hda2 read-only append="gdb gdbbaud=115000"
   
   Note that the default image is the first in the file; the "prompt" and "timeout" arguments will allow you to specify which image to use at boot time. Don't forget to run lilo so it can save the parameters from /etc/lilo.conf.
7. Boot. It should output about 20 lines and then stop, waiting for the debugger.

Compiling and running a 2.6 kernel with KGDB

1. get the kgdb patches for kernel 2.6.15 here: 
2. untar them and then apply them: cd linux-2.6.15.6 patch -b -p1 < ../linux-2.6.15.5-kgdb-2.4/core-lite.patch patch -b -p1 < ../ ... core.patch patch -b -p1 < ../ ... i386-lite.patch i386.patch 8250.patch
3. configure and build your kernel. Under "kernel hacking" enable kgdb, set serial port to "0" (i.e. /dev/ttyS0, or COM1 - the default is ttyS1), and enable kernel debug info.
4. Add the command "kgdbwait" to the kernel command line for the kgdb kernel, and boot it. After printing "Booting the kernel" it should hang.

Connecting to VMWare KGDB, Linux host

1. Power down your virtual machine and click "Edit Virtual Machine Settings". Add a serial port, select "Connect to named pipe", and provide the pipe name "/tmp/com_1". (note that this isn't really a named pipe, but a unix-domain socket) Click "advanced settings" and set "yield CPU on poll". You can also get to this setting from the device list once you finish adding the port.
2. GDB can't connect directly to a unix-domain socket, so you'll need to use the socat ("socket cat") utility. It's installed on the elab machines, or you can install it from FC5 RPM or CentOS 4 RPM.
3. If the socket created by VMWare is /tmp/com_1, then invoke socat as: socat -d -d /tmp/com_1 PTY:
4. Find the line in the output identifying the PTY that was created:-bash-3.00$ socat -d -d /tmp/com_2 PTY: 2006/10/17 10:44:46 socat[20383] N successfully connected via 2006/10/17 10:44:46 socat[20383] N PTY is '''/dev/pts/7''' 2006/10/17 10:44:46 socat[20383] N starting data transfer loop ...
5. Start gdb on the vmlinux executable, and connect to the target:-bash-3.00$ gdb vmlinux.kgdb GNU gdb Red Hat Linux (6.3.0.0-1.96rh) [...] (gdb) target remote /dev/pts/7 Remote debugging using /dev/pts/7 0xc012c138 in .text.lock.kgdbstub () at kgdbstub.c:1007 1007 kgdbstub.c: No such file or directory. in kgdbstub.c warning: shared library handler failed to enable breakpoint (gdb)

And now you're up and running.

Connecting to KGDB, Windows Hhost

1. Add 2 virtual serial ports to your virtual machine. Configure the second one to connect to a named pipe named \\.\pipe\com_2. Under advanced settings, check the box for "yield CPU on poll"
2. Download and install the Named Pipe TCP Proxy utility: nptp_setup.zip
3. Start the proxy, and establish a connector between some port on the local machine (e.g. 8111) and \\.\pipe\com_2. Check the box to enable non-local systems access.
4. In another virtual machine run 'gdb vmlinux'. (you'll have to work out how to share the source tree between the two VMs - clearly you can't use NFS from the VM that you're debugging)
5. type target remote ip_addr:8111 where ip_addr is the windows host IP address that this virtual machine sees. (e.g. on my machine 192.168.149.1, and the VM gets address 192.168.149.128)

KGDB should connect, and away you go.

Caveats - note that I've had a lot of trouble with the DHCP server in VMWare if I'm running multiple machines at once, as it tends to give the same address to each one. You may wish to hardcode the ip address, route, and DNS resolver instead on each VM.

Connecting to KGDB using VMWare Fusion on Mac OS X

The process is a little long but effective:

1. Download socat for mac using the instrcutions on the site 

(i) just in case the site is removed, the key idea is to download darwinports on your mac (socat is ported onto mac under this project).

(ii) install socat: sudo port install socat

(iii) it may take a while (around 10-15 mins) and thus dont cancel the operation; be patient and wait (have faith in Darwin)

(iv) run socat, if mac reports that it cant find socat, dont panic its not in your path; find out its path (like for me its in "/opt/local/bin") and update your path.

1. compile the kernel with all those required patches (mentioned above) make sure that you do make install; do the necessary changes in your grub-file:

2.a) make sure that you have an alternate booting option, which means, leave a copy of vanilla-kernel and copy bzImage as another file in /boot

2.b) dont panic if make install fails ... it always fails on my machine ... just copy bzImage and System.map.... and it should work; if it doesnt, you need to do some simple hacking around to get it working (in the sense that the VM should boot with this new kernel)

2.c) if you are planing to debug a loadable module, then its faster to not to put a kgdbwait in the boot option because that, unnecessarily, makes the machine to wait.

1. install another virtual machine with linux on it (if you have not already done it); this will be the machine on which you will run gdb. Lets name it gdb-VM
2. if you havent figured out how to share the source-code between two virtual machines ... no sweat, you can still start by copying the vmlinux (note that its not vmlinuz) file from the source machine to gdb-VM
3. shutdown both the machines
4. find the ".vmx" file of the source virtual machine .... for e.g. in my case the VMs name is "Other Linux 2.6.x Kernel 3" ... so for me the file is like this "/Users/upendra/Documents/Virtual\ Machines/Other\ Linux\ 2.6.x\ Kernel\ 3/Other\ Linux\ 2.6.x\ Kernel\ 3.vmx" ... phew .. thats something ...
5. open it using vi (because I like vi for such simple work) ...
6. make the following changes (or add these lines if they are not there) ... make sure that specify the correct path for the serial0.fileName... the example has the path of my configuration

serial0.present = "TRUE" serial0.fileType = "pipe" serial0.yieldOnMsrRead = "TRUE" serial0.startConnected = "TRUE" serial0.fileName = "/Users/upendra/tmp/com_1"

1. start both the VMs
2. run the following command on your mac

socat -d -d /Users/upendra/tmp/com_1 TCP4-LISTEN:8888

1. go to your client gdb-VM and run this

gdb vmlinux target remote :8888

so for example, on my machine I run the following command and I think exactly same command should work on your machine also

gdb vmlinux target remote 192.168.44.1:8888

... your are done ... happy kernel debugging ....

Debugging modules under KGDB

1. get the address of the running module by running 'cat /proc/modules' on the target machine.
2. in gdb, type add-symbol-file /path/to/module/object.o 0xADDRESS which will load the symbols.