Srinivas B T

kgdb

 

KGDB is a source level debugger for linux kernel. It is used along with gdb to debug linux kernel. Kernel developers can debug a kernel similar to application programs with use of KGDB. It makes it possible to place breakpoints in kernel code, step through the code and observe variables. Two machines are required for using KGDB. One of these machines is a development machine and the other is a test machine. The machines are connected through a serial line.

 

KGDB is a kernel patch with following components:

1.     gdb stub

2.     modifications to fault handlers

3.     serial communication

 

1. Software setup and applying the kgdbpatch

 

1.     Downloaded Linux kernel source : linux-2.6.16.tar.bz2

 

2.     Downloaded the Kgdbpatch : kgdb-2.6.16.tar.bz2

 

3.     Unzip the kernel sources

cd${BASE_DIR}

tar -jxvflinux-2.6.16.tar.bz2

 

4.     cd ${BASE_DIR}/linux-2.6.16

 

5.     Unzip the kgdb patch

tar -jxvfkgdb-2.6.16.tar.bz2

Make sure that you are in the ${BASE_DIR}/linux-2.6.16

directory and then apply Kgdbpatches :

patch -p1 < ${BASE_DIR}/kgdb-2.6.16/core-lite.patch

...........

patch -p1 < ${BASE_DIR}/kgdb-2.6.16/i386.patch

 

 

2. Compiling the kernel on the development machine

 

 

1.     In the ${BASE_DIR}/linux-2.6.16/Makefile,

set theEXTRAVERSION = -kgdb

 

2.     make xconfig or make oldconfig

Select the options appropriate for the target machine Hardware.

Select the options pertaining to kgdb under "Kernel hacking".

 

Enable the following config options:

Kernel hacking->

KGDB: kernel debugging with remote gdb->

KGDB: Console messages through gdb

Method for KGDB communication

[ ] KGDB –Use only kernel module for I/O

[.] KGDB –On generic serial port (8250)

Simple selection of KGDB serial port

(115200) debug serial port baud rate

(0) Serial port number for KGDB

 

3.     make

 

4.     Transfer the built kernel to the Target machine from the Development machine

 

5.     Edit the /boot/grub/grub.conf file in the target machine to have the kgdbenabled kernel entry.

 

title Linux-2.6.16-kgdb

root (hd0,0)

kernel /boot/vmlinuz-2.6.16-kgdb roroot=/dev/hda1 kgdbwait

kgdb8250=1,115200

 

 

Starting the debug session

 

1.     After booting the target machine will wait for the host development machine to connect, by displaying the message:

Waiting for connection from remote gdb...

 

2.     cd ${BASE_DIR}/linux-2.6.16

 

3.     For setting a debug session with baud rate of 115200 on

/dev/ttyS0 , run as "root" user:

<root#> gdb./vmlinux

(gdb) set remotebaud 115200

(gdb) target remote /dev/ttyS0

(gdb)

 

 

Debugging a kernel

 

Using gdbfor kernel debugging

 

Usage of GDB kernel debugging is similar to that for debugging application processes.

KGDB supports GDB execution control commands, stack trace and thread analysis. It doesn't support GDB watchpoints.

 

 

Debugging a kernel

 

Killing or terminating GDB

If GDB does not respond to user input, you can kill it. If the test kernel was controlled by kgdb when gdb was killed, a new GDB can be started and it can be connected to kgdb. Otherwise you'll first need to make the test kernel drop into kgdb. To do this send an

ascii 3 character into the serial line.

 

An example of a command which does that is:

echo -e "\003" > /dev/ttyS0

 

When GDB has printed a command prompt, you can quit it and start a new GDB and connect to KGDB immediately.

 

 

Stopping kernel execution

 

To stop kernel execution, press Ctrl + C on the gdb terminal. GDB will send a stop message to KGDB stub, which will take control of the kernel and contact GDB. GDB will then present a command prompt and wait for user input. At this point all processors will be controlled by KGDB and no other part of the kernel will be executing. You can now use any GDB commands. GDB will present and prompt and wait for a command till you tell it to continue execution.

 

 

Continuing kernel execution

 

To continue kernel execution, use GDB command continue. Gdb will tell the KGDB stub to continue kernel execution. Kernel execution will continue after this point. GDB will not expect a user command till the user either breaks execution using Ctrl + C, or

KGDB stub gives control to it because of a breakpoint or some other reason.

 

 

Breakpoints

 

Use gdb break command to stop kernel execution at a function or a source code line. The command accepts a function name or a source code file name appended with a : and a line number as an argument.

 

Stepping through code

 

To step a code statement, use gdb command step. This command will run the kernel for one statement and again hand over the control to GDB. This command will step into function calls also. If you  want to skip stepping into function calls, use gdb command next.

 

Stack Trace

 

Once GDB is in command mode, you can look at the stack trace using backtrace  command. This command shows a list of function calls starting from the function where a system call entered the kernel.

For each function, it prints the source code file name and line number where next function was called and for the innermost function, where execution stopped. It also prints argument values to these functions.

We can break the debugger by pressing Ctrl + C, and see a stack trace as follows:

(gdb) backtrace

#0 breakpoint () at gdbstub.c:1160

#1 0xc0188b6c in gdb_interrupt(irq=3, dev_id=0x0, regs=0xc02c9f9c)

at gdbserial.c:143

 

Preparing modules for debugging

 

Requirements and Preparations on development and test machines

(Module debugging setup)

 

GDB on development machine

Install on the development machine, a GDB containing module debugging features. It's sources are available from the kgdb downloads page. This gdb is derived from GDB version 6.0.

It contains all features of GDB 6.0 plus ability to automatically detect module loading and unloading. You can either download the sources, build them and install the GDB at

/usr/local/bin/gdbmod-2.3. Prebuilt version of GDB are also available from the downloads page.

 

 

Loading the module symbols in GDB

 

This step is to be done on Development machine. It loads the  symbols of module in the GDB and makes them available to GDB for debugging.

 

NOTE: The module has to be built by debugging information.

#cd/usr/src/linux2.6.13 -

#gdbmod2.3 vmlinux-

(gdb) target remote /dev/ttyS0

 

Remote debugging using /dev/ttyS0

breakpoint () at gdbstub.c:1153

1153 }

(gdb)set solib-search-path /usr/src/linux2.6.13/drivers/net ---

gdb has to be able to locate module files once kgdb in forms it that a module has been loaded.

 

 

Inserting the module in the kernel:

 

Use 'insmod' command for inserting the module on test-machine.

GDB searches for a module file after appending a “.ko” to the module name.

# insmod mymodule.ko

Now module symbols are loaded and they can be debugged as normal kernel code.

 

 

Debugging modules

 

Unloading a module and loading it again

 

kgdb is capable of detecting loading and unloading of modules.

You can load and unload modules as and when needed. Gdb will automatically load and unload object files corresponding to them.

 

 

Debugging init_module

 

A special procedure is needed to debug init_module function from a module. Module debugging information is available only when gdb is notified of the module. init_module function from a module is already executed by that time. Hence  init_module cannot be debugged with this procedure. To debug init_module, place a breakpoint in the kernel just before the point where it calls init_module function. Load the module with modprobe or insmod and wait for the breakpoint to occur. When the breakpoint occurs gdb has already detected that the module was loaded. You can now place a breakpoint any where in the module.

 

 

Using kgdb over ethernet interface

 

For using kgdb over ethernet interface one has to enable the option in kernel hacking while configuring the kernel. e.g

Kernel hacking ->

[*] KGDB: kernel debugging with remote gdb->

[*] KGDB: Console messages through gdb

Method for KGDBcommunication (KGDB: On generic serial port (8250)) ->

( ) KGDB: Use only kernel modules for I/O

( ) KGDB: On generic serial port (8250)

(X) KGDB: On ethernet-in kernel

 

 

Using kgdb ethernet interface

 

1.     Add following line in the grub entry :kgdboe=@10.0.0.6/,@10.0.0.3/ (that's kgdboe=@LOCAL-IP/,@REMOTE-IP/)

# Sample grub.conf which will by default boot the kgdb enabled kernel

title Linux-2.6.16-kgdb(eth)

root (hd0,0)

kernel /boot/vmlinuz-2.6.16-kgdb ro root = /dev/hda1 kgdboe=@10.0.0.6/,@10.0.0.3/console=ttyS0,1152002

 

2.     Then for starting the debug session give following command on gdb.

(gdb) ./vmlinux

(gdb) target remote udp:HOSTNAME:6443