Linux Device Drivers

Linux Device Drivers

Character drivers
IO & Memory
Linux Kernel
Process Management
Process Address space

Linux Scheduler
Memory Management
System Calls
Kernel Synchronization
Linux Inter Process Communications

Serial Ports
Parallel Ports
Introduction to Hardware
Linux Timers
DMA in Linux
Linux Threads
Linux Thread Synchronization

Linux Multi Threading
Debugging in Linux
GDB GNU Debugger
KDB Kernel Debugger
KGDB Kernel GNU Debugger
Example Ethernet Driver

Debugging Techniques

Debugging Techniques


One of the problem for any one writing kernel code is how to approach debugging. Kernel code cannot be easily executed under a debugger, nor can it be easily traced, because it is set of functionalities not related to a specific process. Some of the techniques used to monitor kernel code and trace errors are:


ü     Debugging by printing

ü     Debugging by query

ü     Debugging by watching

ü     Debugging by tools


Debugging by printing


The most common debugging technique is monitoring. The messages can be printed out by calling printk at suitable points. printk lets you classify messages according to their severity by associating different log levels or priorities with the messages.


Here are two examples of printk commands, a debug message and critical message:


printk(KERN_DEBUG“Here I am: %s:%i\n”, __FILE__, __LINE__);

printk(KERN_CRIT “I’m trashed: giving up on %p\n”, ptr);


There are eight log-level strings, defined in the header <linux/kernel.h>


Based on the loglevel, the kernel may print the message to the current console.


Available priorities (include/linux/kernel.h):


#define KERN_EMERG      "<0>"                /* system is unusable */

#define KERN_ALERT      "<1>"                /* action must be taken immediately */

#define KERN_CRIT       "<2>"             /* critical conditions */

#define KERN_ERR        "<3>"             /* error conditions */

#define KERN_WARNING    "<4>"             /* warning conditions */

#define KERN_NOTICE     "<5>"                /* normal but significant condition*/

#define KERN_INFO       "<6>"                /* informational */

#define KERN_DEBUG      "<7>"                /* debug level messages */


How MesssagesGet Logged


The printk function writes messages into a circular buffer that is LOG_BUF_LEN (defined in kernel/printk.c) bytes long. It then wakes any process that is waiting for messages, that is any process that is sleeping in the syslog system call or that is reading /proc/kmsg. The reading from /proc/kmsg consumes data from the log buffer, whereas

the syslog system call can optionally return log data while leaving it for other processes as well. The klogd daemon process is responsible.


If you want to avoid clobbering your system log with the monitoring messages from your driver, you can either specify the –f(file) option to klogd to instruct it to save messages to a specific file, or modify /etc/syslog.conf to suit your needs.


Debugging by Querying


A few techniques are available to developers for querying the system:

creating a file in the /proc filesystem, using the ioctl driver method, and exporting attributes via sysfs.



ü     Through a file in /proc or /sys, which contents are handled by call backs defined and registered by your driver.

ü     Can be used to show any piece of information about your device or driver

ü     Can also be used to send data to the driver or to control it

ü     Caution: anybody can use these files. You should remove your debugging interface in production!



Debugging with /proc or /sys




ü     cat /proc/acme/stats (dummy example)Displays statistics about your acme driver.

ü     cat /proc/acme/globals (dummy example)Displays values of global variables used by your driver.

ü     echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeedAdjusts the speed of the CPU (controlled by the cpufreq driver).


Debugging by Watching




The strace command traces the execution of another program, listing any system calls the program makes and any signals it receives. To watch the system calls and signals in a program, invoke strace, followed by the program and its command-line arguments.


For example, to watch the system calls that are invoked by the hostname command, use this command:

%strace hostname




Each line corresponds to a single system call. For each call, the system call's name is listed, followed by its arguments and its return value. Where possible, strace conveniently displays symbolic names instead of numerical values for arguments and return values, and it displays the fields of structures passed by a pointer into the system call. In the output from strace hostname, the first line shows the execve system call that invokes the hostname program.


execve(“/bin/hostname”, [“hostname”], [/* 49 vars*/]) = 0



1.     strace does not show ordinary function calls.

2.     It is more convenient to redirect output from strace into a file. Use the option –o filename to do this.


oops message


Most bugs show themselves in NULL pointer dereferences or by the use of other incorrect pointer values. The usual outcome of such bugs is an oops message.


An oops displays the processor status at the time of the fault, including the contents of the CPU registers and other incomprehensible information.


One such message


Unable to handle kernel NULL pointer dereference at virtual address 00000000 printing eip: d083a064

Oops: 0002 [#1]

SMP CPU: 0 EIP: 0060:[<d083a064>] Not tainted

EFLAGS: 00010246 (2.6.6)

EIP is at faulty_write+0x4/0x10 [faulty]

eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 00000000

esi: cf8b2460 edi: cf8b2480 ebp: 00000005 esp: c31c5f74ds: 007b es: 007b ss: 0068

Process bash (pid: 2086, threadinfo=c31c4000 task=cfa0a6c0)


c0150558 cf8b2460 080e9408 00000005 cf8b2480 00000000 cf8b2460 cf8b2460fffffff7 080e9408 c31c4000 c0150682 cf8b2460 080e9408 00000005 cf8b248000000000 00000001 00000005 c0103f8f 00000001 080e9408 00000005 00000005

Call Trace:

[<c0150558>] vfs_write+0xb8/0x130

[<c0150682>] sys_write+0x42/0x70

[<c0103f8f>] syscall_call+0x7/0xb

Code: 89 15 00 00 00 00 c3 90 8d 74 26 00 83 ec0c b8 00 a6 83 d0


Asserting Bugs and dumping information


A number of kernel routines make it easy to flag bugs, provide assertions, and dump information. Two of the most common are BUG() and BUG_ON(). When called, they cause oops, which results in a stack trace and an error message dumped to the kernel.


if (bad_thing)





Asserting Bugs and dumping information…


A more critical error is signaled via panic(). A call to panic() prints an error message and then halts the kernel.

if (terrible_thing)

panic(“foois %ld\n”, foo);


If you just want a simple stack trace issued on the console:

if (debug_check)


printk(KERN_DEBUG “provide some information…\n”);





Debugging by tools


GNU Debugger (gdb)


You can use the standard GNU debugger to look inside a running kernel. Start the debugger on the kernel, type:

#gdb vmlinux/proc/kcore


The vmlinux file is the uncompressed kernel image stored in the root of the build directory.


You can use any of the gdb commands for reading information.


For example, to print the value of a variable

(gdb)p global_variable


To disassemble a function

(gdb)disassemble function


It can not modify kernel data in any way.

It is unable to single step through kernel or set break points.


Kgdb(remote kernel debugger)


Kgdb is a patch that enables a gdb to fully debug the kernel remotely over a serial line. It requires two computers. The first runs a kernel patched with kgdb. The second debugs the first over the serial line (a null modem cable connecting two computers) using gdb.


With kgdb, the entire feature set of gdb is accessible, reading and writing any variable, setting break point, setting watch point, single stepping etc.


kdb (The kernel debugger)


Kdb is a kernel patch that extensively modifies the kernel to allow direct debugging on the host system. It provides variable modification, breakpoints, single stepping on the host system. Running the debugger is simple, just hit the break key on the console. The debugger also automatically executes when the kernel oopses.