Conduct Kernel debugging test on Embedded Linux S3C2440 by using KGDB. KGDB has been integrated into Linux kernel from version 2.6.15, it requires a serial connection to the development host. It is a cross GDB running on the development host workstation. Refer to earlier blog entry Embedded Linux S3C2440 – Kernel Module, all the tests are based on Foo.ko kernel module and test application Footest.
Refer to Embedded Linux S3C2440 – Kernel Module, KGDB has been compiled into kernel and been enabled. We need to configure Embedded Linux S3C2440 to pass parameter to KGDB moudle to inform which port we use and send system request to trigger KGDB, as below steps shows,
Open Minicom and startup the Embedded Linux S3C2440 board, login as root, then do below actions,
[root@mini2440 /root]# echo ttySAC0 > /sys/module/kgdboc/parameters/kgdboc kgdb: Registered I/O driver kgdboc. [root@mini2440 /root]# echo g > /proc/sysrq-trigger SysRq : DEBUGvim Entering KGDBAbove actions tell kernel which UART will be used for KGDB, Kgdboc means KGDB over console, so above commands shows Embedded Linux S3C2440 board kernel has entered KGDB and waiting for the connection from host GDB. KGDB is using serial link, we need to exit minicom. As below steps show,
Press Ctrl-A then Z to enter menu of minicom. Press Q to quit without reset. Select Yes when you have blow screen.
+----------------------+ | Leave without reset? | | Yes No | +----------------------+Start the GDB on host machine,
[root@localhost ~]# cd /home/iot/mini2440/linux-3.8.7/ [root@localhost linux-3.8.7]# arm-linux-gdb vmlinux GNU gdb (GDB) 7.5.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/iot/mini2440/linux-3.8.7/vmlinux...done. (gdb)Set the baud rate,
(gdb) set remotebaud 115200 (gdb)Connect GDB to the S3C2440 board KGDB,
(gdb) target remote /dev/ttyUSB0 Remote debugging using /dev/ttyUSB0 kgdb_breakpoint () at kernel/debug/debug_core.c:1012 1012 arch_kgdb_breakpoint(); (gdb) step arch_kgdb_breakpoint () at kernel/debug/debug_core.c:1011 1011 wmb(); /* Sync point before breakpoint */ (gdb)Above actions include, startup GDB and pass the kernel ELF file vmlinux to GDB, vmlinux is uncompressed version kernel image, connect to S3C2440 board using GDB ‘target remote’ command.
The driver foo.ko was compiled as a module, we need to set a break point at where the module is loaded. System call sys_init_module will copy the module information from user space to kernel space and it calls load_module(), set the break point at line 3068 and continue run kernel, as below shows.
(gdb) b kernel/module.c:3068 Breakpoint 1 at 0xc00525d4: file kernel/module.c, line 3068. (gdb) (gdb) c Continuing.Connect Linux S3C2440 board from host through telnet. And transfer the kernel module foo.ko and application footest from host to S3C2440 board.
[root@localhost ~]# telnet 192.168.0.11 Trying 192.168.0.11... Connected to 192.168.0.11. Escape character is '^]'. mini2440 login: root [root@mini2440 /root]# [root@mini2440 /root]# tftp -g -r foo.ko 192.168.0.1 foo.ko 100% |********************************************************************************************************************************************| 69081 0:00:00 ETA [root@mini2440 /root]# chmod 777 foo.ko [root@mini2440 /root]# tftp -g -r footest 192.168.0.1 footest 100% |********************************************************************************************************************************************| 6122 0:00:00 ETA [root@mini2440 /root]# chmod 777 footest [root@mini2440 /root]# insmod foo.koAfter above insmod command, the foo.ko module is loaded, the GDB will stop at the break point, kernel/module.c:3068, as below gdb output shows,
[New Thread 485] [Switching to Thread 485] Breakpoint 1, do_init_module (mod=0xbf000344) at kernel/module.c:3068 3068 ret = do_one_initcall(mod->init); (gdb) step do_one_initcall (fn=0xbf002000) at init/main.c:682 682 int count = preempt_count(); (gdb)As foo.ko is kernel module, there is no symbol information about the this module in vmlinux. We cannot set a break point at foo.ko module, and we do not know which address will the Linux kernel link to foo.ko module. Linux stores this module’s .text section address in the module information structure struct module in the module_core_element.
By using lsmod macro below, we can obtain the address of struct module allocated to foo.ko module, we will use the address retrieved to obtain the module_core address to be used as module’s .text address. And foo_init(void) function is defined with __init macro #define __init __section(.init.text) __cold notrace, this macro expands into a compiler attribute that directs the linker to place the marked portion of code into a specially named ELF section named .init.text. So the compiled foo_init() function will be placed into .init.text section of the foo.ko object module.
The GDB lsmod macros in file /root/.gdbinit
#gdb List Modules Macro define lsmod printf "Address\t\tModule\n" set $m=(struct list_head *)&modules set $done=0 while ( !$done ) # list_head is 4-bytes into struct module set $mp=(struct module *)((char *)$m->next - (char *)4) printf "0xX\t%s\n", $mp, $mp->name if ( $mp->list->next == &modules) set $done=1 end set $m=$m->next end endWhen the foo.ko module is loaded, the kernel allocates a chunk of memory for the main body of the module, which is pointed to by the struct module member named module_core. It then allocates a separate chunk of memory to hold the .init.text section. After the initialization function called, the kernel frees the memory that contained the initialization function. So the object module is split, and we need to inform gdb to be able to use symbolic data for debugging the initialization function.
Below gdb commands is to add symbol file and set break points.
(gdb) lsmod Address Module 0xBF000344 foo (gdb) set $m=(struct module*)0xBF000344 (gdb) p $m->module_core $1 = (void *) 0xbf000000 (gdb) p $m->module_init $2 = (void *) 0xbf002000 (gdb) add-symbol-file ./drivers/char/foo.ko 0xbf000000 -s .init.text 0xbf002000 add symbol table from file "./drivers/char/foo.ko" at .text_addr = 0xbf000000 .init.text_addr = 0xbf002000 (y or n) y Reading symbols from /home/iot/mini2440/linux-3.8.7/drivers/char/foo.ko...done. (gdb) b foo_read Breakpoint 2 at 0xbf0000a0: file drivers/char/foo.c, line 59. (gdb) b foo_write Breakpoint 3 at 0xbf000148: file drivers/char/foo.c, line 46. (gdb) b do_write Breakpoint 4 at 0xbf000100: file drivers/char/foo.c, line 28. (gdb)Step into kernel module foo.ko,
(gdb) step current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99 99 return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); (gdb) step do_one_initcall (fn=0xbf002000 <foo_init>) at init/main.c:685 685 if (initcall_debug) (gdb) step 682 int count = preempt_count(); (gdb) step current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99 99 return (struct thread_info *)(sp & ~(THREAD_SIZE - 1)); (gdb) step do_one_initcall (fn=0xbf002000 <foo_init>) at init/main.c:685 685 if (initcall_debug) (gdb) step 681 { (gdb) step 685 if (initcall_debug) (gdb) step 682 int count = preempt_count(); (gdb) step 685 if (initcall_debug) (gdb) step 688 ret = fn(); (gdb) step foo_init () at drivers/char/foo.c:119 119 result = register_chrdev(FOO_MAJOR, "scull", &foo_fops); (gdb) step register_chrdev (fops=0x36c <foo_fops>, name=0x150 <Address 0x150 out of bounds>, major=229) at drivers/char/foo.c:119 119 result = register_chrdev(FOO_MAJOR, "scull", &foo_fops); (gdb) step 2125 return __register_chrdev(major, 0, 256, name, fops); (gdb) step __register_chrdev (major=major@entry=229, baseminor=baseminor@entry=0, count=count@entry=256, name=0xbf000299 "scull", name@entry=0x150 <Address 0x150 out of bounds>, fops=0xbf0002d8, fops@entry=0x36c <foo_fops>) at fs/char_dev.c:267 267 { (gdb) step 272 cd = __register_chrdev_region(major, baseminor, count, name); (gdb) step __register_chrdev_region (major=major@entry=229, baseminor=baseminor@entry=0, minorct=minorct@entry=256, name=name@entry=0xbf000299 "scull") at fs/char_dev.c:97 97 { (gdb) step 102 cd = kzalloc(sizeof(struct char_device_struct), GFP_KERNEL); (gdb) c ContinuingNow step into and debug the application,
[root@mini2440 /root]# mknod /dev/foo c 229 0 [root@mini2440 /root]# ./footest [New Thread 488] [Switching to Thread 488] Breakpoint 3, foo_write (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=12, f_pos=0xc2d8ff78) at drivers/char/foo.c:46 46 { (gdb) (gdb) step 50 copy_from_user(drv_buf, buffer, count); (gdb) step copy_from_user (n=12, from=0xbeab9c70, to=0x538 <drv_buf>) at drivers/char/foo.c:50 50 copy_from_user(drv_buf, buffer, count); (gdb) c Continuing.We can see the input string was Hello, there, the expected output should be reversed string ereht ,olleH, however the output was ereho, tlleH. I think below code logic was not correct,
for(i = 0; i<(len>>1); i++, len--) { temp = drv_buf[len-1]; drv_buf[len-1] = drv_buf[i]; drv_buf[i] = temp; }The input len was 12. The wrong logic is as below,
ilenlen>>1status0126i = 0; i < 61115i = 1; i < 52105i = 2; i < 5394i = 3; i < 4Above loop happened 4 times, it should be 6 times. half of 12, so the result was wrongly as ereho, tlleH. The correct logic should be below, and the result shall be ereht ,olleH.
ilen (fixed value)len>>1status0126i = 0; i < 61126i = 1; i < 62126i = 2; i < 63126i = 3; i < 64126i = 4; i < 65126i = 5 i < 6Following below debugging process, the loop stops when i = 4.
Breakpoint 4, do_write () at drivers/char/foo.c:28 28 { (gdb) step 32 for(i = 0; i<(len>>1); i++, len--) (gdb) p i $3 = 0 (gdb) i len Undefined info command: "len". Try "help info". (gdb) p $len $4 = void (gdb) p len Cannot access memory at address 0x938 (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) p i $5 = 0 (gdb) p drv_buf Cannot access memory at address 0x538 (gdb) p drv_buf[0] Cannot access memory at address 0x538 (gdb) p *drv_buf@10 Cannot access memory at address 0x538 (gdb) x/xw 0x538 0x538 <drv_buf>: Cannot access memory at address 0x538 (gdb) list 30 int len = WRT_LEN; 31 char temp; 32 for(i = 0; i<(len>>1); i++, len--) 33 { 34 temp = drv_buf[len-1]; 35 drv_buf[len-1] = drv_buf[i]; 36 drv_buf[i] = temp; 37 } 38 39 } (gdb) p len $6 = 12 (gdb) p *dri_buf No symbol "dri_buf" in current context. (gdb) p *drv_buf Cannot access memory at address 0x538 (gdb) x/xb &drv_buf 0x538 <drv_buf>: Cannot access memory at address 0x538 (gdb) x/xb drv_buf 0x538 <drv_buf>: Cannot access memory at address 0x538 (gdb) x/1xb drv_buf 0x538 <drv_buf>: Cannot access memory at address 0x538 (gdb) p drv_buf[0] Cannot access memory at address 0x538 (gdb) p drv_buf[1] Cannot access memory at address 0x539 (gdb) where #0 do_write () at drivers/char/foo.c:35 #1 0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53 #2 0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428 #3 0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475 #4 0xc000e4a0 in kuser_cmpxchg32_fixup () #5 0xc000e4a0 in kuser_cmpxchg32_fixup () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) whatis drv_buf type = char [1024] (gdb) backtrace #0 do_write () at drivers/char/foo.c:35 #1 0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53 #2 0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428 #3 0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475 #4 0xc000e4a0 in kuser_cmpxchg32_fixup () #5 0xc000e4a0 in kuser_cmpxchg32_fixup () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) p/x 0x538 $7 = 0x538 (gdb) x/i 0x539 <drv_buf+1>: Cannot access memory at address 0x538 (gdb) p tmp No symbol "tmp" in current context. (gdb) l 40 41 42 43 44 45 ssize_t foo_write(struct file *filp, const char *buffer, size_t count, loff_t *f_pos) 46 { 47 if(count > MAX_BUF_LEN) 48 count = MAX_BUF_LEN; 49 (gdb) step 34 temp = drv_buf[len-1]; (gdb) p temp $8 = <optimized out> (gdb) p len $9 = 12 (gdb) p i $10 = 0 (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) step 36 drv_buf[i] = temp; (gdb) step 32 for(i = 0; i<(len>>1); i++, len--) (gdb) p i $11 = 0 (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) p i $12 = 1 (gdb) step 34 temp = drv_buf[len-1]; (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) step 36 drv_buf[i] = temp; (gdb) p i $13 = 1 (gdb) step 32 for(i = 0; i<(len>>1); i++, len--) (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) step 34 temp = drv_buf[len-1]; (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) step 36 drv_buf[i] = temp; (gdb) step ^[[A32 for(i = 0; i<(len>>1); i++, len--) (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) p $14 = 1 (gdb) p i $15 = 3 (gdb) p i $16 = 3 (gdb) p $17 = 3 (gdb) step 34 temp = drv_buf[len-1]; (gdb) step 35 drv_buf[len-1] = drv_buf[i]; (gdb) step 36 drv_buf[i] = temp; (gdb) step 32 for(i = 0; i<(len>>1); i++, len--) (gdb) step 39 } (gdb) p i $18 = 4 (gdb) step foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:55 55 } (gdb) p len No symbol "len" in current context. (gdb) c Continuing. Breakpoint 2, foo_read (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=100, ppos=0xc2d8ff78) at drivers/char/foo.c:59 59 { (gdb) c Continuing.The serial port output was like below,
foo initialized. device open success!. user write data to driver. user read data from driver. Device release. device open success!. user write data to driver. user read data from driver. Device release.Above debugging process has one Cannot access memory at address error, refer to Linux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet, I guess it was the reason.
Note that you can’t disassemble a module function, because the debugger is acting on vmlinux, which doesn’t know about your module. If you try to disassemble a module by address, gdb is most likely to reply “Cannot access memory at xxxx.” For the same reason, you can’t look at data items belonging to a module. They can be read from /dev/mem if you know the address of your variables, but it’s hard to make sense out of raw data extracted from system RAM.
The 101 of ELF files on Linux: Understanding and Analysis Debugging The Linux Kernel Using Gdb Embedded Linux S3C2440 – Kernel Module Linux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet
