gdb-kernel-debugging.rst 6.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175
  1. .. highlight:: none
  2. Debugging kernel and modules via gdb
  3. ====================================
  4. The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
  5. interfaces allow to debug the Linux kernel and its modules during runtime
  6. using gdb. Gdb comes with a powerful scripting interface for python. The
  7. kernel provides a collection of helper scripts that can simplify typical
  8. kernel debugging steps. This is a short tutorial about how to enable and use
  9. them. It focuses on QEMU/KVM virtual machines as target, but the examples can
  10. be transferred to the other gdb stubs as well.
  11. Requirements
  12. ------------
  13. - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
  14. for distributions)
  15. Setup
  16. -----
  17. - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
  18. www.qemu.org for more details). For cross-development,
  19. https://landley.net/aboriginal/bin keeps a pool of machine images and
  20. toolchains that can be helpful to start from.
  21. - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
  22. CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
  23. CONFIG_FRAME_POINTER, keep it enabled.
  24. - Install that kernel on the guest, turn off KASLR if necessary by adding
  25. "nokaslr" to the kernel command line.
  26. Alternatively, QEMU allows to boot the kernel directly using -kernel,
  27. -append, -initrd command line switches. This is generally only useful if
  28. you do not depend on modules. See QEMU documentation for more details on
  29. this mode. In this case, you should build the kernel with
  30. CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
  31. - Enable the gdb stub of QEMU/KVM, either
  32. - at VM startup time by appending "-s" to the QEMU command line
  33. or
  34. - during runtime by issuing "gdbserver" from the QEMU monitor
  35. console
  36. - cd /path/to/linux-build
  37. - Start gdb: gdb vmlinux
  38. Note: Some distros may restrict auto-loading of gdb scripts to known safe
  39. directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
  40. add-auto-load-safe-path /path/to/linux-build
  41. to ~/.gdbinit. See gdb help for more details.
  42. - Attach to the booted guest::
  43. (gdb) target remote :1234
  44. Examples of using the Linux-provided gdb helpers
  45. ------------------------------------------------
  46. - Load module (and main kernel) symbols::
  47. (gdb) lx-symbols
  48. loading vmlinux
  49. scanning for modules in /home/user/linux/build
  50. loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
  51. loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
  52. loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
  53. loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
  54. loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
  55. ...
  56. loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
  57. - Set a breakpoint on some not yet loaded module function, e.g.::
  58. (gdb) b btrfs_init_sysfs
  59. Function "btrfs_init_sysfs" not defined.
  60. Make breakpoint pending on future shared library load? (y or [n]) y
  61. Breakpoint 1 (btrfs_init_sysfs) pending.
  62. - Continue the target::
  63. (gdb) c
  64. - Load the module on the target and watch the symbols being loaded as well as
  65. the breakpoint hit::
  66. loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
  67. loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
  68. loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
  69. loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
  70. Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
  71. 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
  72. - Dump the log buffer of the target kernel::
  73. (gdb) lx-dmesg
  74. [ 0.000000] Initializing cgroup subsys cpuset
  75. [ 0.000000] Initializing cgroup subsys cpu
  76. [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
  77. [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
  78. [ 0.000000] e820: BIOS-provided physical RAM map:
  79. [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
  80. [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
  81. ....
  82. - Examine fields of the current task struct::
  83. (gdb) p $lx_current().pid
  84. $1 = 4998
  85. (gdb) p $lx_current().comm
  86. $2 = "modprobe\000\000\000\000\000\000\000"
  87. - Make use of the per-cpu function for the current or a specified CPU::
  88. (gdb) p $lx_per_cpu("runqueues").nr_running
  89. $3 = 1
  90. (gdb) p $lx_per_cpu("runqueues", 2).nr_running
  91. $4 = 0
  92. - Dig into hrtimers using the container_of helper::
  93. (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
  94. (gdb) p *$container_of($next, "struct hrtimer", "node")
  95. $5 = {
  96. node = {
  97. node = {
  98. __rb_parent_color = 18446612133355256072,
  99. rb_right = 0x0 <irq_stack_union>,
  100. rb_left = 0x0 <irq_stack_union>
  101. },
  102. expires = {
  103. tv64 = 1835268000000
  104. }
  105. },
  106. _softexpires = {
  107. tv64 = 1835268000000
  108. },
  109. function = 0xffffffff81078232 <tick_sched_timer>,
  110. base = 0xffff88003fd0d6f0,
  111. state = 1,
  112. start_pid = 0,
  113. start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
  114. start_comm = "swapper/2\000\000\000\000\000\000"
  115. }
  116. List of commands and functions
  117. ------------------------------
  118. The number of commands and convenience functions may evolve over the time,
  119. this is just a snapshot of the initial version::
  120. (gdb) apropos lx
  121. function lx_current -- Return current task
  122. function lx_module -- Find module by name and return the module variable
  123. function lx_per_cpu -- Return per-cpu variable
  124. function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
  125. function lx_thread_info -- Calculate Linux thread_info from task variable
  126. lx-dmesg -- Print Linux kernel log buffer
  127. lx-lsmod -- List currently loaded modules
  128. lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
  129. Detailed help can be obtained via "help <command-name>" for commands and "help
  130. function <function-name>" for convenience functions.