nmi_watchdog.txt 4.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081
  1. [NMI watchdog is available for x86 and x86-64 architectures]
  2. Is your system locking up unpredictably? No keyboard activity, just
  3. a frustrating complete hard lockup? Do you want to help us debugging
  4. such lockups? If all yes then this document is definitely for you.
  5. On many x86/x86-64 type hardware there is a feature that enables
  6. us to generate 'watchdog NMI interrupts'. (NMI: Non Maskable Interrupt
  7. which get executed even if the system is otherwise locked up hard).
  8. This can be used to debug hard kernel lockups. By executing periodic
  9. NMI interrupts, the kernel can monitor whether any CPU has locked up,
  10. and print out debugging messages if so.
  11. In order to use the NMI watchdog, you need to have APIC support in your
  12. kernel. For SMP kernels, APIC support gets compiled in automatically. For
  13. UP, enable either CONFIG_X86_UP_APIC (Processor type and features -> Local
  14. APIC support on uniprocessors) or CONFIG_X86_UP_IOAPIC (Processor type and
  15. features -> IO-APIC support on uniprocessors) in your kernel config.
  16. CONFIG_X86_UP_APIC is for uniprocessor machines without an IO-APIC.
  17. CONFIG_X86_UP_IOAPIC is for uniprocessor with an IO-APIC. [Note: certain
  18. kernel debugging options, such as Kernel Stack Meter or Kernel Tracer,
  19. may implicitly disable the NMI watchdog.]
  20. For x86-64, the needed APIC is always compiled in, and the NMI watchdog is
  21. always enabled with I/O-APIC mode (nmi_watchdog=1). Currently, local APIC
  22. mode (nmi_watchdog=2) does not work on x86-64.
  23. Using local APIC (nmi_watchdog=2) needs the first performance register, so
  24. you can't use it for other purposes (such as high precision performance
  25. profiling.) However, at least oprofile and the perfctr driver disable the
  26. local APIC NMI watchdog automatically.
  27. To actually enable the NMI watchdog, use the 'nmi_watchdog=N' boot
  28. parameter. Eg. the relevant lilo.conf entry:
  29. append="nmi_watchdog=1"
  30. For SMP machines and UP machines with an IO-APIC use nmi_watchdog=1.
  31. For UP machines without an IO-APIC use nmi_watchdog=2, this only works
  32. for some processor types. If in doubt, boot with nmi_watchdog=1 and
  33. check the NMI count in /proc/interrupts; if the count is zero then
  34. reboot with nmi_watchdog=2 and check the NMI count. If it is still
  35. zero then log a problem, you probably have a processor that needs to be
  36. added to the nmi code.
  37. A 'lockup' is the following scenario: if any CPU in the system does not
  38. execute the period local timer interrupt for more than 5 seconds, then
  39. the NMI handler generates an oops and kills the process. This
  40. 'controlled crash' (and the resulting kernel messages) can be used to
  41. debug the lockup. Thus whenever the lockup happens, wait 5 seconds and
  42. the oops will show up automatically. If the kernel produces no messages
  43. then the system has crashed so hard (eg. hardware-wise) that either it
  44. cannot even accept NMI interrupts, or the crash has made the kernel
  45. unable to print messages.
  46. Be aware that when using local APIC, the frequency of NMI interrupts
  47. it generates, depends on the system load. The local APIC NMI watchdog,
  48. lacking a better source, uses the "cycles unhalted" event. As you may
  49. guess it doesn't tick when the CPU is in the halted state (which happens
  50. when the system is idle), but if your system locks up on anything but the
  51. "hlt" processor instruction, the watchdog will trigger very soon as the
  52. "cycles unhalted" event will happen every clock tick. If it locks up on
  53. "hlt", then you are out of luck -- the event will not happen at all and the
  54. watchdog won't trigger. This is a shortcoming of the local APIC watchdog
  55. -- unfortunately there is no "clock ticks" event that would work all the
  56. time. The I/O APIC watchdog is driven externally and has no such shortcoming.
  57. But its NMI frequency is much higher, resulting in a more significant hit
  58. to the overall system performance.
  59. NOTE: starting with 2.4.2-ac18 the NMI-oopser is disabled by default,
  60. you have to enable it with a boot time parameter. Prior to 2.4.2-ac18
  61. the NMI-oopser is enabled unconditionally on x86 SMP boxes.
  62. On x86-64 the NMI oopser is on by default. On 64bit Intel CPUs
  63. it uses IO-APIC by default and on AMD it uses local APIC.
  64. [ feel free to send bug reports, suggestions and patches to
  65. Ingo Molnar <mingo@redhat.com> or the Linux SMP mailing
  66. list at <linux-smp@vger.kernel.org> ]