firmware-assisted-dump.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381
  1. ======================
  2. Firmware-Assisted Dump
  3. ======================
  4. July 2011
  5. The goal of firmware-assisted dump is to enable the dump of
  6. a crashed system, and to do so from a fully-reset system, and
  7. to minimize the total elapsed time until the system is back
  8. in production use.
  9. - Firmware-Assisted Dump (FADump) infrastructure is intended to replace
  10. the existing phyp assisted dump.
  11. - Fadump uses the same firmware interfaces and memory reservation model
  12. as phyp assisted dump.
  13. - Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
  14. in the ELF format in the same way as kdump. This helps us reuse the
  15. kdump infrastructure for dump capture and filtering.
  16. - Unlike phyp dump, userspace tool does not need to refer any sysfs
  17. interface while reading /proc/vmcore.
  18. - Unlike phyp dump, FADump allows user to release all the memory reserved
  19. for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
  20. - Once enabled through kernel boot parameter, FADump can be
  21. started/stopped through /sys/kernel/fadump_registered interface (see
  22. sysfs files section below) and can be easily integrated with kdump
  23. service start/stop init scripts.
  24. Comparing with kdump or other strategies, firmware-assisted
  25. dump offers several strong, practical advantages:
  26. - Unlike kdump, the system has been reset, and loaded
  27. with a fresh copy of the kernel. In particular,
  28. PCI and I/O devices have been reinitialized and are
  29. in a clean, consistent state.
  30. - Once the dump is copied out, the memory that held the dump
  31. is immediately available to the running kernel. And therefore,
  32. unlike kdump, FADump doesn't need a 2nd reboot to get back
  33. the system to the production configuration.
  34. The above can only be accomplished by coordination with,
  35. and assistance from the Power firmware. The procedure is
  36. as follows:
  37. - The first kernel registers the sections of memory with the
  38. Power firmware for dump preservation during OS initialization.
  39. These registered sections of memory are reserved by the first
  40. kernel during early boot.
  41. - When system crashes, the Power firmware will copy the registered
  42. low memory regions (boot memory) from source to destination area.
  43. It will also save hardware PTE's.
  44. NOTE:
  45. The term 'boot memory' means size of the low memory chunk
  46. that is required for a kernel to boot successfully when
  47. booted with restricted memory. By default, the boot memory
  48. size will be the larger of 5% of system RAM or 256MB.
  49. Alternatively, user can also specify boot memory size
  50. through boot parameter 'crashkernel=' which will override
  51. the default calculated size. Use this option if default
  52. boot memory size is not sufficient for second kernel to
  53. boot successfully. For syntax of crashkernel= parameter,
  54. refer to Documentation/admin-guide/kdump/kdump.rst. If any
  55. offset is provided in crashkernel= parameter, it will be
  56. ignored as FADump uses a predefined offset to reserve memory
  57. for boot memory dump preservation in case of a crash.
  58. - After the low memory (boot memory) area has been saved, the
  59. firmware will reset PCI and other hardware state. It will
  60. *not* clear the RAM. It will then launch the bootloader, as
  61. normal.
  62. - The freshly booted kernel will notice that there is a new node
  63. (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
  64. on OPAL platform) in the device tree, indicating that
  65. there is crash data available from a previous boot. During
  66. the early boot OS will reserve rest of the memory above
  67. boot memory size effectively booting with restricted memory
  68. size. This will make sure that this kernel (also, referred
  69. to as second kernel or capture kernel) will not touch any
  70. of the dump memory area.
  71. - User-space tools will read /proc/vmcore to obtain the contents
  72. of memory, which holds the previous crashed kernel dump in ELF
  73. format. The userspace tools may copy this info to disk, or
  74. network, nas, san, iscsi, etc. as desired.
  75. - Once the userspace tool is done saving dump, it will echo
  76. '1' to /sys/kernel/fadump_release_mem to release the reserved
  77. memory back to general use, except the memory required for
  78. next firmware-assisted dump registration.
  79. e.g.::
  80. # echo 1 > /sys/kernel/fadump_release_mem
  81. Please note that the firmware-assisted dump feature
  82. is only available on POWER6 and above systems on pSeries
  83. (PowerVM) platform and POWER9 and above systems with OP940
  84. or later firmware versions on PowerNV (OPAL) platform.
  85. Note that, OPAL firmware exports ibm,opal/dump node when
  86. FADump is supported on PowerNV platform.
  87. On OPAL based machines, system first boots into an intermittent
  88. kernel (referred to as petitboot kernel) before booting into the
  89. capture kernel. This kernel would have minimal kernel and/or
  90. userspace support to process crash data. Such kernel needs to
  91. preserve previously crash'ed kernel's memory for the subsequent
  92. capture kernel boot to process this crash data. Kernel config
  93. option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
  94. to ensure that crash data is preserved to process later.
  95. -- On OPAL based machines (PowerNV), if the kernel is build with
  96. CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
  97. exported as /sys/firmware/opal/mpipl/core file. This procfs file is
  98. helpful in debugging OPAL crashes with GDB. The kernel memory
  99. used for exporting this procfs file can be released by echo'ing
  100. '1' to /sys/firmware/opal/mpipl/release_core node.
  101. e.g.
  102. # echo 1 > /sys/firmware/opal/mpipl/release_core
  103. Implementation details:
  104. -----------------------
  105. During boot, a check is made to see if firmware supports
  106. this feature on that particular machine. If it does, then
  107. we check to see if an active dump is waiting for us. If yes
  108. then everything but boot memory size of RAM is reserved during
  109. early boot (See Fig. 2). This area is released once we finish
  110. collecting the dump from user land scripts (e.g. kdump scripts)
  111. that are run. If there is dump data, then the
  112. /sys/kernel/fadump_release_mem file is created, and the reserved
  113. memory is held.
  114. If there is no waiting dump data, then only the memory required to
  115. hold CPU state, HPTE region, boot memory dump, FADump header and
  116. elfcore header, is usually reserved at an offset greater than boot
  117. memory size (see Fig. 1). This area is *not* released: this region
  118. will be kept permanently reserved, so that it can act as a receptacle
  119. for a copy of the boot memory content in addition to CPU state and
  120. HPTE region, in the case a crash does occur.
  121. Since this reserved memory area is used only after the system crash,
  122. there is no point in blocking this significant chunk of memory from
  123. production kernel. Hence, the implementation uses the Linux kernel's
  124. Contiguous Memory Allocator (CMA) for memory reservation if CMA is
  125. configured for kernel. With CMA reservation this memory will be
  126. available for applications to use it, while kernel is prevented from
  127. using it. With this FADump will still be able to capture all of the
  128. kernel memory and most of the user space memory except the user pages
  129. that were present in CMA region::
  130. o Memory Reservation during first kernel
  131. Low memory Top of memory
  132. 0 boot memory size |<--- Reserved dump area --->| |
  133. | | | Permanent Reservation | |
  134. V V | | V
  135. +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  136. | | |///|////| DUMP | HDR | ELF |////| |
  137. +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  138. | ^ ^ ^ ^ ^
  139. | | | | | |
  140. \ CPU HPTE / | |
  141. ------------------------------ | |
  142. Boot memory content gets transferred | |
  143. to reserved area by firmware at the | |
  144. time of crash. | |
  145. FADump Header |
  146. (meta area) |
  147. |
  148. |
  149. Metadata: This area holds a metadata struture whose
  150. address is registered with f/w and retrieved in the
  151. second kernel after crash, on platforms that support
  152. tags (OPAL). Having such structure with info needed
  153. to process the crashdump eases dump capture process.
  154. Fig. 1
  155. o Memory Reservation during second kernel after crash
  156. Low memory Top of memory
  157. 0 boot memory size |
  158. | |<------------ Crash preserved area ------------>|
  159. V V |<--- Reserved dump area --->| |
  160. +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  161. | | |///|////| DUMP | HDR | ELF |////| |
  162. +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  163. | |
  164. V V
  165. Used by second /proc/vmcore
  166. kernel to boot
  167. +---+
  168. |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
  169. +---+ figures are not always present. For example, OPAL platform
  170. does not have CPU & HPTE regions while Metadata region is
  171. not supported on pSeries currently.
  172. Fig. 2
  173. Currently the dump will be copied from /proc/vmcore to a new file upon
  174. user intervention. The dump data available through /proc/vmcore will be
  175. in ELF format. Hence the existing kdump infrastructure (kdump scripts)
  176. to save the dump works fine with minor modifications. KDump scripts on
  177. major Distro releases have already been modified to work seemlessly (no
  178. user intervention in saving the dump) when FADump is used, instead of
  179. KDump, as dump mechanism.
  180. The tools to examine the dump will be same as the ones
  181. used for kdump.
  182. How to enable firmware-assisted dump (FADump):
  183. ----------------------------------------------
  184. 1. Set config option CONFIG_FA_DUMP=y and build kernel.
  185. 2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
  186. By default, FADump reserved memory will be initialized as CMA area.
  187. Alternatively, user can boot linux kernel with 'fadump=nocma' to
  188. prevent FADump to use CMA.
  189. 3. Optionally, user can also set 'crashkernel=' kernel cmdline
  190. to specify size of the memory to reserve for boot memory dump
  191. preservation.
  192. NOTE:
  193. 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
  194. use 'crashkernel=' to specify size of the memory to reserve
  195. for boot memory dump preservation.
  196. 2. If firmware-assisted dump fails to reserve memory then it
  197. will fallback to existing kdump mechanism if 'crashkernel='
  198. option is set at kernel cmdline.
  199. 3. if user wants to capture all of user space memory and ok with
  200. reserved memory not available to production system, then
  201. 'fadump=nocma' kernel parameter can be used to fallback to
  202. old behaviour.
  203. Sysfs/debugfs files:
  204. --------------------
  205. Firmware-assisted dump feature uses sysfs file system to hold
  206. the control files and debugfs file to display memory reserved region.
  207. Here is the list of files under kernel sysfs:
  208. /sys/kernel/fadump_enabled
  209. This is used to display the FADump status.
  210. - 0 = FADump is disabled
  211. - 1 = FADump is enabled
  212. This interface can be used by kdump init scripts to identify if
  213. FADump is enabled in the kernel and act accordingly.
  214. /sys/kernel/fadump_registered
  215. This is used to display the FADump registration status as well
  216. as to control (start/stop) the FADump registration.
  217. - 0 = FADump is not registered.
  218. - 1 = FADump is registered and ready to handle system crash.
  219. To register FADump echo 1 > /sys/kernel/fadump_registered and
  220. echo 0 > /sys/kernel/fadump_registered for un-register and stop the
  221. FADump. Once the FADump is un-registered, the system crash will not
  222. be handled and vmcore will not be captured. This interface can be
  223. easily integrated with kdump service start/stop.
  224. /sys/kernel/fadump/mem_reserved
  225. This is used to display the memory reserved by FADump for saving the
  226. crash dump.
  227. /sys/kernel/fadump_release_mem
  228. This file is available only when FADump is active during
  229. second kernel. This is used to release the reserved memory
  230. region that are held for saving crash dump. To release the
  231. reserved memory echo 1 to it::
  232. echo 1 > /sys/kernel/fadump_release_mem
  233. After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
  234. file will change to reflect the new memory reservations.
  235. The existing userspace tools (kdump infrastructure) can be easily
  236. enhanced to use this interface to release the memory reserved for
  237. dump and continue without 2nd reboot.
  238. Note: /sys/kernel/fadump_release_opalcore sysfs has moved to
  239. /sys/firmware/opal/mpipl/release_core
  240. /sys/firmware/opal/mpipl/release_core
  241. This file is available only on OPAL based machines when FADump is
  242. active during capture kernel. This is used to release the memory
  243. used by the kernel to export /sys/firmware/opal/mpipl/core file. To
  244. release this memory, echo '1' to it:
  245. echo 1 > /sys/firmware/opal/mpipl/release_core
  246. Note: The following FADump sysfs files are deprecated.
  247. +----------------------------------+--------------------------------+
  248. | Deprecated | Alternative |
  249. +----------------------------------+--------------------------------+
  250. | /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled |
  251. +----------------------------------+--------------------------------+
  252. | /sys/kernel/fadump_registered | /sys/kernel/fadump/registered |
  253. +----------------------------------+--------------------------------+
  254. | /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem |
  255. +----------------------------------+--------------------------------+
  256. Here is the list of files under powerpc debugfs:
  257. (Assuming debugfs is mounted on /sys/kernel/debug directory.)
  258. /sys/kernel/debug/powerpc/fadump_region
  259. This file shows the reserved memory regions if FADump is
  260. enabled otherwise this file is empty. The output format
  261. is::
  262. <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
  263. and for kernel DUMP region is:
  264. DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes
  265. e.g.
  266. Contents when FADump is registered during first kernel::
  267. # cat /sys/kernel/debug/powerpc/fadump_region
  268. CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
  269. HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
  270. DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
  271. Contents when FADump is active during second kernel::
  272. # cat /sys/kernel/debug/powerpc/fadump_region
  273. CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
  274. HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
  275. DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
  276. : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
  277. NOTE:
  278. Please refer to Documentation/filesystems/debugfs.rst on
  279. how to mount the debugfs filesystem.
  280. TODO:
  281. -----
  282. - Need to come up with the better approach to find out more
  283. accurate boot memory size that is required for a kernel to
  284. boot successfully when booted with restricted memory.
  285. - The FADump implementation introduces a FADump crash info structure
  286. in the scratch area before the ELF core header. The idea of introducing
  287. this structure is to pass some important crash info data to the second
  288. kernel which will help second kernel to populate ELF core header with
  289. correct data before it gets exported through /proc/vmcore. The current
  290. design implementation does not address a possibility of introducing
  291. additional fields (in future) to this structure without affecting
  292. compatibility. Need to come up with the better approach to address this.
  293. The possible approaches are:
  294. 1. Introduce version field for version tracking, bump up the version
  295. whenever a new field is added to the structure in future. The version
  296. field can be used to find out what fields are valid for the current
  297. version of the structure.
  298. 2. Reserve the area of predefined size (say PAGE_SIZE) for this
  299. structure and have unused area as reserved (initialized to zero)
  300. for future field additions.
  301. The advantage of approach 1 over 2 is we don't need to reserve extra space.
  302. Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
  303. This document is based on the original documentation written for phyp
  304. assisted dump by Linas Vepstas and Manish Ahuja.