suspend-and-cpuhotplug.rst 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287
  1. ====================================================================
  2. Interaction of Suspend code (S3) with the CPU hotplug infrastructure
  3. ====================================================================
  4. (C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
  5. I. Differences between CPU hotplug and Suspend-to-RAM
  6. ======================================================
  7. How does the regular CPU hotplug code differ from how the Suspend-to-RAM
  8. infrastructure uses it internally? And where do they share common code?
  9. Well, a picture is worth a thousand words... So ASCII art follows :-)
  10. [This depicts the current design in the kernel, and focusses only on the
  11. interactions involving the freezer and CPU hotplug and also tries to explain
  12. the locking involved. It outlines the notifications involved as well.
  13. But please note that here, only the call paths are illustrated, with the aim
  14. of describing where they take different paths and where they share code.
  15. What happens when regular CPU hotplug and Suspend-to-RAM race with each other
  16. is not depicted here.]
  17. On a high level, the suspend-resume cycle goes like this::
  18. |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw |
  19. |tasks | | cpus | | | | cpus | |tasks|
  20. More details follow::
  21. Suspend call path
  22. -----------------
  23. Write 'mem' to
  24. /sys/power/state
  25. sysfs file
  26. |
  27. v
  28. Acquire system_transition_mutex lock
  29. |
  30. v
  31. Send PM_SUSPEND_PREPARE
  32. notifications
  33. |
  34. v
  35. Freeze tasks
  36. |
  37. |
  38. v
  39. freeze_secondary_cpus()
  40. /* start */
  41. |
  42. v
  43. Acquire cpu_add_remove_lock
  44. |
  45. v
  46. Iterate over CURRENTLY
  47. online CPUs
  48. |
  49. |
  50. | ----------
  51. v | L
  52. ======> _cpu_down() |
  53. | [This takes cpuhotplug.lock |
  54. Common | before taking down the CPU |
  55. code | and releases it when done] | O
  56. | While it is at it, notifications |
  57. | are sent when notable events occur, |
  58. ======> by running all registered callbacks. |
  59. | | O
  60. | |
  61. | |
  62. v |
  63. Note down these cpus in | P
  64. frozen_cpus mask ----------
  65. |
  66. v
  67. Disable regular cpu hotplug
  68. by increasing cpu_hotplug_disabled
  69. |
  70. v
  71. Release cpu_add_remove_lock
  72. |
  73. v
  74. /* freeze_secondary_cpus() complete */
  75. |
  76. v
  77. Do suspend
  78. Resuming back is likewise, with the counterparts being (in the order of
  79. execution during resume):
  80. * thaw_secondary_cpus() which involves::
  81. | Acquire cpu_add_remove_lock
  82. | Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
  83. | Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
  84. | Release cpu_add_remove_lock
  85. v
  86. * thaw tasks
  87. * send PM_POST_SUSPEND notifications
  88. * Release system_transition_mutex lock.
  89. It is to be noted here that the system_transition_mutex lock is acquired at the
  90. very beginning, when we are just starting out to suspend, and then released only
  91. after the entire cycle is complete (i.e., suspend + resume).
  92. ::
  93. Regular CPU hotplug call path
  94. -----------------------------
  95. Write 0 (or 1) to
  96. /sys/devices/system/cpu/cpu*/online
  97. sysfs file
  98. |
  99. |
  100. v
  101. cpu_down()
  102. |
  103. v
  104. Acquire cpu_add_remove_lock
  105. |
  106. v
  107. If cpu_hotplug_disabled > 0
  108. return gracefully
  109. |
  110. |
  111. v
  112. ======> _cpu_down()
  113. | [This takes cpuhotplug.lock
  114. Common | before taking down the CPU
  115. code | and releases it when done]
  116. | While it is at it, notifications
  117. | are sent when notable events occur,
  118. ======> by running all registered callbacks.
  119. |
  120. |
  121. v
  122. Release cpu_add_remove_lock
  123. [That's it!, for
  124. regular CPU hotplug]
  125. So, as can be seen from the two diagrams (the parts marked as "Common code"),
  126. regular CPU hotplug and the suspend code path converge at the _cpu_down() and
  127. _cpu_up() functions. They differ in the arguments passed to these functions,
  128. in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen'
  129. argument. But during suspend, since the tasks are already frozen by the time
  130. the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called
  131. with the 'tasks_frozen' argument set to 1.
  132. [See below for some known issues regarding this.]
  133. Important files and functions/entry points:
  134. -------------------------------------------
  135. - kernel/power/process.c : freeze_processes(), thaw_processes()
  136. - kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish()
  137. - kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](),
  138. [disable|enable]_nonboot_cpus()
  139. II. What are the issues involved in CPU hotplug?
  140. ------------------------------------------------
  141. There are some interesting situations involving CPU hotplug and microcode
  142. update on the CPUs, as discussed below:
  143. [Please bear in mind that the kernel requests the microcode images from
  144. userspace, using the request_firmware() function defined in
  145. drivers/base/firmware_loader/main.c]
  146. a. When all the CPUs are identical:
  147. This is the most common situation and it is quite straightforward: we want
  148. to apply the same microcode revision to each of the CPUs.
  149. To give an example of x86, the collect_cpu_info() function defined in
  150. arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU
  151. and thereby in applying the correct microcode revision to it.
  152. But note that the kernel does not maintain a common microcode image for the
  153. all CPUs, in order to handle case 'b' described below.
  154. b. When some of the CPUs are different than the rest:
  155. In this case since we probably need to apply different microcode revisions
  156. to different CPUs, the kernel maintains a copy of the correct microcode
  157. image for each CPU (after appropriate CPU type/model discovery using
  158. functions such as collect_cpu_info()).
  159. c. When a CPU is physically hot-unplugged and a new (and possibly different
  160. type of) CPU is hot-plugged into the system:
  161. In the current design of the kernel, whenever a CPU is taken offline during
  162. a regular CPU hotplug operation, upon receiving the CPU_DEAD notification
  163. (which is sent by the CPU hotplug code), the microcode update driver's
  164. callback for that event reacts by freeing the kernel's copy of the
  165. microcode image for that CPU.
  166. Hence, when a new CPU is brought online, since the kernel finds that it
  167. doesn't have the microcode image, it does the CPU type/model discovery
  168. afresh and then requests the userspace for the appropriate microcode image
  169. for that CPU, which is subsequently applied.
  170. For example, in x86, the mc_cpu_callback() function (which is the microcode
  171. update driver's callback registered for CPU hotplug events) calls
  172. microcode_update_cpu() which would call microcode_init_cpu() in this case,
  173. instead of microcode_resume_cpu() when it finds that the kernel doesn't
  174. have a valid microcode image. This ensures that the CPU type/model
  175. discovery is performed and the right microcode is applied to the CPU after
  176. getting it from userspace.
  177. d. Handling microcode update during suspend/hibernate:
  178. Strictly speaking, during a CPU hotplug operation which does not involve
  179. physically removing or inserting CPUs, the CPUs are not actually powered
  180. off during a CPU offline. They are just put to the lowest C-states possible.
  181. Hence, in such a case, it is not really necessary to re-apply microcode
  182. when the CPUs are brought back online, since they wouldn't have lost the
  183. image during the CPU offline operation.
  184. This is the usual scenario encountered during a resume after a suspend.
  185. However, in the case of hibernation, since all the CPUs are completely
  186. powered off, during restore it becomes necessary to apply the microcode
  187. images to all the CPUs.
  188. [Note that we don't expect someone to physically pull out nodes and insert
  189. nodes with a different type of CPUs in-between a suspend-resume or a
  190. hibernate/restore cycle.]
  191. In the current design of the kernel however, during a CPU offline operation
  192. as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set),
  193. the existing copy of microcode image in the kernel is not freed up.
  194. And during the CPU online operations (during resume/restore), since the
  195. kernel finds that it already has copies of the microcode images for all the
  196. CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU
  197. type/model and the need for validating whether the microcode revisions are
  198. right for the CPUs or not (due to the above assumption that physical CPU
  199. hotplug will not be done in-between suspend/resume or hibernate/restore
  200. cycles).
  201. III. Known problems
  202. ===================
  203. Are there any known problems when regular CPU hotplug and suspend race
  204. with each other?
  205. Yes, they are listed below:
  206. 1. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to
  207. the _cpu_down() and _cpu_up() functions is *always* 0.
  208. This might not reflect the true current state of the system, since the
  209. tasks could have been frozen by an out-of-band event such as a suspend
  210. operation in progress. Hence, the cpuhp_tasks_frozen variable will not
  211. reflect the frozen state and the CPU hotplug callbacks which evaluate
  212. that variable might execute the wrong code path.
  213. 2. If a regular CPU hotplug stress test happens to race with the freezer due
  214. to a suspend operation in progress at the same time, then we could hit the
  215. situation described below:
  216. * A regular cpu online operation continues its journey from userspace
  217. into the kernel, since the freezing has not yet begun.
  218. * Then freezer gets to work and freezes userspace.
  219. * If cpu online has not yet completed the microcode update stuff by now,
  220. it will now start waiting on the frozen userspace in the
  221. TASK_UNINTERRUPTIBLE state, in order to get the microcode image.
  222. * Now the freezer continues and tries to freeze the remaining tasks. But
  223. due to this wait mentioned above, the freezer won't be able to freeze
  224. the cpu online hotplug task and hence freezing of tasks fails.
  225. As a result of this task freezing failure, the suspend operation gets
  226. aborted.