freezing-of-tasks.rst 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245
  1. =================
  2. Freezing of tasks
  3. =================
  4. (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
  5. I. What is the freezing of tasks?
  6. =================================
  7. The freezing of tasks is a mechanism by which user space processes and some
  8. kernel threads are controlled during hibernation or system-wide suspend (on some
  9. architectures).
  10. II. How does it work?
  11. =====================
  12. There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
  13. and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have
  14. PF_NOFREEZE unset (all user space processes and some kernel threads) are
  15. regarded as 'freezable' and treated in a special way before the system enters a
  16. suspend state as well as before a hibernation image is created (in what follows
  17. we only consider hibernation, but the description also applies to suspend).
  18. Namely, as the first step of the hibernation procedure the function
  19. freeze_processes() (defined in kernel/power/process.c) is called. A system-wide
  20. variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
  21. whether the system is to undergo a freezing operation. And freeze_processes()
  22. sets this variable. After this, it executes try_to_freeze_tasks() that sends a
  23. fake signal to all user space processes, and wakes up all the kernel threads.
  24. All freezable tasks must react to that by calling try_to_freeze(), which
  25. results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
  26. the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
  27. it loop until PF_FROZEN is cleared for it. Then, we say that the task is
  28. 'frozen' and therefore the set of functions handling this mechanism is referred
  29. to as 'the freezer' (these functions are defined in kernel/power/process.c,
  30. kernel/freezer.c & include/linux/freezer.h). User space processes are generally
  31. frozen before kernel threads.
  32. __refrigerator() must not be called directly. Instead, use the
  33. try_to_freeze() function (defined in include/linux/freezer.h), that checks
  34. if the task is to be frozen and makes the task enter __refrigerator().
  35. For user space processes try_to_freeze() is called automatically from the
  36. signal-handling code, but the freezable kernel threads need to call it
  37. explicitly in suitable places or use the wait_event_freezable() or
  38. wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
  39. that combine interruptible sleep with checking if the task is to be frozen and
  40. calling try_to_freeze(). The main loop of a freezable kernel thread may look
  41. like the following one::
  42. set_freezable();
  43. do {
  44. hub_events();
  45. wait_event_freezable(khubd_wait,
  46. !list_empty(&hub_event_list) ||
  47. kthread_should_stop());
  48. } while (!kthread_should_stop() || !list_empty(&hub_event_list));
  49. (from drivers/usb/core/hub.c::hub_thread()).
  50. If a freezable kernel thread fails to call try_to_freeze() after the freezer has
  51. initiated a freezing operation, the freezing of tasks will fail and the entire
  52. hibernation operation will be cancelled. For this reason, freezable kernel
  53. threads must call try_to_freeze() somewhere or use one of the
  54. wait_event_freezable() and wait_event_freezable_timeout() macros.
  55. After the system memory state has been restored from a hibernation image and
  56. devices have been reinitialized, the function thaw_processes() is called in
  57. order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that
  58. have been frozen leave __refrigerator() and continue running.
  59. Rationale behind the functions dealing with freezing and thawing of tasks
  60. -------------------------------------------------------------------------
  61. freeze_processes():
  62. - freezes only userspace tasks
  63. freeze_kernel_threads():
  64. - freezes all tasks (including kernel threads) because we can't freeze
  65. kernel threads without freezing userspace tasks
  66. thaw_kernel_threads():
  67. - thaws only kernel threads; this is particularly useful if we need to do
  68. anything special in between thawing of kernel threads and thawing of
  69. userspace tasks, or if we want to postpone the thawing of userspace tasks
  70. thaw_processes():
  71. - thaws all tasks (including kernel threads) because we can't thaw userspace
  72. tasks without thawing kernel threads
  73. III. Which kernel threads are freezable?
  74. ========================================
  75. Kernel threads are not freezable by default. However, a kernel thread may clear
  76. PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
  77. directly is not allowed). From this point it is regarded as freezable
  78. and must call try_to_freeze() in a suitable place.
  79. IV. Why do we do that?
  80. ======================
  81. Generally speaking, there is a couple of reasons to use the freezing of tasks:
  82. 1. The principal reason is to prevent filesystems from being damaged after
  83. hibernation. At the moment we have no simple means of checkpointing
  84. filesystems, so if there are any modifications made to filesystem data and/or
  85. metadata on disks, we cannot bring them back to the state from before the
  86. modifications. At the same time each hibernation image contains some
  87. filesystem-related information that must be consistent with the state of the
  88. on-disk data and metadata after the system memory state has been restored
  89. from the image (otherwise the filesystems will be damaged in a nasty way,
  90. usually making them almost impossible to repair). We therefore freeze
  91. tasks that might cause the on-disk filesystems' data and metadata to be
  92. modified after the hibernation image has been created and before the
  93. system is finally powered off. The majority of these are user space
  94. processes, but if any of the kernel threads may cause something like this
  95. to happen, they have to be freezable.
  96. 2. Next, to create the hibernation image we need to free a sufficient amount of
  97. memory (approximately 50% of available RAM) and we need to do that before
  98. devices are deactivated, because we generally need them for swapping out.
  99. Then, after the memory for the image has been freed, we don't want tasks
  100. to allocate additional memory and we prevent them from doing that by
  101. freezing them earlier. [Of course, this also means that device drivers
  102. should not allocate substantial amounts of memory from their .suspend()
  103. callbacks before hibernation, but this is a separate issue.]
  104. 3. The third reason is to prevent user space processes and some kernel threads
  105. from interfering with the suspending and resuming of devices. A user space
  106. process running on a second CPU while we are suspending devices may, for
  107. example, be troublesome and without the freezing of tasks we would need some
  108. safeguards against race conditions that might occur in such a case.
  109. Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
  110. of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):
  111. "RJW:> Why we freeze tasks at all or why we freeze kernel threads?
  112. Linus: In many ways, 'at all'.
  113. I **do** realize the IO request queue issues, and that we cannot actually do
  114. s2ram with some devices in the middle of a DMA. So we want to be able to
  115. avoid *that*, there's no question about that. And I suspect that stopping
  116. user threads and then waiting for a sync is practically one of the easier
  117. ways to do so.
  118. So in practice, the 'at all' may become a 'why freeze kernel threads?' and
  119. freezing user threads I don't find really objectionable."
  120. Still, there are kernel threads that may want to be freezable. For example, if
  121. a kernel thread that belongs to a device driver accesses the device directly, it
  122. in principle needs to know when the device is suspended, so that it doesn't try
  123. to access it at that time. However, if the kernel thread is freezable, it will
  124. be frozen before the driver's .suspend() callback is executed and it will be
  125. thawed after the driver's .resume() callback has run, so it won't be accessing
  126. the device while it's suspended.
  127. 4. Another reason for freezing tasks is to prevent user space processes from
  128. realizing that hibernation (or suspend) operation takes place. Ideally, user
  129. space processes should not notice that such a system-wide operation has
  130. occurred and should continue running without any problems after the restore
  131. (or resume from suspend). Unfortunately, in the most general case this
  132. is quite difficult to achieve without the freezing of tasks. Consider,
  133. for example, a process that depends on all CPUs being online while it's
  134. running. Since we need to disable nonboot CPUs during the hibernation,
  135. if this process is not frozen, it may notice that the number of CPUs has
  136. changed and may start to work incorrectly because of that.
  137. V. Are there any problems related to the freezing of tasks?
  138. ===========================================================
  139. Yes, there are.
  140. First of all, the freezing of kernel threads may be tricky if they depend one
  141. on another. For example, if kernel thread A waits for a completion (in the
  142. TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
  143. and B is frozen in the meantime, then A will be blocked until B is thawed, which
  144. may be undesirable. That's why kernel threads are not freezable by default.
  145. Second, there are the following two problems related to the freezing of user
  146. space processes:
  147. 1. Putting processes into an uninterruptible sleep distorts the load average.
  148. 2. Now that we have FUSE, plus the framework for doing device drivers in
  149. userspace, it gets even more complicated because some userspace processes are
  150. now doing the sorts of things that kernel threads do
  151. (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
  152. The problem 1. seems to be fixable, although it hasn't been fixed so far. The
  153. other one is more serious, but it seems that we can work around it by using
  154. hibernation (and suspend) notifiers (in that case, though, we won't be able to
  155. avoid the realization by the user space processes that the hibernation is taking
  156. place).
  157. There are also problems that the freezing of tasks tends to expose, although
  158. they are not directly related to it. For example, if request_firmware() is
  159. called from a device driver's .resume() routine, it will timeout and eventually
  160. fail, because the user land process that should respond to the request is frozen
  161. at this point. So, seemingly, the failure is due to the freezing of tasks.
  162. Suppose, however, that the firmware file is located on a filesystem accessible
  163. only through another device that hasn't been resumed yet. In that case,
  164. request_firmware() will fail regardless of whether or not the freezing of tasks
  165. is used. Consequently, the problem is not really related to the freezing of
  166. tasks, since it generally exists anyway.
  167. A driver must have all firmwares it may need in RAM before suspend() is called.
  168. If keeping them is not practical, for example due to their size, they must be
  169. requested early enough using the suspend notifier API described in
  170. Documentation/driver-api/pm/notifiers.rst.
  171. VI. Are there any precautions to be taken to prevent freezing failures?
  172. =======================================================================
  173. Yes, there are.
  174. First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a
  175. piece of code from system-wide sleep such as suspend/hibernation is not
  176. encouraged. If possible, that piece of code must instead hook onto the
  177. suspend/hibernation notifiers to achieve mutual exclusion. Look at the
  178. CPU-Hotplug code (kernel/cpu.c) for an example.
  179. However, if that is not feasible, and grabbing 'system_transition_mutex' is
  180. deemed necessary, it is strongly discouraged to directly call
  181. mutex_[un]lock(&system_transition_mutex) since that could lead to freezing
  182. failures, because if the suspend/hibernate code successfully acquired the
  183. 'system_transition_mutex' lock, and hence that other entity failed to acquire
  184. the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a
  185. consequence, the freezer would not be able to freeze that task, leading to
  186. freezing failure.
  187. However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
  188. since they ask the freezer to skip freezing this task, since it is anyway
  189. "frozen enough" as it is blocked on 'system_transition_mutex', which will be
  190. released only after the entire suspend/hibernation sequence is complete. So, to
  191. summarize, use [un]lock_system_sleep() instead of directly using
  192. mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures.
  193. V. Miscellaneous
  194. ================
  195. /sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
  196. all user space processes or all freezable kernel threads, in unit of
  197. millisecond. The default value is 20000, with range of unsigned integer.