cpu_hotplug.rst 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365
  1. =========================
  2. CPU hotplug in the Kernel
  3. =========================
  4. :Date: December, 2016
  5. :Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
  6. Rusty Russell <rusty@rustcorp.com.au>,
  7. Srivatsa Vaddagiri <vatsa@in.ibm.com>,
  8. Ashok Raj <ashok.raj@intel.com>,
  9. Joel Schopp <jschopp@austin.ibm.com>
  10. Introduction
  11. ============
  12. Modern advances in system architectures have introduced advanced error
  13. reporting and correction capabilities in processors. There are couple OEMS that
  14. support NUMA hardware which are hot pluggable as well, where physical node
  15. insertion and removal require support for CPU hotplug.
  16. Such advances require CPUs available to a kernel to be removed either for
  17. provisioning reasons, or for RAS purposes to keep an offending CPU off
  18. system execution path. Hence the need for CPU hotplug support in the
  19. Linux kernel.
  20. A more novel use of CPU-hotplug support is its use today in suspend resume
  21. support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
  22. which didn't support these methods.
  23. Command Line Switches
  24. =====================
  25. ``maxcpus=n``
  26. Restrict boot time CPUs to *n*. Say if you have four CPUs, using
  27. ``maxcpus=2`` will only boot two. You can choose to bring the
  28. other CPUs later online.
  29. ``nr_cpus=n``
  30. Restrict the total amount of CPUs the kernel will support. If the number
  31. supplied here is lower than the number of physically available CPUs, then
  32. those CPUs can not be brought online later.
  33. ``additional_cpus=n``
  34. Use this to limit hotpluggable CPUs. This option sets
  35. ``cpu_possible_mask = cpu_present_mask + additional_cpus``
  36. This option is limited to the IA64 architecture.
  37. ``possible_cpus=n``
  38. This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
  39. This option is limited to the X86 and S390 architecture.
  40. ``cpu0_hotplug``
  41. Allow to shutdown CPU0.
  42. This option is limited to the X86 architecture.
  43. CPU maps
  44. ========
  45. ``cpu_possible_mask``
  46. Bitmap of possible CPUs that can ever be available in the
  47. system. This is used to allocate some boot time memory for per_cpu variables
  48. that aren't designed to grow/shrink as CPUs are made available or removed.
  49. Once set during boot time discovery phase, the map is static, i.e no bits
  50. are added or removed anytime. Trimming it accurately for your system needs
  51. upfront can save some boot time memory.
  52. ``cpu_online_mask``
  53. Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
  54. after a CPU is available for kernel scheduling and ready to receive
  55. interrupts from devices. Its cleared when a CPU is brought down using
  56. ``__cpu_disable()``, before which all OS services including interrupts are
  57. migrated to another target CPU.
  58. ``cpu_present_mask``
  59. Bitmap of CPUs currently present in the system. Not all
  60. of them may be online. When physical hotplug is processed by the relevant
  61. subsystem (e.g ACPI) can change and new bit either be added or removed
  62. from the map depending on the event is hot-add/hot-remove. There are currently
  63. no locking rules as of now. Typical usage is to init topology during boot,
  64. at which time hotplug is disabled.
  65. You really don't need to manipulate any of the system CPU maps. They should
  66. be read-only for most use. When setting up per-cpu resources almost always use
  67. ``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
  68. ``for_each_cpu()`` can be used to iterate over a custom CPU mask.
  69. Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
  70. Using CPU hotplug
  71. =================
  72. The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
  73. available on multiple architectures including ARM, MIPS, PowerPC and X86. The
  74. configuration is done via the sysfs interface: ::
  75. $ ls -lh /sys/devices/system/cpu
  76. total 0
  77. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu0
  78. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu1
  79. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu2
  80. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu3
  81. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu4
  82. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu5
  83. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu6
  84. drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu7
  85. drwxr-xr-x 2 root root 0 Dec 21 16:33 hotplug
  86. -r--r--r-- 1 root root 4.0K Dec 21 16:33 offline
  87. -r--r--r-- 1 root root 4.0K Dec 21 16:33 online
  88. -r--r--r-- 1 root root 4.0K Dec 21 16:33 possible
  89. -r--r--r-- 1 root root 4.0K Dec 21 16:33 present
  90. The files *offline*, *online*, *possible*, *present* represent the CPU masks.
  91. Each CPU folder contains an *online* file which controls the logical on (1) and
  92. off (0) state. To logically shutdown CPU4: ::
  93. $ echo 0 > /sys/devices/system/cpu/cpu4/online
  94. smpboot: CPU 4 is now offline
  95. Once the CPU is shutdown, it will be removed from */proc/interrupts*,
  96. */proc/cpuinfo* and should also not be shown visible by the *top* command. To
  97. bring CPU4 back online: ::
  98. $ echo 1 > /sys/devices/system/cpu/cpu4/online
  99. smpboot: Booting Node 0 Processor 4 APIC 0x1
  100. The CPU is usable again. This should work on all CPUs. CPU0 is often special
  101. and excluded from CPU hotplug. On X86 the kernel option
  102. *CONFIG_BOOTPARAM_HOTPLUG_CPU0* has to be enabled in order to be able to
  103. shutdown CPU0. Alternatively the kernel command option *cpu0_hotplug* can be
  104. used. Some known dependencies of CPU0:
  105. * Resume from hibernate/suspend. Hibernate/suspend will fail if CPU0 is offline.
  106. * PIC interrupts. CPU0 can't be removed if a PIC interrupt is detected.
  107. Please let Fenghua Yu <fenghua.yu@intel.com> know if you find any dependencies
  108. on CPU0.
  109. The CPU hotplug coordination
  110. ============================
  111. The offline case
  112. ----------------
  113. Once a CPU has been logically shutdown the teardown callbacks of registered
  114. hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
  115. at state ``CPUHP_OFFLINE``. This includes:
  116. * If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
  117. will be set to true.
  118. * All processes are migrated away from this outgoing CPU to new CPUs.
  119. The new CPU is chosen from each process' current cpuset, which may be
  120. a subset of all online CPUs.
  121. * All interrupts targeted to this CPU are migrated to a new CPU
  122. * timers are also migrated to a new CPU
  123. * Once all services are migrated, kernel calls an arch specific routine
  124. ``__cpu_disable()`` to perform arch specific cleanup.
  125. Using the hotplug API
  126. ---------------------
  127. It is possible to receive notifications once a CPU is offline or onlined. This
  128. might be important to certain drivers which need to perform some kind of setup
  129. or clean up functions based on the number of available CPUs: ::
  130. #include <linux/cpuhotplug.h>
  131. ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "X/Y:online",
  132. Y_online, Y_prepare_down);
  133. *X* is the subsystem and *Y* the particular driver. The *Y_online* callback
  134. will be invoked during registration on all online CPUs. If an error
  135. occurs during the online callback the *Y_prepare_down* callback will be
  136. invoked on all CPUs on which the online callback was previously invoked.
  137. After registration completed, the *Y_online* callback will be invoked
  138. once a CPU is brought online and *Y_prepare_down* will be invoked when a
  139. CPU is shutdown. All resources which were previously allocated in
  140. *Y_online* should be released in *Y_prepare_down*.
  141. The return value *ret* is negative if an error occurred during the
  142. registration process. Otherwise a positive value is returned which
  143. contains the allocated hotplug for dynamically allocated states
  144. (*CPUHP_AP_ONLINE_DYN*). It will return zero for predefined states.
  145. The callback can be remove by invoking ``cpuhp_remove_state()``. In case of a
  146. dynamically allocated state (*CPUHP_AP_ONLINE_DYN*) use the returned state.
  147. During the removal of a hotplug state the teardown callback will be invoked.
  148. Multiple instances
  149. ~~~~~~~~~~~~~~~~~~
  150. If a driver has multiple instances and each instance needs to perform the
  151. callback independently then it is likely that a ''multi-state'' should be used.
  152. First a multi-state state needs to be registered: ::
  153. ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "X/Y:online,
  154. Y_online, Y_prepare_down);
  155. Y_hp_online = ret;
  156. The ``cpuhp_setup_state_multi()`` behaves similar to ``cpuhp_setup_state()``
  157. except it prepares the callbacks for a multi state and does not invoke
  158. the callbacks. This is a one time setup.
  159. Once a new instance is allocated, you need to register this new instance: ::
  160. ret = cpuhp_state_add_instance(Y_hp_online, &d->node);
  161. This function will add this instance to your previously allocated
  162. *Y_hp_online* state and invoke the previously registered callback
  163. (*Y_online*) on all online CPUs. The *node* element is a ``struct
  164. hlist_node`` member of your per-instance data structure.
  165. On removal of the instance: ::
  166. cpuhp_state_remove_instance(Y_hp_online, &d->node)
  167. should be invoked which will invoke the teardown callback on all online
  168. CPUs.
  169. Manual setup
  170. ~~~~~~~~~~~~
  171. Usually it is handy to invoke setup and teardown callbacks on registration or
  172. removal of a state because usually the operation needs to performed once a CPU
  173. goes online (offline) and during initial setup (shutdown) of the driver. However
  174. each registration and removal function is also available with a ``_nocalls``
  175. suffix which does not invoke the provided callbacks if the invocation of the
  176. callbacks is not desired. During the manual setup (or teardown) the functions
  177. ``get_online_cpus()`` and ``put_online_cpus()`` should be used to inhibit CPU
  178. hotplug operations.
  179. The ordering of the events
  180. --------------------------
  181. The hotplug states are defined in ``include/linux/cpuhotplug.h``:
  182. * The states *CPUHP_OFFLINE* … *CPUHP_AP_OFFLINE* are invoked before the
  183. CPU is up.
  184. * The states *CPUHP_AP_OFFLINE* … *CPUHP_AP_ONLINE* are invoked
  185. just the after the CPU has been brought up. The interrupts are off and
  186. the scheduler is not yet active on this CPU. Starting with *CPUHP_AP_OFFLINE*
  187. the callbacks are invoked on the target CPU.
  188. * The states between *CPUHP_AP_ONLINE_DYN* and *CPUHP_AP_ONLINE_DYN_END* are
  189. reserved for the dynamic allocation.
  190. * The states are invoked in the reverse order on CPU shutdown starting with
  191. *CPUHP_ONLINE* and stopping at *CPUHP_OFFLINE*. Here the callbacks are
  192. invoked on the CPU that will be shutdown until *CPUHP_AP_OFFLINE*.
  193. A dynamically allocated state via *CPUHP_AP_ONLINE_DYN* is often enough.
  194. However if an earlier invocation during the bring up or shutdown is required
  195. then an explicit state should be acquired. An explicit state might also be
  196. required if the hotplug event requires specific ordering in respect to
  197. another hotplug event.
  198. Testing of hotplug states
  199. =========================
  200. One way to verify whether a custom state is working as expected or not is to
  201. shutdown a CPU and then put it online again. It is also possible to put the CPU
  202. to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
  203. *CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
  204. which would lead to rollback to the online state.
  205. All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states``: ::
  206. $ tail /sys/devices/system/cpu/hotplug/states
  207. 138: mm/vmscan:online
  208. 139: mm/vmstat:online
  209. 140: lib/percpu_cnt:online
  210. 141: acpi/cpu-drv:online
  211. 142: base/cacheinfo:online
  212. 143: virtio/net:online
  213. 144: x86/mce:online
  214. 145: printk:online
  215. 168: sched:active
  216. 169: online
  217. To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue: ::
  218. $ cat /sys/devices/system/cpu/cpu4/hotplug/state
  219. 169
  220. $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
  221. $ cat /sys/devices/system/cpu/cpu4/hotplug/state
  222. 140
  223. It is important to note that the teardown callbac of state 140 have been
  224. invoked. And now get back online: ::
  225. $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
  226. $ cat /sys/devices/system/cpu/cpu4/hotplug/state
  227. 169
  228. With trace events enabled, the individual steps are visible, too: ::
  229. # TASK-PID CPU# TIMESTAMP FUNCTION
  230. # | | | | |
  231. bash-394 [001] 22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
  232. cpuhp/4-31 [004] 22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
  233. cpuhp/4-31 [004] 22.990: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
  234. cpuhp/4-31 [004] 22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
  235. cpuhp/4-31 [004] 22.992: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
  236. cpuhp/4-31 [004] 22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
  237. cpuhp/4-31 [004] 22.994: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
  238. cpuhp/4-31 [004] 22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
  239. cpuhp/4-31 [004] 22.996: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
  240. bash-394 [001] 22.997: cpuhp_exit: cpu: 0004 state: 140 step: 169 ret: 0
  241. bash-394 [005] 95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
  242. cpuhp/4-31 [004] 95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
  243. cpuhp/4-31 [004] 95.542: cpuhp_exit: cpu: 0004 state: 141 step: 141 ret: 0
  244. cpuhp/4-31 [004] 95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
  245. cpuhp/4-31 [004] 95.544: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0
  246. cpuhp/4-31 [004] 95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
  247. cpuhp/4-31 [004] 95.546: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0
  248. cpuhp/4-31 [004] 95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
  249. cpuhp/4-31 [004] 95.548: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0
  250. cpuhp/4-31 [004] 95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
  251. cpuhp/4-31 [004] 95.550: cpuhp_exit: cpu: 0004 state: 145 step: 145 ret: 0
  252. cpuhp/4-31 [004] 95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
  253. cpuhp/4-31 [004] 95.552: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0
  254. bash-394 [005] 95.553: cpuhp_exit: cpu: 0004 state: 169 step: 140 ret: 0
  255. As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
  256. 95.552. All invoked callbacks including their return codes are visible in the
  257. trace.
  258. Architecture's requirements
  259. ===========================
  260. The following functions and configurations are required:
  261. ``CONFIG_HOTPLUG_CPU``
  262. This entry needs to be enabled in Kconfig
  263. ``__cpu_up()``
  264. Arch interface to bring up a CPU
  265. ``__cpu_disable()``
  266. Arch interface to shutdown a CPU, no more interrupts can be handled by the
  267. kernel after the routine returns. This includes the shutdown of the timer.
  268. ``__cpu_die()``
  269. This actually supposed to ensure death of the CPU. Actually look at some
  270. example code in other arch that implement CPU hotplug. The processor is taken
  271. down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
  272. typically waits for some per_cpu state to be set, to ensure the processor dead
  273. routine is called to be sure positively.
  274. User Space Notification
  275. =======================
  276. After CPU successfully onlined or offline udev events are sent. A udev rule like: ::
  277. SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
  278. will receive all events. A script like: ::
  279. #!/bin/sh
  280. if [ "${ACTION}" = "offline" ]
  281. then
  282. echo "CPU ${DEVPATH##*/} offline"
  283. elif [ "${ACTION}" = "online" ]
  284. then
  285. echo "CPU ${DEVPATH##*/} online"
  286. fi
  287. can process the event further.
  288. Kernel Inline Documentations Reference
  289. ======================================
  290. .. kernel-doc:: include/linux/cpuhotplug.h