rt-mutex-design.rst 22 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574
  1. ==============================
  2. RT-mutex implementation design
  3. ==============================
  4. Copyright (c) 2006 Steven Rostedt
  5. Licensed under the GNU Free Documentation License, Version 1.2
  6. This document tries to describe the design of the rtmutex.c implementation.
  7. It doesn't describe the reasons why rtmutex.c exists. For that please see
  8. Documentation/locking/rt-mutex.rst. Although this document does explain problems
  9. that happen without this code, but that is in the concept to understand
  10. what the code actually is doing.
  11. The goal of this document is to help others understand the priority
  12. inheritance (PI) algorithm that is used, as well as reasons for the
  13. decisions that were made to implement PI in the manner that was done.
  14. Unbounded Priority Inversion
  15. ----------------------------
  16. Priority inversion is when a lower priority process executes while a higher
  17. priority process wants to run. This happens for several reasons, and
  18. most of the time it can't be helped. Anytime a high priority process wants
  19. to use a resource that a lower priority process has (a mutex for example),
  20. the high priority process must wait until the lower priority process is done
  21. with the resource. This is a priority inversion. What we want to prevent
  22. is something called unbounded priority inversion. That is when the high
  23. priority process is prevented from running by a lower priority process for
  24. an undetermined amount of time.
  25. The classic example of unbounded priority inversion is where you have three
  26. processes, let's call them processes A, B, and C, where A is the highest
  27. priority process, C is the lowest, and B is in between. A tries to grab a lock
  28. that C owns and must wait and lets C run to release the lock. But in the
  29. meantime, B executes, and since B is of a higher priority than C, it preempts C,
  30. but by doing so, it is in fact preempting A which is a higher priority process.
  31. Now there's no way of knowing how long A will be sleeping waiting for C
  32. to release the lock, because for all we know, B is a CPU hog and will
  33. never give C a chance to release the lock. This is called unbounded priority
  34. inversion.
  35. Here's a little ASCII art to show the problem::
  36. grab lock L1 (owned by C)
  37. |
  38. A ---+
  39. C preempted by B
  40. |
  41. C +----+
  42. B +-------->
  43. B now keeps A from running.
  44. Priority Inheritance (PI)
  45. -------------------------
  46. There are several ways to solve this issue, but other ways are out of scope
  47. for this document. Here we only discuss PI.
  48. PI is where a process inherits the priority of another process if the other
  49. process blocks on a lock owned by the current process. To make this easier
  50. to understand, let's use the previous example, with processes A, B, and C again.
  51. This time, when A blocks on the lock owned by C, C would inherit the priority
  52. of A. So now if B becomes runnable, it would not preempt C, since C now has
  53. the high priority of A. As soon as C releases the lock, it loses its
  54. inherited priority, and A then can continue with the resource that C had.
  55. Terminology
  56. -----------
  57. Here I explain some terminology that is used in this document to help describe
  58. the design that is used to implement PI.
  59. PI chain
  60. - The PI chain is an ordered series of locks and processes that cause
  61. processes to inherit priorities from a previous process that is
  62. blocked on one of its locks. This is described in more detail
  63. later in this document.
  64. mutex
  65. - In this document, to differentiate from locks that implement
  66. PI and spin locks that are used in the PI code, from now on
  67. the PI locks will be called a mutex.
  68. lock
  69. - In this document from now on, I will use the term lock when
  70. referring to spin locks that are used to protect parts of the PI
  71. algorithm. These locks disable preemption for UP (when
  72. CONFIG_PREEMPT is enabled) and on SMP prevents multiple CPUs from
  73. entering critical sections simultaneously.
  74. spin lock
  75. - Same as lock above.
  76. waiter
  77. - A waiter is a struct that is stored on the stack of a blocked
  78. process. Since the scope of the waiter is within the code for
  79. a process being blocked on the mutex, it is fine to allocate
  80. the waiter on the process's stack (local variable). This
  81. structure holds a pointer to the task, as well as the mutex that
  82. the task is blocked on. It also has rbtree node structures to
  83. place the task in the waiters rbtree of a mutex as well as the
  84. pi_waiters rbtree of a mutex owner task (described below).
  85. waiter is sometimes used in reference to the task that is waiting
  86. on a mutex. This is the same as waiter->task.
  87. waiters
  88. - A list of processes that are blocked on a mutex.
  89. top waiter
  90. - The highest priority process waiting on a specific mutex.
  91. top pi waiter
  92. - The highest priority process waiting on one of the mutexes
  93. that a specific process owns.
  94. Note:
  95. task and process are used interchangeably in this document, mostly to
  96. differentiate between two processes that are being described together.
  97. PI chain
  98. --------
  99. The PI chain is a list of processes and mutexes that may cause priority
  100. inheritance to take place. Multiple chains may converge, but a chain
  101. would never diverge, since a process can't be blocked on more than one
  102. mutex at a time.
  103. Example::
  104. Process: A, B, C, D, E
  105. Mutexes: L1, L2, L3, L4
  106. A owns: L1
  107. B blocked on L1
  108. B owns L2
  109. C blocked on L2
  110. C owns L3
  111. D blocked on L3
  112. D owns L4
  113. E blocked on L4
  114. The chain would be::
  115. E->L4->D->L3->C->L2->B->L1->A
  116. To show where two chains merge, we could add another process F and
  117. another mutex L5 where B owns L5 and F is blocked on mutex L5.
  118. The chain for F would be::
  119. F->L5->B->L1->A
  120. Since a process may own more than one mutex, but never be blocked on more than
  121. one, the chains merge.
  122. Here we show both chains::
  123. E->L4->D->L3->C->L2-+
  124. |
  125. +->B->L1->A
  126. |
  127. F->L5-+
  128. For PI to work, the processes at the right end of these chains (or we may
  129. also call it the Top of the chain) must be equal to or higher in priority
  130. than the processes to the left or below in the chain.
  131. Also since a mutex may have more than one process blocked on it, we can
  132. have multiple chains merge at mutexes. If we add another process G that is
  133. blocked on mutex L2::
  134. G->L2->B->L1->A
  135. And once again, to show how this can grow I will show the merging chains
  136. again::
  137. E->L4->D->L3->C-+
  138. +->L2-+
  139. | |
  140. G-+ +->B->L1->A
  141. |
  142. F->L5-+
  143. If process G has the highest priority in the chain, then all the tasks up
  144. the chain (A and B in this example), must have their priorities increased
  145. to that of G.
  146. Mutex Waiters Tree
  147. ------------------
  148. Every mutex keeps track of all the waiters that are blocked on itself. The
  149. mutex has a rbtree to store these waiters by priority. This tree is protected
  150. by a spin lock that is located in the struct of the mutex. This lock is called
  151. wait_lock.
  152. Task PI Tree
  153. ------------
  154. To keep track of the PI chains, each process has its own PI rbtree. This is
  155. a tree of all top waiters of the mutexes that are owned by the process.
  156. Note that this tree only holds the top waiters and not all waiters that are
  157. blocked on mutexes owned by the process.
  158. The top of the task's PI tree is always the highest priority task that
  159. is waiting on a mutex that is owned by the task. So if the task has
  160. inherited a priority, it will always be the priority of the task that is
  161. at the top of this tree.
  162. This tree is stored in the task structure of a process as a rbtree called
  163. pi_waiters. It is protected by a spin lock also in the task structure,
  164. called pi_lock. This lock may also be taken in interrupt context, so when
  165. locking the pi_lock, interrupts must be disabled.
  166. Depth of the PI Chain
  167. ---------------------
  168. The maximum depth of the PI chain is not dynamic, and could actually be
  169. defined. But is very complex to figure it out, since it depends on all
  170. the nesting of mutexes. Let's look at the example where we have 3 mutexes,
  171. L1, L2, and L3, and four separate functions func1, func2, func3 and func4.
  172. The following shows a locking order of L1->L2->L3, but may not actually
  173. be directly nested that way::
  174. void func1(void)
  175. {
  176. mutex_lock(L1);
  177. /* do anything */
  178. mutex_unlock(L1);
  179. }
  180. void func2(void)
  181. {
  182. mutex_lock(L1);
  183. mutex_lock(L2);
  184. /* do something */
  185. mutex_unlock(L2);
  186. mutex_unlock(L1);
  187. }
  188. void func3(void)
  189. {
  190. mutex_lock(L2);
  191. mutex_lock(L3);
  192. /* do something else */
  193. mutex_unlock(L3);
  194. mutex_unlock(L2);
  195. }
  196. void func4(void)
  197. {
  198. mutex_lock(L3);
  199. /* do something again */
  200. mutex_unlock(L3);
  201. }
  202. Now we add 4 processes that run each of these functions separately.
  203. Processes A, B, C, and D which run functions func1, func2, func3 and func4
  204. respectively, and such that D runs first and A last. With D being preempted
  205. in func4 in the "do something again" area, we have a locking that follows::
  206. D owns L3
  207. C blocked on L3
  208. C owns L2
  209. B blocked on L2
  210. B owns L1
  211. A blocked on L1
  212. And thus we have the chain A->L1->B->L2->C->L3->D.
  213. This gives us a PI depth of 4 (four processes), but looking at any of the
  214. functions individually, it seems as though they only have at most a locking
  215. depth of two. So, although the locking depth is defined at compile time,
  216. it still is very difficult to find the possibilities of that depth.
  217. Now since mutexes can be defined by user-land applications, we don't want a DOS
  218. type of application that nests large amounts of mutexes to create a large
  219. PI chain, and have the code holding spin locks while looking at a large
  220. amount of data. So to prevent this, the implementation not only implements
  221. a maximum lock depth, but also only holds at most two different locks at a
  222. time, as it walks the PI chain. More about this below.
  223. Mutex owner and flags
  224. ---------------------
  225. The mutex structure contains a pointer to the owner of the mutex. If the
  226. mutex is not owned, this owner is set to NULL. Since all architectures
  227. have the task structure on at least a two byte alignment (and if this is
  228. not true, the rtmutex.c code will be broken!), this allows for the least
  229. significant bit to be used as a flag. Bit 0 is used as the "Has Waiters"
  230. flag. It's set whenever there are waiters on a mutex.
  231. See Documentation/locking/rt-mutex.rst for further details.
  232. cmpxchg Tricks
  233. --------------
  234. Some architectures implement an atomic cmpxchg (Compare and Exchange). This
  235. is used (when applicable) to keep the fast path of grabbing and releasing
  236. mutexes short.
  237. cmpxchg is basically the following function performed atomically::
  238. unsigned long _cmpxchg(unsigned long *A, unsigned long *B, unsigned long *C)
  239. {
  240. unsigned long T = *A;
  241. if (*A == *B) {
  242. *A = *C;
  243. }
  244. return T;
  245. }
  246. #define cmpxchg(a,b,c) _cmpxchg(&a,&b,&c)
  247. This is really nice to have, since it allows you to only update a variable
  248. if the variable is what you expect it to be. You know if it succeeded if
  249. the return value (the old value of A) is equal to B.
  250. The macro rt_mutex_cmpxchg is used to try to lock and unlock mutexes. If
  251. the architecture does not support CMPXCHG, then this macro is simply set
  252. to fail every time. But if CMPXCHG is supported, then this will
  253. help out extremely to keep the fast path short.
  254. The use of rt_mutex_cmpxchg with the flags in the owner field help optimize
  255. the system for architectures that support it. This will also be explained
  256. later in this document.
  257. Priority adjustments
  258. --------------------
  259. The implementation of the PI code in rtmutex.c has several places that a
  260. process must adjust its priority. With the help of the pi_waiters of a
  261. process this is rather easy to know what needs to be adjusted.
  262. The functions implementing the task adjustments are rt_mutex_adjust_prio
  263. and rt_mutex_setprio. rt_mutex_setprio is only used in rt_mutex_adjust_prio.
  264. rt_mutex_adjust_prio examines the priority of the task, and the highest
  265. priority process that is waiting any of mutexes owned by the task. Since
  266. the pi_waiters of a task holds an order by priority of all the top waiters
  267. of all the mutexes that the task owns, we simply need to compare the top
  268. pi waiter to its own normal/deadline priority and take the higher one.
  269. Then rt_mutex_setprio is called to adjust the priority of the task to the
  270. new priority. Note that rt_mutex_setprio is defined in kernel/sched/core.c
  271. to implement the actual change in priority.
  272. Note:
  273. For the "prio" field in task_struct, the lower the number, the
  274. higher the priority. A "prio" of 5 is of higher priority than a
  275. "prio" of 10.
  276. It is interesting to note that rt_mutex_adjust_prio can either increase
  277. or decrease the priority of the task. In the case that a higher priority
  278. process has just blocked on a mutex owned by the task, rt_mutex_adjust_prio
  279. would increase/boost the task's priority. But if a higher priority task
  280. were for some reason to leave the mutex (timeout or signal), this same function
  281. would decrease/unboost the priority of the task. That is because the pi_waiters
  282. always contains the highest priority task that is waiting on a mutex owned
  283. by the task, so we only need to compare the priority of that top pi waiter
  284. to the normal priority of the given task.
  285. High level overview of the PI chain walk
  286. ----------------------------------------
  287. The PI chain walk is implemented by the function rt_mutex_adjust_prio_chain.
  288. The implementation has gone through several iterations, and has ended up
  289. with what we believe is the best. It walks the PI chain by only grabbing
  290. at most two locks at a time, and is very efficient.
  291. The rt_mutex_adjust_prio_chain can be used either to boost or lower process
  292. priorities.
  293. rt_mutex_adjust_prio_chain is called with a task to be checked for PI
  294. (de)boosting (the owner of a mutex that a process is blocking on), a flag to
  295. check for deadlocking, the mutex that the task owns, a pointer to a waiter
  296. that is the process's waiter struct that is blocked on the mutex (although this
  297. parameter may be NULL for deboosting), a pointer to the mutex on which the task
  298. is blocked, and a top_task as the top waiter of the mutex.
  299. For this explanation, I will not mention deadlock detection. This explanation
  300. will try to stay at a high level.
  301. When this function is called, there are no locks held. That also means
  302. that the state of the owner and lock can change when entered into this function.
  303. Before this function is called, the task has already had rt_mutex_adjust_prio
  304. performed on it. This means that the task is set to the priority that it
  305. should be at, but the rbtree nodes of the task's waiter have not been updated
  306. with the new priorities, and this task may not be in the proper locations
  307. in the pi_waiters and waiters trees that the task is blocked on. This function
  308. solves all that.
  309. The main operation of this function is summarized by Thomas Gleixner in
  310. rtmutex.c. See the 'Chain walk basics and protection scope' comment for further
  311. details.
  312. Taking of a mutex (The walk through)
  313. ------------------------------------
  314. OK, now let's take a look at the detailed walk through of what happens when
  315. taking a mutex.
  316. The first thing that is tried is the fast taking of the mutex. This is
  317. done when we have CMPXCHG enabled (otherwise the fast taking automatically
  318. fails). Only when the owner field of the mutex is NULL can the lock be
  319. taken with the CMPXCHG and nothing else needs to be done.
  320. If there is contention on the lock, we go about the slow path
  321. (rt_mutex_slowlock).
  322. The slow path function is where the task's waiter structure is created on
  323. the stack. This is because the waiter structure is only needed for the
  324. scope of this function. The waiter structure holds the nodes to store
  325. the task on the waiters tree of the mutex, and if need be, the pi_waiters
  326. tree of the owner.
  327. The wait_lock of the mutex is taken since the slow path of unlocking the
  328. mutex also takes this lock.
  329. We then call try_to_take_rt_mutex. This is where the architecture that
  330. does not implement CMPXCHG would always grab the lock (if there's no
  331. contention).
  332. try_to_take_rt_mutex is used every time the task tries to grab a mutex in the
  333. slow path. The first thing that is done here is an atomic setting of
  334. the "Has Waiters" flag of the mutex's owner field. By setting this flag
  335. now, the current owner of the mutex being contended for can't release the mutex
  336. without going into the slow unlock path, and it would then need to grab the
  337. wait_lock, which this code currently holds. So setting the "Has Waiters" flag
  338. forces the current owner to synchronize with this code.
  339. The lock is taken if the following are true:
  340. 1) The lock has no owner
  341. 2) The current task is the highest priority against all other
  342. waiters of the lock
  343. If the task succeeds to acquire the lock, then the task is set as the
  344. owner of the lock, and if the lock still has waiters, the top_waiter
  345. (highest priority task waiting on the lock) is added to this task's
  346. pi_waiters tree.
  347. If the lock is not taken by try_to_take_rt_mutex(), then the
  348. task_blocks_on_rt_mutex() function is called. This will add the task to
  349. the lock's waiter tree and propagate the pi chain of the lock as well
  350. as the lock's owner's pi_waiters tree. This is described in the next
  351. section.
  352. Task blocks on mutex
  353. --------------------
  354. The accounting of a mutex and process is done with the waiter structure of
  355. the process. The "task" field is set to the process, and the "lock" field
  356. to the mutex. The rbtree node of waiter are initialized to the processes
  357. current priority.
  358. Since the wait_lock was taken at the entry of the slow lock, we can safely
  359. add the waiter to the task waiter tree. If the current process is the
  360. highest priority process currently waiting on this mutex, then we remove the
  361. previous top waiter process (if it exists) from the pi_waiters of the owner,
  362. and add the current process to that tree. Since the pi_waiter of the owner
  363. has changed, we call rt_mutex_adjust_prio on the owner to see if the owner
  364. should adjust its priority accordingly.
  365. If the owner is also blocked on a lock, and had its pi_waiters changed
  366. (or deadlock checking is on), we unlock the wait_lock of the mutex and go ahead
  367. and run rt_mutex_adjust_prio_chain on the owner, as described earlier.
  368. Now all locks are released, and if the current process is still blocked on a
  369. mutex (waiter "task" field is not NULL), then we go to sleep (call schedule).
  370. Waking up in the loop
  371. ---------------------
  372. The task can then wake up for a couple of reasons:
  373. 1) The previous lock owner released the lock, and the task now is top_waiter
  374. 2) we received a signal or timeout
  375. In both cases, the task will try again to acquire the lock. If it
  376. does, then it will take itself off the waiters tree and set itself back
  377. to the TASK_RUNNING state.
  378. In first case, if the lock was acquired by another task before this task
  379. could get the lock, then it will go back to sleep and wait to be woken again.
  380. The second case is only applicable for tasks that are grabbing a mutex
  381. that can wake up before getting the lock, either due to a signal or
  382. a timeout (i.e. rt_mutex_timed_futex_lock()). When woken, it will try to
  383. take the lock again, if it succeeds, then the task will return with the
  384. lock held, otherwise it will return with -EINTR if the task was woken
  385. by a signal, or -ETIMEDOUT if it timed out.
  386. Unlocking the Mutex
  387. -------------------
  388. The unlocking of a mutex also has a fast path for those architectures with
  389. CMPXCHG. Since the taking of a mutex on contention always sets the
  390. "Has Waiters" flag of the mutex's owner, we use this to know if we need to
  391. take the slow path when unlocking the mutex. If the mutex doesn't have any
  392. waiters, the owner field of the mutex would equal the current process and
  393. the mutex can be unlocked by just replacing the owner field with NULL.
  394. If the owner field has the "Has Waiters" bit set (or CMPXCHG is not available),
  395. the slow unlock path is taken.
  396. The first thing done in the slow unlock path is to take the wait_lock of the
  397. mutex. This synchronizes the locking and unlocking of the mutex.
  398. A check is made to see if the mutex has waiters or not. On architectures that
  399. do not have CMPXCHG, this is the location that the owner of the mutex will
  400. determine if a waiter needs to be awoken or not. On architectures that
  401. do have CMPXCHG, that check is done in the fast path, but it is still needed
  402. in the slow path too. If a waiter of a mutex woke up because of a signal
  403. or timeout between the time the owner failed the fast path CMPXCHG check and
  404. the grabbing of the wait_lock, the mutex may not have any waiters, thus the
  405. owner still needs to make this check. If there are no waiters then the mutex
  406. owner field is set to NULL, the wait_lock is released and nothing more is
  407. needed.
  408. If there are waiters, then we need to wake one up.
  409. On the wake up code, the pi_lock of the current owner is taken. The top
  410. waiter of the lock is found and removed from the waiters tree of the mutex
  411. as well as the pi_waiters tree of the current owner. The "Has Waiters" bit is
  412. marked to prevent lower priority tasks from stealing the lock.
  413. Finally we unlock the pi_lock of the pending owner and wake it up.
  414. Contact
  415. -------
  416. For updates on this document, please email Steven Rostedt <rostedt@goodmis.org>
  417. Credits
  418. -------
  419. Author: Steven Rostedt <rostedt@goodmis.org>
  420. Updated: Alex Shi <alex.shi@linaro.org> - 7/6/2017
  421. Original Reviewers:
  422. Ingo Molnar, Thomas Gleixner, Thomas Duetsch, and
  423. Randy Dunlap
  424. Update (7/6/2017) Reviewers: Steven Rostedt and Sebastian Siewior
  425. Updates
  426. -------
  427. This document was originally written for 2.6.17-rc3-mm1
  428. was updated on 4.12