rt-mutex-design.txt 33 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781
  1. #
  2. # Copyright (c) 2006 Steven Rostedt
  3. # Licensed under the GNU Free Documentation License, Version 1.2
  4. #
  5. RT-mutex implementation design
  6. ------------------------------
  7. This document tries to describe the design of the rtmutex.c implementation.
  8. It doesn't describe the reasons why rtmutex.c exists. For that please see
  9. Documentation/rt-mutex.txt. Although this document does explain problems
  10. that happen without this code, but that is in the concept to understand
  11. what the code actually is doing.
  12. The goal of this document is to help others understand the priority
  13. inheritance (PI) algorithm that is used, as well as reasons for the
  14. decisions that were made to implement PI in the manner that was done.
  15. Unbounded Priority Inversion
  16. ----------------------------
  17. Priority inversion is when a lower priority process executes while a higher
  18. priority process wants to run. This happens for several reasons, and
  19. most of the time it can't be helped. Anytime a high priority process wants
  20. to use a resource that a lower priority process has (a mutex for example),
  21. the high priority process must wait until the lower priority process is done
  22. with the resource. This is a priority inversion. What we want to prevent
  23. is something called unbounded priority inversion. That is when the high
  24. priority process is prevented from running by a lower priority process for
  25. an undetermined amount of time.
  26. The classic example of unbounded priority inversion is were you have three
  27. processes, let's call them processes A, B, and C, where A is the highest
  28. priority process, C is the lowest, and B is in between. A tries to grab a lock
  29. that C owns and must wait and lets C run to release the lock. But in the
  30. meantime, B executes, and since B is of a higher priority than C, it preempts C,
  31. but by doing so, it is in fact preempting A which is a higher priority process.
  32. Now there's no way of knowing how long A will be sleeping waiting for C
  33. to release the lock, because for all we know, B is a CPU hog and will
  34. never give C a chance to release the lock. This is called unbounded priority
  35. inversion.
  36. Here's a little ASCII art to show the problem.
  37. grab lock L1 (owned by C)
  38. |
  39. A ---+
  40. C preempted by B
  41. |
  42. C +----+
  43. B +-------->
  44. B now keeps A from running.
  45. Priority Inheritance (PI)
  46. -------------------------
  47. There are several ways to solve this issue, but other ways are out of scope
  48. for this document. Here we only discuss PI.
  49. PI is where a process inherits the priority of another process if the other
  50. process blocks on a lock owned by the current process. To make this easier
  51. to understand, let's use the previous example, with processes A, B, and C again.
  52. This time, when A blocks on the lock owned by C, C would inherit the priority
  53. of A. So now if B becomes runnable, it would not preempt C, since C now has
  54. the high priority of A. As soon as C releases the lock, it loses its
  55. inherited priority, and A then can continue with the resource that C had.
  56. Terminology
  57. -----------
  58. Here I explain some terminology that is used in this document to help describe
  59. the design that is used to implement PI.
  60. PI chain - The PI chain is an ordered series of locks and processes that cause
  61. processes to inherit priorities from a previous process that is
  62. blocked on one of its locks. This is described in more detail
  63. later in this document.
  64. mutex - In this document, to differentiate from locks that implement
  65. PI and spin locks that are used in the PI code, from now on
  66. the PI locks will be called a mutex.
  67. lock - In this document from now on, I will use the term lock when
  68. referring to spin locks that are used to protect parts of the PI
  69. algorithm. These locks disable preemption for UP (when
  70. CONFIG_PREEMPT is enabled) and on SMP prevents multiple CPUs from
  71. entering critical sections simultaneously.
  72. spin lock - Same as lock above.
  73. waiter - A waiter is a struct that is stored on the stack of a blocked
  74. process. Since the scope of the waiter is within the code for
  75. a process being blocked on the mutex, it is fine to allocate
  76. the waiter on the process's stack (local variable). This
  77. structure holds a pointer to the task, as well as the mutex that
  78. the task is blocked on. It also has the plist node structures to
  79. place the task in the waiter_list of a mutex as well as the
  80. pi_list of a mutex owner task (described below).
  81. waiter is sometimes used in reference to the task that is waiting
  82. on a mutex. This is the same as waiter->task.
  83. waiters - A list of processes that are blocked on a mutex.
  84. top waiter - The highest priority process waiting on a specific mutex.
  85. top pi waiter - The highest priority process waiting on one of the mutexes
  86. that a specific process owns.
  87. Note: task and process are used interchangeably in this document, mostly to
  88. differentiate between two processes that are being described together.
  89. PI chain
  90. --------
  91. The PI chain is a list of processes and mutexes that may cause priority
  92. inheritance to take place. Multiple chains may converge, but a chain
  93. would never diverge, since a process can't be blocked on more than one
  94. mutex at a time.
  95. Example:
  96. Process: A, B, C, D, E
  97. Mutexes: L1, L2, L3, L4
  98. A owns: L1
  99. B blocked on L1
  100. B owns L2
  101. C blocked on L2
  102. C owns L3
  103. D blocked on L3
  104. D owns L4
  105. E blocked on L4
  106. The chain would be:
  107. E->L4->D->L3->C->L2->B->L1->A
  108. To show where two chains merge, we could add another process F and
  109. another mutex L5 where B owns L5 and F is blocked on mutex L5.
  110. The chain for F would be:
  111. F->L5->B->L1->A
  112. Since a process may own more than one mutex, but never be blocked on more than
  113. one, the chains merge.
  114. Here we show both chains:
  115. E->L4->D->L3->C->L2-+
  116. |
  117. +->B->L1->A
  118. |
  119. F->L5-+
  120. For PI to work, the processes at the right end of these chains (or we may
  121. also call it the Top of the chain) must be equal to or higher in priority
  122. than the processes to the left or below in the chain.
  123. Also since a mutex may have more than one process blocked on it, we can
  124. have multiple chains merge at mutexes. If we add another process G that is
  125. blocked on mutex L2:
  126. G->L2->B->L1->A
  127. And once again, to show how this can grow I will show the merging chains
  128. again.
  129. E->L4->D->L3->C-+
  130. +->L2-+
  131. | |
  132. G-+ +->B->L1->A
  133. |
  134. F->L5-+
  135. Plist
  136. -----
  137. Before I go further and talk about how the PI chain is stored through lists
  138. on both mutexes and processes, I'll explain the plist. This is similar to
  139. the struct list_head functionality that is already in the kernel.
  140. The implementation of plist is out of scope for this document, but it is
  141. very important to understand what it does.
  142. There are a few differences between plist and list, the most important one
  143. being that plist is a priority sorted linked list. This means that the
  144. priorities of the plist are sorted, such that it takes O(1) to retrieve the
  145. highest priority item in the list. Obviously this is useful to store processes
  146. based on their priorities.
  147. Another difference, which is important for implementation, is that, unlike
  148. list, the head of the list is a different element than the nodes of a list.
  149. So the head of the list is declared as struct plist_head and nodes that will
  150. be added to the list are declared as struct plist_node.
  151. Mutex Waiter List
  152. -----------------
  153. Every mutex keeps track of all the waiters that are blocked on itself. The mutex
  154. has a plist to store these waiters by priority. This list is protected by
  155. a spin lock that is located in the struct of the mutex. This lock is called
  156. wait_lock. Since the modification of the waiter list is never done in
  157. interrupt context, the wait_lock can be taken without disabling interrupts.
  158. Task PI List
  159. ------------
  160. To keep track of the PI chains, each process has its own PI list. This is
  161. a list of all top waiters of the mutexes that are owned by the process.
  162. Note that this list only holds the top waiters and not all waiters that are
  163. blocked on mutexes owned by the process.
  164. The top of the task's PI list is always the highest priority task that
  165. is waiting on a mutex that is owned by the task. So if the task has
  166. inherited a priority, it will always be the priority of the task that is
  167. at the top of this list.
  168. This list is stored in the task structure of a process as a plist called
  169. pi_list. This list is protected by a spin lock also in the task structure,
  170. called pi_lock. This lock may also be taken in interrupt context, so when
  171. locking the pi_lock, interrupts must be disabled.
  172. Depth of the PI Chain
  173. ---------------------
  174. The maximum depth of the PI chain is not dynamic, and could actually be
  175. defined. But is very complex to figure it out, since it depends on all
  176. the nesting of mutexes. Let's look at the example where we have 3 mutexes,
  177. L1, L2, and L3, and four separate functions func1, func2, func3 and func4.
  178. The following shows a locking order of L1->L2->L3, but may not actually
  179. be directly nested that way.
  180. void func1(void)
  181. {
  182. mutex_lock(L1);
  183. /* do anything */
  184. mutex_unlock(L1);
  185. }
  186. void func2(void)
  187. {
  188. mutex_lock(L1);
  189. mutex_lock(L2);
  190. /* do something */
  191. mutex_unlock(L2);
  192. mutex_unlock(L1);
  193. }
  194. void func3(void)
  195. {
  196. mutex_lock(L2);
  197. mutex_lock(L3);
  198. /* do something else */
  199. mutex_unlock(L3);
  200. mutex_unlock(L2);
  201. }
  202. void func4(void)
  203. {
  204. mutex_lock(L3);
  205. /* do something again */
  206. mutex_unlock(L3);
  207. }
  208. Now we add 4 processes that run each of these functions separately.
  209. Processes A, B, C, and D which run functions func1, func2, func3 and func4
  210. respectively, and such that D runs first and A last. With D being preempted
  211. in func4 in the "do something again" area, we have a locking that follows:
  212. D owns L3
  213. C blocked on L3
  214. C owns L2
  215. B blocked on L2
  216. B owns L1
  217. A blocked on L1
  218. And thus we have the chain A->L1->B->L2->C->L3->D.
  219. This gives us a PI depth of 4 (four processes), but looking at any of the
  220. functions individually, it seems as though they only have at most a locking
  221. depth of two. So, although the locking depth is defined at compile time,
  222. it still is very difficult to find the possibilities of that depth.
  223. Now since mutexes can be defined by user-land applications, we don't want a DOS
  224. type of application that nests large amounts of mutexes to create a large
  225. PI chain, and have the code holding spin locks while looking at a large
  226. amount of data. So to prevent this, the implementation not only implements
  227. a maximum lock depth, but also only holds at most two different locks at a
  228. time, as it walks the PI chain. More about this below.
  229. Mutex owner and flags
  230. ---------------------
  231. The mutex structure contains a pointer to the owner of the mutex. If the
  232. mutex is not owned, this owner is set to NULL. Since all architectures
  233. have the task structure on at least a four byte alignment (and if this is
  234. not true, the rtmutex.c code will be broken!), this allows for the two
  235. least significant bits to be used as flags. This part is also described
  236. in Documentation/rt-mutex.txt, but will also be briefly described here.
  237. Bit 0 is used as the "Pending Owner" flag. This is described later.
  238. Bit 1 is used as the "Has Waiters" flags. This is also described later
  239. in more detail, but is set whenever there are waiters on a mutex.
  240. cmpxchg Tricks
  241. --------------
  242. Some architectures implement an atomic cmpxchg (Compare and Exchange). This
  243. is used (when applicable) to keep the fast path of grabbing and releasing
  244. mutexes short.
  245. cmpxchg is basically the following function performed atomically:
  246. unsigned long _cmpxchg(unsigned long *A, unsigned long *B, unsigned long *C)
  247. {
  248. unsigned long T = *A;
  249. if (*A == *B) {
  250. *A = *C;
  251. }
  252. return T;
  253. }
  254. #define cmpxchg(a,b,c) _cmpxchg(&a,&b,&c)
  255. This is really nice to have, since it allows you to only update a variable
  256. if the variable is what you expect it to be. You know if it succeeded if
  257. the return value (the old value of A) is equal to B.
  258. The macro rt_mutex_cmpxchg is used to try to lock and unlock mutexes. If
  259. the architecture does not support CMPXCHG, then this macro is simply set
  260. to fail every time. But if CMPXCHG is supported, then this will
  261. help out extremely to keep the fast path short.
  262. The use of rt_mutex_cmpxchg with the flags in the owner field help optimize
  263. the system for architectures that support it. This will also be explained
  264. later in this document.
  265. Priority adjustments
  266. --------------------
  267. The implementation of the PI code in rtmutex.c has several places that a
  268. process must adjust its priority. With the help of the pi_list of a
  269. process this is rather easy to know what needs to be adjusted.
  270. The functions implementing the task adjustments are rt_mutex_adjust_prio,
  271. __rt_mutex_adjust_prio (same as the former, but expects the task pi_lock
  272. to already be taken), rt_mutex_get_prio, and rt_mutex_setprio.
  273. rt_mutex_getprio and rt_mutex_setprio are only used in __rt_mutex_adjust_prio.
  274. rt_mutex_getprio returns the priority that the task should have. Either the
  275. task's own normal priority, or if a process of a higher priority is waiting on
  276. a mutex owned by the task, then that higher priority should be returned.
  277. Since the pi_list of a task holds an order by priority list of all the top
  278. waiters of all the mutexes that the task owns, rt_mutex_getprio simply needs
  279. to compare the top pi waiter to its own normal priority, and return the higher
  280. priority back.
  281. (Note: if looking at the code, you will notice that the lower number of
  282. prio is returned. This is because the prio field in the task structure
  283. is an inverse order of the actual priority. So a "prio" of 5 is
  284. of higher priority than a "prio" of 10.)
  285. __rt_mutex_adjust_prio examines the result of rt_mutex_getprio, and if the
  286. result does not equal the task's current priority, then rt_mutex_setprio
  287. is called to adjust the priority of the task to the new priority.
  288. Note that rt_mutex_setprio is defined in kernel/sched.c to implement the
  289. actual change in priority.
  290. It is interesting to note that __rt_mutex_adjust_prio can either increase
  291. or decrease the priority of the task. In the case that a higher priority
  292. process has just blocked on a mutex owned by the task, __rt_mutex_adjust_prio
  293. would increase/boost the task's priority. But if a higher priority task
  294. were for some reason to leave the mutex (timeout or signal), this same function
  295. would decrease/unboost the priority of the task. That is because the pi_list
  296. always contains the highest priority task that is waiting on a mutex owned
  297. by the task, so we only need to compare the priority of that top pi waiter
  298. to the normal priority of the given task.
  299. High level overview of the PI chain walk
  300. ----------------------------------------
  301. The PI chain walk is implemented by the function rt_mutex_adjust_prio_chain.
  302. The implementation has gone through several iterations, and has ended up
  303. with what we believe is the best. It walks the PI chain by only grabbing
  304. at most two locks at a time, and is very efficient.
  305. The rt_mutex_adjust_prio_chain can be used either to boost or lower process
  306. priorities.
  307. rt_mutex_adjust_prio_chain is called with a task to be checked for PI
  308. (de)boosting (the owner of a mutex that a process is blocking on), a flag to
  309. check for deadlocking, the mutex that the task owns, and a pointer to a waiter
  310. that is the process's waiter struct that is blocked on the mutex (although this
  311. parameter may be NULL for deboosting).
  312. For this explanation, I will not mention deadlock detection. This explanation
  313. will try to stay at a high level.
  314. When this function is called, there are no locks held. That also means
  315. that the state of the owner and lock can change when entered into this function.
  316. Before this function is called, the task has already had rt_mutex_adjust_prio
  317. performed on it. This means that the task is set to the priority that it
  318. should be at, but the plist nodes of the task's waiter have not been updated
  319. with the new priorities, and that this task may not be in the proper locations
  320. in the pi_lists and wait_lists that the task is blocked on. This function
  321. solves all that.
  322. A loop is entered, where task is the owner to be checked for PI changes that
  323. was passed by parameter (for the first iteration). The pi_lock of this task is
  324. taken to prevent any more changes to the pi_list of the task. This also
  325. prevents new tasks from completing the blocking on a mutex that is owned by this
  326. task.
  327. If the task is not blocked on a mutex then the loop is exited. We are at
  328. the top of the PI chain.
  329. A check is now done to see if the original waiter (the process that is blocked
  330. on the current mutex) is the top pi waiter of the task. That is, is this
  331. waiter on the top of the task's pi_list. If it is not, it either means that
  332. there is another process higher in priority that is blocked on one of the
  333. mutexes that the task owns, or that the waiter has just woken up via a signal
  334. or timeout and has left the PI chain. In either case, the loop is exited, since
  335. we don't need to do any more changes to the priority of the current task, or any
  336. task that owns a mutex that this current task is waiting on. A priority chain
  337. walk is only needed when a new top pi waiter is made to a task.
  338. The next check sees if the task's waiter plist node has the priority equal to
  339. the priority the task is set at. If they are equal, then we are done with
  340. the loop. Remember that the function started with the priority of the
  341. task adjusted, but the plist nodes that hold the task in other processes
  342. pi_lists have not been adjusted.
  343. Next, we look at the mutex that the task is blocked on. The mutex's wait_lock
  344. is taken. This is done by a spin_trylock, because the locking order of the
  345. pi_lock and wait_lock goes in the opposite direction. If we fail to grab the
  346. lock, the pi_lock is released, and we restart the loop.
  347. Now that we have both the pi_lock of the task as well as the wait_lock of
  348. the mutex the task is blocked on, we update the task's waiter's plist node
  349. that is located on the mutex's wait_list.
  350. Now we release the pi_lock of the task.
  351. Next the owner of the mutex has its pi_lock taken, so we can update the
  352. task's entry in the owner's pi_list. If the task is the highest priority
  353. process on the mutex's wait_list, then we remove the previous top waiter
  354. from the owner's pi_list, and replace it with the task.
  355. Note: It is possible that the task was the current top waiter on the mutex,
  356. in which case the task is not yet on the pi_list of the waiter. This
  357. is OK, since plist_del does nothing if the plist node is not on any
  358. list.
  359. If the task was not the top waiter of the mutex, but it was before we
  360. did the priority updates, that means we are deboosting/lowering the
  361. task. In this case, the task is removed from the pi_list of the owner,
  362. and the new top waiter is added.
  363. Lastly, we unlock both the pi_lock of the task, as well as the mutex's
  364. wait_lock, and continue the loop again. On the next iteration of the
  365. loop, the previous owner of the mutex will be the task that will be
  366. processed.
  367. Note: One might think that the owner of this mutex might have changed
  368. since we just grab the mutex's wait_lock. And one could be right.
  369. The important thing to remember is that the owner could not have
  370. become the task that is being processed in the PI chain, since
  371. we have taken that task's pi_lock at the beginning of the loop.
  372. So as long as there is an owner of this mutex that is not the same
  373. process as the tasked being worked on, we are OK.
  374. Looking closely at the code, one might be confused. The check for the
  375. end of the PI chain is when the task isn't blocked on anything or the
  376. task's waiter structure "task" element is NULL. This check is
  377. protected only by the task's pi_lock. But the code to unlock the mutex
  378. sets the task's waiter structure "task" element to NULL with only
  379. the protection of the mutex's wait_lock, which was not taken yet.
  380. Isn't this a race condition if the task becomes the new owner?
  381. The answer is No! The trick is the spin_trylock of the mutex's
  382. wait_lock. If we fail that lock, we release the pi_lock of the
  383. task and continue the loop, doing the end of PI chain check again.
  384. In the code to release the lock, the wait_lock of the mutex is held
  385. the entire time, and it is not let go when we grab the pi_lock of the
  386. new owner of the mutex. So if the switch of a new owner were to happen
  387. after the check for end of the PI chain and the grabbing of the
  388. wait_lock, the unlocking code would spin on the new owner's pi_lock
  389. but never give up the wait_lock. So the PI chain loop is guaranteed to
  390. fail the spin_trylock on the wait_lock, release the pi_lock, and
  391. try again.
  392. If you don't quite understand the above, that's OK. You don't have to,
  393. unless you really want to make a proof out of it ;)
  394. Pending Owners and Lock stealing
  395. --------------------------------
  396. One of the flags in the owner field of the mutex structure is "Pending Owner".
  397. What this means is that an owner was chosen by the process releasing the
  398. mutex, but that owner has yet to wake up and actually take the mutex.
  399. Why is this important? Why can't we just give the mutex to another process
  400. and be done with it?
  401. The PI code is to help with real-time processes, and to let the highest
  402. priority process run as long as possible with little latencies and delays.
  403. If a high priority process owns a mutex that a lower priority process is
  404. blocked on, when the mutex is released it would be given to the lower priority
  405. process. What if the higher priority process wants to take that mutex again.
  406. The high priority process would fail to take that mutex that it just gave up
  407. and it would need to boost the lower priority process to run with full
  408. latency of that critical section (since the low priority process just entered
  409. it).
  410. There's no reason a high priority process that gives up a mutex should be
  411. penalized if it tries to take that mutex again. If the new owner of the
  412. mutex has not woken up yet, there's no reason that the higher priority process
  413. could not take that mutex away.
  414. To solve this, we introduced Pending Ownership and Lock Stealing. When a
  415. new process is given a mutex that it was blocked on, it is only given
  416. pending ownership. This means that it's the new owner, unless a higher
  417. priority process comes in and tries to grab that mutex. If a higher priority
  418. process does come along and wants that mutex, we let the higher priority
  419. process "steal" the mutex from the pending owner (only if it is still pending)
  420. and continue with the mutex.
  421. Taking of a mutex (The walk through)
  422. ------------------------------------
  423. OK, now let's take a look at the detailed walk through of what happens when
  424. taking a mutex.
  425. The first thing that is tried is the fast taking of the mutex. This is
  426. done when we have CMPXCHG enabled (otherwise the fast taking automatically
  427. fails). Only when the owner field of the mutex is NULL can the lock be
  428. taken with the CMPXCHG and nothing else needs to be done.
  429. If there is contention on the lock, whether it is owned or pending owner
  430. we go about the slow path (rt_mutex_slowlock).
  431. The slow path function is where the task's waiter structure is created on
  432. the stack. This is because the waiter structure is only needed for the
  433. scope of this function. The waiter structure holds the nodes to store
  434. the task on the wait_list of the mutex, and if need be, the pi_list of
  435. the owner.
  436. The wait_lock of the mutex is taken since the slow path of unlocking the
  437. mutex also takes this lock.
  438. We then call try_to_take_rt_mutex. This is where the architecture that
  439. does not implement CMPXCHG would always grab the lock (if there's no
  440. contention).
  441. try_to_take_rt_mutex is used every time the task tries to grab a mutex in the
  442. slow path. The first thing that is done here is an atomic setting of
  443. the "Has Waiters" flag of the mutex's owner field. Yes, this could really
  444. be false, because if the mutex has no owner, there are no waiters and
  445. the current task also won't have any waiters. But we don't have the lock
  446. yet, so we assume we are going to be a waiter. The reason for this is to
  447. play nice for those architectures that do have CMPXCHG. By setting this flag
  448. now, the owner of the mutex can't release the mutex without going into the
  449. slow unlock path, and it would then need to grab the wait_lock, which this
  450. code currently holds. So setting the "Has Waiters" flag forces the owner
  451. to synchronize with this code.
  452. Now that we know that we can't have any races with the owner releasing the
  453. mutex, we check to see if we can take the ownership. This is done if the
  454. mutex doesn't have a owner, or if we can steal the mutex from a pending
  455. owner. Let's look at the situations we have here.
  456. 1) Has owner that is pending
  457. ----------------------------
  458. The mutex has a owner, but it hasn't woken up and the mutex flag
  459. "Pending Owner" is set. The first check is to see if the owner isn't the
  460. current task. This is because this function is also used for the pending
  461. owner to grab the mutex. When a pending owner wakes up, it checks to see
  462. if it can take the mutex, and this is done if the owner is already set to
  463. itself. If so, we succeed and leave the function, clearing the "Pending
  464. Owner" bit.
  465. If the pending owner is not current, we check to see if the current priority is
  466. higher than the pending owner. If not, we fail the function and return.
  467. There's also something special about a pending owner. That is a pending owner
  468. is never blocked on a mutex. So there is no PI chain to worry about. It also
  469. means that if the mutex doesn't have any waiters, there's no accounting needed
  470. to update the pending owner's pi_list, since we only worry about processes
  471. blocked on the current mutex.
  472. If there are waiters on this mutex, and we just stole the ownership, we need
  473. to take the top waiter, remove it from the pi_list of the pending owner, and
  474. add it to the current pi_list. Note that at this moment, the pending owner
  475. is no longer on the list of waiters. This is fine, since the pending owner
  476. would add itself back when it realizes that it had the ownership stolen
  477. from itself. When the pending owner tries to grab the mutex, it will fail
  478. in try_to_take_rt_mutex if the owner field points to another process.
  479. 2) No owner
  480. -----------
  481. If there is no owner (or we successfully stole the lock), we set the owner
  482. of the mutex to current, and set the flag of "Has Waiters" if the current
  483. mutex actually has waiters, or we clear the flag if it doesn't. See, it was
  484. OK that we set that flag early, since now it is cleared.
  485. 3) Failed to grab ownership
  486. ---------------------------
  487. The most interesting case is when we fail to take ownership. This means that
  488. there exists an owner, or there's a pending owner with equal or higher
  489. priority than the current task.
  490. We'll continue on the failed case.
  491. If the mutex has a timeout, we set up a timer to go off to break us out
  492. of this mutex if we failed to get it after a specified amount of time.
  493. Now we enter a loop that will continue to try to take ownership of the mutex, or
  494. fail from a timeout or signal.
  495. Once again we try to take the mutex. This will usually fail the first time
  496. in the loop, since it had just failed to get the mutex. But the second time
  497. in the loop, this would likely succeed, since the task would likely be
  498. the pending owner.
  499. If the mutex is TASK_INTERRUPTIBLE a check for signals and timeout is done
  500. here.
  501. The waiter structure has a "task" field that points to the task that is blocked
  502. on the mutex. This field can be NULL the first time it goes through the loop
  503. or if the task is a pending owner and had it's mutex stolen. If the "task"
  504. field is NULL then we need to set up the accounting for it.
  505. Task blocks on mutex
  506. --------------------
  507. The accounting of a mutex and process is done with the waiter structure of
  508. the process. The "task" field is set to the process, and the "lock" field
  509. to the mutex. The plist nodes are initialized to the processes current
  510. priority.
  511. Since the wait_lock was taken at the entry of the slow lock, we can safely
  512. add the waiter to the wait_list. If the current process is the highest
  513. priority process currently waiting on this mutex, then we remove the
  514. previous top waiter process (if it exists) from the pi_list of the owner,
  515. and add the current process to that list. Since the pi_list of the owner
  516. has changed, we call rt_mutex_adjust_prio on the owner to see if the owner
  517. should adjust its priority accordingly.
  518. If the owner is also blocked on a lock, and had its pi_list changed
  519. (or deadlock checking is on), we unlock the wait_lock of the mutex and go ahead
  520. and run rt_mutex_adjust_prio_chain on the owner, as described earlier.
  521. Now all locks are released, and if the current process is still blocked on a
  522. mutex (waiter "task" field is not NULL), then we go to sleep (call schedule).
  523. Waking up in the loop
  524. ---------------------
  525. The schedule can then wake up for a few reasons.
  526. 1) we were given pending ownership of the mutex.
  527. 2) we received a signal and was TASK_INTERRUPTIBLE
  528. 3) we had a timeout and was TASK_INTERRUPTIBLE
  529. In any of these cases, we continue the loop and once again try to grab the
  530. ownership of the mutex. If we succeed, we exit the loop, otherwise we continue
  531. and on signal and timeout, will exit the loop, or if we had the mutex stolen
  532. we just simply add ourselves back on the lists and go back to sleep.
  533. Note: For various reasons, because of timeout and signals, the steal mutex
  534. algorithm needs to be careful. This is because the current process is
  535. still on the wait_list. And because of dynamic changing of priorities,
  536. especially on SCHED_OTHER tasks, the current process can be the
  537. highest priority task on the wait_list.
  538. Failed to get mutex on Timeout or Signal
  539. ----------------------------------------
  540. If a timeout or signal occurred, the waiter's "task" field would not be
  541. NULL and the task needs to be taken off the wait_list of the mutex and perhaps
  542. pi_list of the owner. If this process was a high priority process, then
  543. the rt_mutex_adjust_prio_chain needs to be executed again on the owner,
  544. but this time it will be lowering the priorities.
  545. Unlocking the Mutex
  546. -------------------
  547. The unlocking of a mutex also has a fast path for those architectures with
  548. CMPXCHG. Since the taking of a mutex on contention always sets the
  549. "Has Waiters" flag of the mutex's owner, we use this to know if we need to
  550. take the slow path when unlocking the mutex. If the mutex doesn't have any
  551. waiters, the owner field of the mutex would equal the current process and
  552. the mutex can be unlocked by just replacing the owner field with NULL.
  553. If the owner field has the "Has Waiters" bit set (or CMPXCHG is not available),
  554. the slow unlock path is taken.
  555. The first thing done in the slow unlock path is to take the wait_lock of the
  556. mutex. This synchronizes the locking and unlocking of the mutex.
  557. A check is made to see if the mutex has waiters or not. On architectures that
  558. do not have CMPXCHG, this is the location that the owner of the mutex will
  559. determine if a waiter needs to be awoken or not. On architectures that
  560. do have CMPXCHG, that check is done in the fast path, but it is still needed
  561. in the slow path too. If a waiter of a mutex woke up because of a signal
  562. or timeout between the time the owner failed the fast path CMPXCHG check and
  563. the grabbing of the wait_lock, the mutex may not have any waiters, thus the
  564. owner still needs to make this check. If there are no waiters then the mutex
  565. owner field is set to NULL, the wait_lock is released and nothing more is
  566. needed.
  567. If there are waiters, then we need to wake one up and give that waiter
  568. pending ownership.
  569. On the wake up code, the pi_lock of the current owner is taken. The top
  570. waiter of the lock is found and removed from the wait_list of the mutex
  571. as well as the pi_list of the current owner. The task field of the new
  572. pending owner's waiter structure is set to NULL, and the owner field of the
  573. mutex is set to the new owner with the "Pending Owner" bit set, as well
  574. as the "Has Waiters" bit if there still are other processes blocked on the
  575. mutex.
  576. The pi_lock of the previous owner is released, and the new pending owner's
  577. pi_lock is taken. Remember that this is the trick to prevent the race
  578. condition in rt_mutex_adjust_prio_chain from adding itself as a waiter
  579. on the mutex.
  580. We now clear the "pi_blocked_on" field of the new pending owner, and if
  581. the mutex still has waiters pending, we add the new top waiter to the pi_list
  582. of the pending owner.
  583. Finally we unlock the pi_lock of the pending owner and wake it up.
  584. Contact
  585. -------
  586. For updates on this document, please email Steven Rostedt <rostedt@goodmis.org>
  587. Credits
  588. -------
  589. Author: Steven Rostedt <rostedt@goodmis.org>
  590. Reviewers: Ingo Molnar, Thomas Gleixner, Thomas Duetsch, and Randy Dunlap
  591. Updates
  592. -------
  593. This document was originally written for 2.6.17-rc3-mm1