futex-requeue-pi.rst 5.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132
  1. ================
  2. Futex Requeue PI
  3. ================
  4. Requeueing of tasks from a non-PI futex to a PI futex requires
  5. special handling in order to ensure the underlying rt_mutex is never
  6. left without an owner if it has waiters; doing so would break the PI
  7. boosting logic [see rt-mutex-desgin.txt] For the purposes of
  8. brevity, this action will be referred to as "requeue_pi" throughout
  9. this document. Priority inheritance is abbreviated throughout as
  10. "PI".
  11. Motivation
  12. ----------
  13. Without requeue_pi, the glibc implementation of
  14. pthread_cond_broadcast() must resort to waking all the tasks waiting
  15. on a pthread_condvar and letting them try to sort out which task
  16. gets to run first in classic thundering-herd formation. An ideal
  17. implementation would wake the highest-priority waiter, and leave the
  18. rest to the natural wakeup inherent in unlocking the mutex
  19. associated with the condvar.
  20. Consider the simplified glibc calls::
  21. /* caller must lock mutex */
  22. pthread_cond_wait(cond, mutex)
  23. {
  24. lock(cond->__data.__lock);
  25. unlock(mutex);
  26. do {
  27. unlock(cond->__data.__lock);
  28. futex_wait(cond->__data.__futex);
  29. lock(cond->__data.__lock);
  30. } while(...)
  31. unlock(cond->__data.__lock);
  32. lock(mutex);
  33. }
  34. pthread_cond_broadcast(cond)
  35. {
  36. lock(cond->__data.__lock);
  37. unlock(cond->__data.__lock);
  38. futex_requeue(cond->data.__futex, cond->mutex);
  39. }
  40. Once pthread_cond_broadcast() requeues the tasks, the cond->mutex
  41. has waiters. Note that pthread_cond_wait() attempts to lock the
  42. mutex only after it has returned to user space. This will leave the
  43. underlying rt_mutex with waiters, and no owner, breaking the
  44. previously mentioned PI-boosting algorithms.
  45. In order to support PI-aware pthread_condvar's, the kernel needs to
  46. be able to requeue tasks to PI futexes. This support implies that
  47. upon a successful futex_wait system call, the caller would return to
  48. user space already holding the PI futex. The glibc implementation
  49. would be modified as follows::
  50. /* caller must lock mutex */
  51. pthread_cond_wait_pi(cond, mutex)
  52. {
  53. lock(cond->__data.__lock);
  54. unlock(mutex);
  55. do {
  56. unlock(cond->__data.__lock);
  57. futex_wait_requeue_pi(cond->__data.__futex);
  58. lock(cond->__data.__lock);
  59. } while(...)
  60. unlock(cond->__data.__lock);
  61. /* the kernel acquired the mutex for us */
  62. }
  63. pthread_cond_broadcast_pi(cond)
  64. {
  65. lock(cond->__data.__lock);
  66. unlock(cond->__data.__lock);
  67. futex_requeue_pi(cond->data.__futex, cond->mutex);
  68. }
  69. The actual glibc implementation will likely test for PI and make the
  70. necessary changes inside the existing calls rather than creating new
  71. calls for the PI cases. Similar changes are needed for
  72. pthread_cond_timedwait() and pthread_cond_signal().
  73. Implementation
  74. --------------
  75. In order to ensure the rt_mutex has an owner if it has waiters, it
  76. is necessary for both the requeue code, as well as the waiting code,
  77. to be able to acquire the rt_mutex before returning to user space.
  78. The requeue code cannot simply wake the waiter and leave it to
  79. acquire the rt_mutex as it would open a race window between the
  80. requeue call returning to user space and the waiter waking and
  81. starting to run. This is especially true in the uncontended case.
  82. The solution involves two new rt_mutex helper routines,
  83. rt_mutex_start_proxy_lock() and rt_mutex_finish_proxy_lock(), which
  84. allow the requeue code to acquire an uncontended rt_mutex on behalf
  85. of the waiter and to enqueue the waiter on a contended rt_mutex.
  86. Two new system calls provide the kernel<->user interface to
  87. requeue_pi: FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI.
  88. FUTEX_WAIT_REQUEUE_PI is called by the waiter (pthread_cond_wait()
  89. and pthread_cond_timedwait()) to block on the initial futex and wait
  90. to be requeued to a PI-aware futex. The implementation is the
  91. result of a high-speed collision between futex_wait() and
  92. futex_lock_pi(), with some extra logic to check for the additional
  93. wake-up scenarios.
  94. FUTEX_CMP_REQUEUE_PI is called by the waker
  95. (pthread_cond_broadcast() and pthread_cond_signal()) to requeue and
  96. possibly wake the waiting tasks. Internally, this system call is
  97. still handled by futex_requeue (by passing requeue_pi=1). Before
  98. requeueing, futex_requeue() attempts to acquire the requeue target
  99. PI futex on behalf of the top waiter. If it can, this waiter is
  100. woken. futex_requeue() then proceeds to requeue the remaining
  101. nr_wake+nr_requeue tasks to the PI futex, calling
  102. rt_mutex_start_proxy_lock() prior to each requeue to prepare the
  103. task as a waiter on the underlying rt_mutex. It is possible that
  104. the lock can be acquired at this stage as well, if so, the next
  105. waiter is woken to finish the acquisition of the lock.
  106. FUTEX_CMP_REQUEUE_PI accepts nr_wake and nr_requeue as arguments, but
  107. their sum is all that really matters. futex_requeue() will wake or
  108. requeue up to nr_wake + nr_requeue tasks. It will wake only as many
  109. tasks as it can acquire the lock for, which in the majority of cases
  110. should be 0 as good programming practice dictates that the caller of
  111. either pthread_cond_broadcast() or pthread_cond_signal() acquire the
  112. mutex prior to making the call. FUTEX_CMP_REQUEUE_PI requires that
  113. nr_wake=1. nr_requeue should be INT_MAX for broadcast and 0 for
  114. signal.