pin_user_pages.rst 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279
  1. .. SPDX-License-Identifier: GPL-2.0
  2. ====================================================
  3. pin_user_pages() and related calls
  4. ====================================================
  5. .. contents:: :local:
  6. Overview
  7. ========
  8. This document describes the following functions::
  9. pin_user_pages()
  10. pin_user_pages_fast()
  11. pin_user_pages_remote()
  12. Basic description of FOLL_PIN
  13. =============================
  14. FOLL_PIN and FOLL_LONGTERM are flags that can be passed to the get_user_pages*()
  15. ("gup") family of functions. FOLL_PIN has significant interactions and
  16. interdependencies with FOLL_LONGTERM, so both are covered here.
  17. FOLL_PIN is internal to gup, meaning that it should not appear at the gup call
  18. sites. This allows the associated wrapper functions (pin_user_pages*() and
  19. others) to set the correct combination of these flags, and to check for problems
  20. as well.
  21. FOLL_LONGTERM, on the other hand, *is* allowed to be set at the gup call sites.
  22. This is in order to avoid creating a large number of wrapper functions to cover
  23. all combinations of get*(), pin*(), FOLL_LONGTERM, and more. Also, the
  24. pin_user_pages*() APIs are clearly distinct from the get_user_pages*() APIs, so
  25. that's a natural dividing line, and a good point to make separate wrapper calls.
  26. In other words, use pin_user_pages*() for DMA-pinned pages, and
  27. get_user_pages*() for other cases. There are five cases described later on in
  28. this document, to further clarify that concept.
  29. FOLL_PIN and FOLL_GET are mutually exclusive for a given gup call. However,
  30. multiple threads and call sites are free to pin the same struct pages, via both
  31. FOLL_PIN and FOLL_GET. It's just the call site that needs to choose one or the
  32. other, not the struct page(s).
  33. The FOLL_PIN implementation is nearly the same as FOLL_GET, except that FOLL_PIN
  34. uses a different reference counting technique.
  35. FOLL_PIN is a prerequisite to FOLL_LONGTERM. Another way of saying that is,
  36. FOLL_LONGTERM is a specific case, more restrictive case of FOLL_PIN.
  37. Which flags are set by each wrapper
  38. ===================================
  39. For these pin_user_pages*() functions, FOLL_PIN is OR'd in with whatever gup
  40. flags the caller provides. The caller is required to pass in a non-null struct
  41. pages* array, and the function then pins pages by incrementing each by a special
  42. value: GUP_PIN_COUNTING_BIAS.
  43. For huge pages (and in fact, any compound page of more than 2 pages), the
  44. GUP_PIN_COUNTING_BIAS scheme is not used. Instead, an exact form of pin counting
  45. is achieved, by using the 3rd struct page in the compound page. A new struct
  46. page field, hpage_pinned_refcount, has been added in order to support this.
  47. This approach for compound pages avoids the counting upper limit problems that
  48. are discussed below. Those limitations would have been aggravated severely by
  49. huge pages, because each tail page adds a refcount to the head page. And in
  50. fact, testing revealed that, without a separate hpage_pinned_refcount field,
  51. page overflows were seen in some huge page stress tests.
  52. This also means that huge pages and compound pages (of order > 1) do not suffer
  53. from the false positives problem that is mentioned below.::
  54. Function
  55. --------
  56. pin_user_pages FOLL_PIN is always set internally by this function.
  57. pin_user_pages_fast FOLL_PIN is always set internally by this function.
  58. pin_user_pages_remote FOLL_PIN is always set internally by this function.
  59. For these get_user_pages*() functions, FOLL_GET might not even be specified.
  60. Behavior is a little more complex than above. If FOLL_GET was *not* specified,
  61. but the caller passed in a non-null struct pages* array, then the function
  62. sets FOLL_GET for you, and proceeds to pin pages by incrementing the refcount
  63. of each page by +1.::
  64. Function
  65. --------
  66. get_user_pages FOLL_GET is sometimes set internally by this function.
  67. get_user_pages_fast FOLL_GET is sometimes set internally by this function.
  68. get_user_pages_remote FOLL_GET is sometimes set internally by this function.
  69. Tracking dma-pinned pages
  70. =========================
  71. Some of the key design constraints, and solutions, for tracking dma-pinned
  72. pages:
  73. * An actual reference count, per struct page, is required. This is because
  74. multiple processes may pin and unpin a page.
  75. * False positives (reporting that a page is dma-pinned, when in fact it is not)
  76. are acceptable, but false negatives are not.
  77. * struct page may not be increased in size for this, and all fields are already
  78. used.
  79. * Given the above, we can overload the page->_refcount field by using, sort of,
  80. the upper bits in that field for a dma-pinned count. "Sort of", means that,
  81. rather than dividing page->_refcount into bit fields, we simple add a medium-
  82. large value (GUP_PIN_COUNTING_BIAS, initially chosen to be 1024: 10 bits) to
  83. page->_refcount. This provides fuzzy behavior: if a page has get_page() called
  84. on it 1024 times, then it will appear to have a single dma-pinned count.
  85. And again, that's acceptable.
  86. This also leads to limitations: there are only 31-10==21 bits available for a
  87. counter that increments 10 bits at a time.
  88. * Callers must specifically request "dma-pinned tracking of pages". In other
  89. words, just calling get_user_pages() will not suffice; a new set of functions,
  90. pin_user_page() and related, must be used.
  91. FOLL_PIN, FOLL_GET, FOLL_LONGTERM: when to use which flags
  92. ==========================================================
  93. Thanks to Jan Kara, Vlastimil Babka and several other -mm people, for describing
  94. these categories:
  95. CASE 1: Direct IO (DIO)
  96. -----------------------
  97. There are GUP references to pages that are serving
  98. as DIO buffers. These buffers are needed for a relatively short time (so they
  99. are not "long term"). No special synchronization with page_mkclean() or
  100. munmap() is provided. Therefore, flags to set at the call site are: ::
  101. FOLL_PIN
  102. ...but rather than setting FOLL_PIN directly, call sites should use one of
  103. the pin_user_pages*() routines that set FOLL_PIN.
  104. CASE 2: RDMA
  105. ------------
  106. There are GUP references to pages that are serving as DMA
  107. buffers. These buffers are needed for a long time ("long term"). No special
  108. synchronization with page_mkclean() or munmap() is provided. Therefore, flags
  109. to set at the call site are: ::
  110. FOLL_PIN | FOLL_LONGTERM
  111. NOTE: Some pages, such as DAX pages, cannot be pinned with longterm pins. That's
  112. because DAX pages do not have a separate page cache, and so "pinning" implies
  113. locking down file system blocks, which is not (yet) supported in that way.
  114. CASE 3: MMU notifier registration, with or without page faulting hardware
  115. -------------------------------------------------------------------------
  116. Device drivers can pin pages via get_user_pages*(), and register for mmu
  117. notifier callbacks for the memory range. Then, upon receiving a notifier
  118. "invalidate range" callback , stop the device from using the range, and unpin
  119. the pages. There may be other possible schemes, such as for example explicitly
  120. synchronizing against pending IO, that accomplish approximately the same thing.
  121. Or, if the hardware supports replayable page faults, then the device driver can
  122. avoid pinning entirely (this is ideal), as follows: register for mmu notifier
  123. callbacks as above, but instead of stopping the device and unpinning in the
  124. callback, simply remove the range from the device's page tables.
  125. Either way, as long as the driver unpins the pages upon mmu notifier callback,
  126. then there is proper synchronization with both filesystem and mm
  127. (page_mkclean(), munmap(), etc). Therefore, neither flag needs to be set.
  128. CASE 4: Pinning for struct page manipulation only
  129. -------------------------------------------------
  130. If only struct page data (as opposed to the actual memory contents that a page
  131. is tracking) is affected, then normal GUP calls are sufficient, and neither flag
  132. needs to be set.
  133. CASE 5: Pinning in order to write to the data within the page
  134. -------------------------------------------------------------
  135. Even though neither DMA nor Direct IO is involved, just a simple case of "pin,
  136. write to a page's data, unpin" can cause a problem. Case 5 may be considered a
  137. superset of Case 1, plus Case 2, plus anything that invokes that pattern. In
  138. other words, if the code is neither Case 1 nor Case 2, it may still require
  139. FOLL_PIN, for patterns like this:
  140. Correct (uses FOLL_PIN calls):
  141. pin_user_pages()
  142. write to the data within the pages
  143. unpin_user_pages()
  144. INCORRECT (uses FOLL_GET calls):
  145. get_user_pages()
  146. write to the data within the pages
  147. put_page()
  148. page_maybe_dma_pinned(): the whole point of pinning
  149. ===================================================
  150. The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able
  151. to query, "is this page DMA-pinned?" That allows code such as page_mkclean()
  152. (and file system writeback code in general) to make informed decisions about
  153. what to do when a page cannot be unmapped due to such pins.
  154. What to do in those cases is the subject of a years-long series of discussions
  155. and debates (see the References at the end of this document). It's a TODO item
  156. here: fill in the details once that's worked out. Meanwhile, it's safe to say
  157. that having this available: ::
  158. static inline bool page_maybe_dma_pinned(struct page *page)
  159. ...is a prerequisite to solving the long-running gup+DMA problem.
  160. Another way of thinking about FOLL_GET, FOLL_PIN, and FOLL_LONGTERM
  161. ===================================================================
  162. Another way of thinking about these flags is as a progression of restrictions:
  163. FOLL_GET is for struct page manipulation, without affecting the data that the
  164. struct page refers to. FOLL_PIN is a *replacement* for FOLL_GET, and is for
  165. short term pins on pages whose data *will* get accessed. As such, FOLL_PIN is
  166. a "more severe" form of pinning. And finally, FOLL_LONGTERM is an even more
  167. restrictive case that has FOLL_PIN as a prerequisite: this is for pages that
  168. will be pinned longterm, and whose data will be accessed.
  169. Unit testing
  170. ============
  171. This file::
  172. tools/testing/selftests/vm/gup_benchmark.c
  173. has the following new calls to exercise the new pin*() wrapper functions:
  174. * PIN_FAST_BENCHMARK (./gup_benchmark -a)
  175. * PIN_BENCHMARK (./gup_benchmark -b)
  176. You can monitor how many total dma-pinned pages have been acquired and released
  177. since the system was booted, via two new /proc/vmstat entries: ::
  178. /proc/vmstat/nr_foll_pin_acquired
  179. /proc/vmstat/nr_foll_pin_released
  180. Under normal conditions, these two values will be equal unless there are any
  181. long-term [R]DMA pins in place, or during pin/unpin transitions.
  182. * nr_foll_pin_acquired: This is the number of logical pins that have been
  183. acquired since the system was powered on. For huge pages, the head page is
  184. pinned once for each page (head page and each tail page) within the huge page.
  185. This follows the same sort of behavior that get_user_pages() uses for huge
  186. pages: the head page is refcounted once for each tail or head page in the huge
  187. page, when get_user_pages() is applied to a huge page.
  188. * nr_foll_pin_released: The number of logical pins that have been released since
  189. the system was powered on. Note that pages are released (unpinned) on a
  190. PAGE_SIZE granularity, even if the original pin was applied to a huge page.
  191. Becaused of the pin count behavior described above in "nr_foll_pin_acquired",
  192. the accounting balances out, so that after doing this::
  193. pin_user_pages(huge_page);
  194. for (each page in huge_page)
  195. unpin_user_page(page);
  196. ...the following is expected::
  197. nr_foll_pin_released == nr_foll_pin_acquired
  198. (...unless it was already out of balance due to a long-term RDMA pin being in
  199. place.)
  200. Other diagnostics
  201. =================
  202. dump_page() has been enhanced slightly, to handle these new counting fields, and
  203. to better report on compound pages in general. Specifically, for compound pages
  204. with order > 1, the exact (hpage_pinned_refcount) pincount is reported.
  205. References
  206. ==========
  207. * `Some slow progress on get_user_pages() (Apr 2, 2019) <https://lwn.net/Articles/784574/>`_
  208. * `DMA and get_user_pages() (LPC: Dec 12, 2018) <https://lwn.net/Articles/774411/>`_
  209. * `The trouble with get_user_pages() (Apr 30, 2018) <https://lwn.net/Articles/753027/>`_
  210. * `LWN kernel index: get_user_pages() <https://lwn.net/Kernel/Index/#Memory_management-get_user_pages>`_
  211. John Hubbard, October, 2019