frontswap.rst 15 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293
  1. .. _frontswap:
  2. =========
  3. Frontswap
  4. =========
  5. Frontswap provides a "transcendent memory" interface for swap pages.
  6. In some environments, dramatic performance savings may be obtained because
  7. swapped pages are saved in RAM (or a RAM-like device) instead of a swap disk.
  8. (Note, frontswap -- and :ref:`cleancache` (merged at 3.0) -- are the "frontends"
  9. and the only necessary changes to the core kernel for transcendent memory;
  10. all other supporting code -- the "backends" -- is implemented as drivers.
  11. See the LWN.net article `Transcendent memory in a nutshell`_
  12. for a detailed overview of frontswap and related kernel parts)
  13. .. _Transcendent memory in a nutshell: https://lwn.net/Articles/454795/
  14. Frontswap is so named because it can be thought of as the opposite of
  15. a "backing" store for a swap device. The storage is assumed to be
  16. a synchronous concurrency-safe page-oriented "pseudo-RAM device" conforming
  17. to the requirements of transcendent memory (such as Xen's "tmem", or
  18. in-kernel compressed memory, aka "zcache", or future RAM-like devices);
  19. this pseudo-RAM device is not directly accessible or addressable by the
  20. kernel and is of unknown and possibly time-varying size. The driver
  21. links itself to frontswap by calling frontswap_register_ops to set the
  22. frontswap_ops funcs appropriately and the functions it provides must
  23. conform to certain policies as follows:
  24. An "init" prepares the device to receive frontswap pages associated
  25. with the specified swap device number (aka "type"). A "store" will
  26. copy the page to transcendent memory and associate it with the type and
  27. offset associated with the page. A "load" will copy the page, if found,
  28. from transcendent memory into kernel memory, but will NOT remove the page
  29. from transcendent memory. An "invalidate_page" will remove the page
  30. from transcendent memory and an "invalidate_area" will remove ALL pages
  31. associated with the swap type (e.g., like swapoff) and notify the "device"
  32. to refuse further stores with that swap type.
  33. Once a page is successfully stored, a matching load on the page will normally
  34. succeed. So when the kernel finds itself in a situation where it needs
  35. to swap out a page, it first attempts to use frontswap. If the store returns
  36. success, the data has been successfully saved to transcendent memory and
  37. a disk write and, if the data is later read back, a disk read are avoided.
  38. If a store returns failure, transcendent memory has rejected the data, and the
  39. page can be written to swap as usual.
  40. If a backend chooses, frontswap can be configured as a "writethrough
  41. cache" by calling frontswap_writethrough(). In this mode, the reduction
  42. in swap device writes is lost (and also a non-trivial performance advantage)
  43. in order to allow the backend to arbitrarily "reclaim" space used to
  44. store frontswap pages to more completely manage its memory usage.
  45. Note that if a page is stored and the page already exists in transcendent memory
  46. (a "duplicate" store), either the store succeeds and the data is overwritten,
  47. or the store fails AND the page is invalidated. This ensures stale data may
  48. never be obtained from frontswap.
  49. If properly configured, monitoring of frontswap is done via debugfs in
  50. the `/sys/kernel/debug/frontswap` directory. The effectiveness of
  51. frontswap can be measured (across all swap devices) with:
  52. ``failed_stores``
  53. how many store attempts have failed
  54. ``loads``
  55. how many loads were attempted (all should succeed)
  56. ``succ_stores``
  57. how many store attempts have succeeded
  58. ``invalidates``
  59. how many invalidates were attempted
  60. A backend implementation may provide additional metrics.
  61. FAQ
  62. ===
  63. * Where's the value?
  64. When a workload starts swapping, performance falls through the floor.
  65. Frontswap significantly increases performance in many such workloads by
  66. providing a clean, dynamic interface to read and write swap pages to
  67. "transcendent memory" that is otherwise not directly addressable to the kernel.
  68. This interface is ideal when data is transformed to a different form
  69. and size (such as with compression) or secretly moved (as might be
  70. useful for write-balancing for some RAM-like devices). Swap pages (and
  71. evicted page-cache pages) are a great use for this kind of slower-than-RAM-
  72. but-much-faster-than-disk "pseudo-RAM device" and the frontswap (and
  73. cleancache) interface to transcendent memory provides a nice way to read
  74. and write -- and indirectly "name" -- the pages.
  75. Frontswap -- and cleancache -- with a fairly small impact on the kernel,
  76. provides a huge amount of flexibility for more dynamic, flexible RAM
  77. utilization in various system configurations:
  78. In the single kernel case, aka "zcache", pages are compressed and
  79. stored in local memory, thus increasing the total anonymous pages
  80. that can be safely kept in RAM. Zcache essentially trades off CPU
  81. cycles used in compression/decompression for better memory utilization.
  82. Benchmarks have shown little or no impact when memory pressure is
  83. low while providing a significant performance improvement (25%+)
  84. on some workloads under high memory pressure.
  85. "RAMster" builds on zcache by adding "peer-to-peer" transcendent memory
  86. support for clustered systems. Frontswap pages are locally compressed
  87. as in zcache, but then "remotified" to another system's RAM. This
  88. allows RAM to be dynamically load-balanced back-and-forth as needed,
  89. i.e. when system A is overcommitted, it can swap to system B, and
  90. vice versa. RAMster can also be configured as a memory server so
  91. many servers in a cluster can swap, dynamically as needed, to a single
  92. server configured with a large amount of RAM... without pre-configuring
  93. how much of the RAM is available for each of the clients!
  94. In the virtual case, the whole point of virtualization is to statistically
  95. multiplex physical resources across the varying demands of multiple
  96. virtual machines. This is really hard to do with RAM and efforts to do
  97. it well with no kernel changes have essentially failed (except in some
  98. well-publicized special-case workloads).
  99. Specifically, the Xen Transcendent Memory backend allows otherwise
  100. "fallow" hypervisor-owned RAM to not only be "time-shared" between multiple
  101. virtual machines, but the pages can be compressed and deduplicated to
  102. optimize RAM utilization. And when guest OS's are induced to surrender
  103. underutilized RAM (e.g. with "selfballooning"), sudden unexpected
  104. memory pressure may result in swapping; frontswap allows those pages
  105. to be swapped to and from hypervisor RAM (if overall host system memory
  106. conditions allow), thus mitigating the potentially awful performance impact
  107. of unplanned swapping.
  108. A KVM implementation is underway and has been RFC'ed to lkml. And,
  109. using frontswap, investigation is also underway on the use of NVM as
  110. a memory extension technology.
  111. * Sure there may be performance advantages in some situations, but
  112. what's the space/time overhead of frontswap?
  113. If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles into
  114. nothingness and the only overhead is a few extra bytes per swapon'ed
  115. swap device. If CONFIG_FRONTSWAP is enabled but no frontswap "backend"
  116. registers, there is one extra global variable compared to zero for
  117. every swap page read or written. If CONFIG_FRONTSWAP is enabled
  118. AND a frontswap backend registers AND the backend fails every "store"
  119. request (i.e. provides no memory despite claiming it might),
  120. CPU overhead is still negligible -- and since every frontswap fail
  121. precedes a swap page write-to-disk, the system is highly likely
  122. to be I/O bound and using a small fraction of a percent of a CPU
  123. will be irrelevant anyway.
  124. As for space, if CONFIG_FRONTSWAP is enabled AND a frontswap backend
  125. registers, one bit is allocated for every swap page for every swap
  126. device that is swapon'd. This is added to the EIGHT bits (which
  127. was sixteen until about 2.6.34) that the kernel already allocates
  128. for every swap page for every swap device that is swapon'd. (Hugh
  129. Dickins has observed that frontswap could probably steal one of
  130. the existing eight bits, but let's worry about that minor optimization
  131. later.) For very large swap disks (which are rare) on a standard
  132. 4K pagesize, this is 1MB per 32GB swap.
  133. When swap pages are stored in transcendent memory instead of written
  134. out to disk, there is a side effect that this may create more memory
  135. pressure that can potentially outweigh the other advantages. A
  136. backend, such as zcache, must implement policies to carefully (but
  137. dynamically) manage memory limits to ensure this doesn't happen.
  138. * OK, how about a quick overview of what this frontswap patch does
  139. in terms that a kernel hacker can grok?
  140. Let's assume that a frontswap "backend" has registered during
  141. kernel initialization; this registration indicates that this
  142. frontswap backend has access to some "memory" that is not directly
  143. accessible by the kernel. Exactly how much memory it provides is
  144. entirely dynamic and random.
  145. Whenever a swap-device is swapon'd frontswap_init() is called,
  146. passing the swap device number (aka "type") as a parameter.
  147. This notifies frontswap to expect attempts to "store" swap pages
  148. associated with that number.
  149. Whenever the swap subsystem is readying a page to write to a swap
  150. device (c.f swap_writepage()), frontswap_store is called. Frontswap
  151. consults with the frontswap backend and if the backend says it does NOT
  152. have room, frontswap_store returns -1 and the kernel swaps the page
  153. to the swap device as normal. Note that the response from the frontswap
  154. backend is unpredictable to the kernel; it may choose to never accept a
  155. page, it could accept every ninth page, or it might accept every
  156. page. But if the backend does accept a page, the data from the page
  157. has already been copied and associated with the type and offset,
  158. and the backend guarantees the persistence of the data. In this case,
  159. frontswap sets a bit in the "frontswap_map" for the swap device
  160. corresponding to the page offset on the swap device to which it would
  161. otherwise have written the data.
  162. When the swap subsystem needs to swap-in a page (swap_readpage()),
  163. it first calls frontswap_load() which checks the frontswap_map to
  164. see if the page was earlier accepted by the frontswap backend. If
  165. it was, the page of data is filled from the frontswap backend and
  166. the swap-in is complete. If not, the normal swap-in code is
  167. executed to obtain the page of data from the real swap device.
  168. So every time the frontswap backend accepts a page, a swap device read
  169. and (potentially) a swap device write are replaced by a "frontswap backend
  170. store" and (possibly) a "frontswap backend loads", which are presumably much
  171. faster.
  172. * Can't frontswap be configured as a "special" swap device that is
  173. just higher priority than any real swap device (e.g. like zswap,
  174. or maybe swap-over-nbd/NFS)?
  175. No. First, the existing swap subsystem doesn't allow for any kind of
  176. swap hierarchy. Perhaps it could be rewritten to accommodate a hierarchy,
  177. but this would require fairly drastic changes. Even if it were
  178. rewritten, the existing swap subsystem uses the block I/O layer which
  179. assumes a swap device is fixed size and any page in it is linearly
  180. addressable. Frontswap barely touches the existing swap subsystem,
  181. and works around the constraints of the block I/O subsystem to provide
  182. a great deal of flexibility and dynamicity.
  183. For example, the acceptance of any swap page by the frontswap backend is
  184. entirely unpredictable. This is critical to the definition of frontswap
  185. backends because it grants completely dynamic discretion to the
  186. backend. In zcache, one cannot know a priori how compressible a page is.
  187. "Poorly" compressible pages can be rejected, and "poorly" can itself be
  188. defined dynamically depending on current memory constraints.
  189. Further, frontswap is entirely synchronous whereas a real swap
  190. device is, by definition, asynchronous and uses block I/O. The
  191. block I/O layer is not only unnecessary, but may perform "optimizations"
  192. that are inappropriate for a RAM-oriented device including delaying
  193. the write of some pages for a significant amount of time. Synchrony is
  194. required to ensure the dynamicity of the backend and to avoid thorny race
  195. conditions that would unnecessarily and greatly complicate frontswap
  196. and/or the block I/O subsystem. That said, only the initial "store"
  197. and "load" operations need be synchronous. A separate asynchronous thread
  198. is free to manipulate the pages stored by frontswap. For example,
  199. the "remotification" thread in RAMster uses standard asynchronous
  200. kernel sockets to move compressed frontswap pages to a remote machine.
  201. Similarly, a KVM guest-side implementation could do in-guest compression
  202. and use "batched" hypercalls.
  203. In a virtualized environment, the dynamicity allows the hypervisor
  204. (or host OS) to do "intelligent overcommit". For example, it can
  205. choose to accept pages only until host-swapping might be imminent,
  206. then force guests to do their own swapping.
  207. There is a downside to the transcendent memory specifications for
  208. frontswap: Since any "store" might fail, there must always be a real
  209. slot on a real swap device to swap the page. Thus frontswap must be
  210. implemented as a "shadow" to every swapon'd device with the potential
  211. capability of holding every page that the swap device might have held
  212. and the possibility that it might hold no pages at all. This means
  213. that frontswap cannot contain more pages than the total of swapon'd
  214. swap devices. For example, if NO swap device is configured on some
  215. installation, frontswap is useless. Swapless portable devices
  216. can still use frontswap but a backend for such devices must configure
  217. some kind of "ghost" swap device and ensure that it is never used.
  218. * Why this weird definition about "duplicate stores"? If a page
  219. has been previously successfully stored, can't it always be
  220. successfully overwritten?
  221. Nearly always it can, but no, sometimes it cannot. Consider an example
  222. where data is compressed and the original 4K page has been compressed
  223. to 1K. Now an attempt is made to overwrite the page with data that
  224. is non-compressible and so would take the entire 4K. But the backend
  225. has no more space. In this case, the store must be rejected. Whenever
  226. frontswap rejects a store that would overwrite, it also must invalidate
  227. the old data and ensure that it is no longer accessible. Since the
  228. swap subsystem then writes the new data to the read swap device,
  229. this is the correct course of action to ensure coherency.
  230. * What is frontswap_shrink for?
  231. When the (non-frontswap) swap subsystem swaps out a page to a real
  232. swap device, that page is only taking up low-value pre-allocated disk
  233. space. But if frontswap has placed a page in transcendent memory, that
  234. page may be taking up valuable real estate. The frontswap_shrink
  235. routine allows code outside of the swap subsystem to force pages out
  236. of the memory managed by frontswap and back into kernel-addressable memory.
  237. For example, in RAMster, a "suction driver" thread will attempt
  238. to "repatriate" pages sent to a remote machine back to the local machine;
  239. this is driven using the frontswap_shrink mechanism when memory pressure
  240. subsides.
  241. * Why does the frontswap patch create the new include file swapfile.h?
  242. The frontswap code depends on some swap-subsystem-internal data
  243. structures that have, over the years, moved back and forth between
  244. static and global. This seemed a reasonable compromise: Define
  245. them as global but declare them in a new include file that isn't
  246. included by the large number of source files that include swap.h.
  247. Dan Magenheimer, last updated April 9, 2012