slub.rst 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388
  1. .. _slub:
  2. ==========================
  3. Short users guide for SLUB
  4. ==========================
  5. The basic philosophy of SLUB is very different from SLAB. SLAB
  6. requires rebuilding the kernel to activate debug options for all
  7. slab caches. SLUB always includes full debugging but it is off by default.
  8. SLUB can enable debugging only for selected slabs in order to avoid
  9. an impact on overall system performance which may make a bug more
  10. difficult to find.
  11. In order to switch debugging on one can add an option ``slub_debug``
  12. to the kernel command line. That will enable full debugging for
  13. all slabs.
  14. Typically one would then use the ``slabinfo`` command to get statistical
  15. data and perform operation on the slabs. By default ``slabinfo`` only lists
  16. slabs that have data in them. See "slabinfo -h" for more options when
  17. running the command. ``slabinfo`` can be compiled with
  18. ::
  19. gcc -o slabinfo tools/vm/slabinfo.c
  20. Some of the modes of operation of ``slabinfo`` require that slub debugging
  21. be enabled on the command line. F.e. no tracking information will be
  22. available without debugging on and validation can only partially
  23. be performed if debugging was not switched on.
  24. Some more sophisticated uses of slub_debug:
  25. -------------------------------------------
  26. Parameters may be given to ``slub_debug``. If none is specified then full
  27. debugging is enabled. Format:
  28. slub_debug=<Debug-Options>
  29. Enable options for all slabs
  30. slub_debug=<Debug-Options>,<slab name1>,<slab name2>,...
  31. Enable options only for select slabs (no spaces
  32. after a comma)
  33. Multiple blocks of options for all slabs or selected slabs can be given, with
  34. blocks of options delimited by ';'. The last of "all slabs" blocks is applied
  35. to all slabs except those that match one of the "select slabs" block. Options
  36. of the first "select slabs" blocks that matches the slab's name are applied.
  37. Possible debug options are::
  38. F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS
  39. Sorry SLAB legacy issues)
  40. Z Red zoning
  41. P Poisoning (object and padding)
  42. U User tracking (free and alloc)
  43. T Trace (please only use on single slabs)
  44. A Enable failslab filter mark for the cache
  45. O Switch debugging off for caches that would have
  46. caused higher minimum slab orders
  47. - Switch all debugging off (useful if the kernel is
  48. configured with CONFIG_SLUB_DEBUG_ON)
  49. F.e. in order to boot just with sanity checks and red zoning one would specify::
  50. slub_debug=FZ
  51. Trying to find an issue in the dentry cache? Try::
  52. slub_debug=,dentry
  53. to only enable debugging on the dentry cache. You may use an asterisk at the
  54. end of the slab name, in order to cover all slabs with the same prefix. For
  55. example, here's how you can poison the dentry cache as well as all kmalloc
  56. slabs::
  57. slub_debug=P,kmalloc-*,dentry
  58. Red zoning and tracking may realign the slab. We can just apply sanity checks
  59. to the dentry cache with::
  60. slub_debug=F,dentry
  61. Debugging options may require the minimum possible slab order to increase as
  62. a result of storing the metadata (for example, caches with PAGE_SIZE object
  63. sizes). This has a higher liklihood of resulting in slab allocation errors
  64. in low memory situations or if there's high fragmentation of memory. To
  65. switch off debugging for such caches by default, use::
  66. slub_debug=O
  67. You can apply different options to different list of slab names, using blocks
  68. of options. This will enable red zoning for dentry and user tracking for
  69. kmalloc. All other slabs will not get any debugging enabled::
  70. slub_debug=Z,dentry;U,kmalloc-*
  71. You can also enable options (e.g. sanity checks and poisoning) for all caches
  72. except some that are deemed too performance critical and don't need to be
  73. debugged by specifying global debug options followed by a list of slab names
  74. with "-" as options::
  75. slub_debug=FZ;-,zs_handle,zspage
  76. The state of each debug option for a slab can be found in the respective files
  77. under::
  78. /sys/kernel/slab/<slab name>/
  79. If the file contains 1, the option is enabled, 0 means disabled. The debug
  80. options from the ``slub_debug`` parameter translate to the following files::
  81. F sanity_checks
  82. Z red_zone
  83. P poison
  84. U store_user
  85. T trace
  86. A failslab
  87. Careful with tracing: It may spew out lots of information and never stop if
  88. used on the wrong slab.
  89. Slab merging
  90. ============
  91. If no debug options are specified then SLUB may merge similar slabs together
  92. in order to reduce overhead and increase cache hotness of objects.
  93. ``slabinfo -a`` displays which slabs were merged together.
  94. Slab validation
  95. ===============
  96. SLUB can validate all object if the kernel was booted with slub_debug. In
  97. order to do so you must have the ``slabinfo`` tool. Then you can do
  98. ::
  99. slabinfo -v
  100. which will test all objects. Output will be generated to the syslog.
  101. This also works in a more limited way if boot was without slab debug.
  102. In that case ``slabinfo -v`` simply tests all reachable objects. Usually
  103. these are in the cpu slabs and the partial slabs. Full slabs are not
  104. tracked by SLUB in a non debug situation.
  105. Getting more performance
  106. ========================
  107. To some degree SLUB's performance is limited by the need to take the
  108. list_lock once in a while to deal with partial slabs. That overhead is
  109. governed by the order of the allocation for each slab. The allocations
  110. can be influenced by kernel parameters:
  111. .. slub_min_objects=x (default 4)
  112. .. slub_min_order=x (default 0)
  113. .. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER))
  114. ``slub_min_objects``
  115. allows to specify how many objects must at least fit into one
  116. slab in order for the allocation order to be acceptable. In
  117. general slub will be able to perform this number of
  118. allocations on a slab without consulting centralized resources
  119. (list_lock) where contention may occur.
  120. ``slub_min_order``
  121. specifies a minimum order of slabs. A similar effect like
  122. ``slub_min_objects``.
  123. ``slub_max_order``
  124. specified the order at which ``slub_min_objects`` should no
  125. longer be checked. This is useful to avoid SLUB trying to
  126. generate super large order pages to fit ``slub_min_objects``
  127. of a slab cache with large object sizes into one high order
  128. page. Setting command line parameter
  129. ``debug_guardpage_minorder=N`` (N > 0), forces setting
  130. ``slub_max_order`` to 0, what cause minimum possible order of
  131. slabs allocation.
  132. SLUB Debug output
  133. =================
  134. Here is a sample of slub debug output::
  135. ====================================================================
  136. BUG kmalloc-8: Right Redzone overwritten
  137. --------------------------------------------------------------------
  138. INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc
  139. INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58
  140. INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58
  141. INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554
  142. Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
  143. Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005
  144. Redzone (0xc90f6d28): 00 cc cc cc .
  145. Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
  146. [<c010523d>] dump_trace+0x63/0x1eb
  147. [<c01053df>] show_trace_log_lvl+0x1a/0x2f
  148. [<c010601d>] show_trace+0x12/0x14
  149. [<c0106035>] dump_stack+0x16/0x18
  150. [<c017e0fa>] object_err+0x143/0x14b
  151. [<c017e2cc>] check_object+0x66/0x234
  152. [<c017eb43>] __slab_free+0x239/0x384
  153. [<c017f446>] kfree+0xa6/0xc6
  154. [<c02e2335>] get_modalias+0xb9/0xf5
  155. [<c02e23b7>] dmi_dev_uevent+0x27/0x3c
  156. [<c027866a>] dev_uevent+0x1ad/0x1da
  157. [<c0205024>] kobject_uevent_env+0x20a/0x45b
  158. [<c020527f>] kobject_uevent+0xa/0xf
  159. [<c02779f1>] store_uevent+0x4f/0x58
  160. [<c027758e>] dev_attr_store+0x29/0x2f
  161. [<c01bec4f>] sysfs_write_file+0x16e/0x19c
  162. [<c0183ba7>] vfs_write+0xd1/0x15a
  163. [<c01841d7>] sys_write+0x3d/0x72
  164. [<c0104112>] sysenter_past_esp+0x5f/0x99
  165. [<b7f7b410>] 0xb7f7b410
  166. =======================
  167. FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc
  168. If SLUB encounters a corrupted object (full detection requires the kernel
  169. to be booted with slub_debug) then the following output will be dumped
  170. into the syslog:
  171. 1. Description of the problem encountered
  172. This will be a message in the system log starting with::
  173. ===============================================
  174. BUG <slab cache affected>: <What went wrong>
  175. -----------------------------------------------
  176. INFO: <corruption start>-<corruption_end> <more info>
  177. INFO: Slab <address> <slab information>
  178. INFO: Object <address> <object information>
  179. INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by
  180. cpu> pid=<pid of the process>
  181. INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu>
  182. pid=<pid of the process>
  183. (Object allocation / free information is only available if SLAB_STORE_USER is
  184. set for the slab. slub_debug sets that option)
  185. 2. The object contents if an object was involved.
  186. Various types of lines can follow the BUG SLUB line:
  187. Bytes b4 <address> : <bytes>
  188. Shows a few bytes before the object where the problem was detected.
  189. Can be useful if the corruption does not stop with the start of the
  190. object.
  191. Object <address> : <bytes>
  192. The bytes of the object. If the object is inactive then the bytes
  193. typically contain poison values. Any non-poison value shows a
  194. corruption by a write after free.
  195. Redzone <address> : <bytes>
  196. The Redzone following the object. The Redzone is used to detect
  197. writes after the object. All bytes should always have the same
  198. value. If there is any deviation then it is due to a write after
  199. the object boundary.
  200. (Redzone information is only available if SLAB_RED_ZONE is set.
  201. slub_debug sets that option)
  202. Padding <address> : <bytes>
  203. Unused data to fill up the space in order to get the next object
  204. properly aligned. In the debug case we make sure that there are
  205. at least 4 bytes of padding. This allows the detection of writes
  206. before the object.
  207. 3. A stackdump
  208. The stackdump describes the location where the error was detected. The cause
  209. of the corruption is may be more likely found by looking at the function that
  210. allocated or freed the object.
  211. 4. Report on how the problem was dealt with in order to ensure the continued
  212. operation of the system.
  213. These are messages in the system log beginning with::
  214. FIX <slab cache affected>: <corrective action taken>
  215. In the above sample SLUB found that the Redzone of an active object has
  216. been overwritten. Here a string of 8 characters was written into a slab that
  217. has the length of 8 characters. However, a 8 character string needs a
  218. terminating 0. That zero has overwritten the first byte of the Redzone field.
  219. After reporting the details of the issue encountered the FIX SLUB message
  220. tells us that SLUB has restored the Redzone to its proper value and then
  221. system operations continue.
  222. Emergency operations
  223. ====================
  224. Minimal debugging (sanity checks alone) can be enabled by booting with::
  225. slub_debug=F
  226. This will be generally be enough to enable the resiliency features of slub
  227. which will keep the system running even if a bad kernel component will
  228. keep corrupting objects. This may be important for production systems.
  229. Performance will be impacted by the sanity checks and there will be a
  230. continual stream of error messages to the syslog but no additional memory
  231. will be used (unlike full debugging).
  232. No guarantees. The kernel component still needs to be fixed. Performance
  233. may be optimized further by locating the slab that experiences corruption
  234. and enabling debugging only for that cache
  235. I.e.::
  236. slub_debug=F,dentry
  237. If the corruption occurs by writing after the end of the object then it
  238. may be advisable to enable a Redzone to avoid corrupting the beginning
  239. of other objects::
  240. slub_debug=FZ,dentry
  241. Extended slabinfo mode and plotting
  242. ===================================
  243. The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes:
  244. - Slabcache Totals
  245. - Slabs sorted by size (up to -N <num> slabs, default 1)
  246. - Slabs sorted by loss (up to -N <num> slabs, default 1)
  247. Additionally, in this mode ``slabinfo`` does not dynamically scale
  248. sizes (G/M/K) and reports everything in bytes (this functionality is
  249. also available to other slabinfo modes via '-B' option) which makes
  250. reporting more precise and accurate. Moreover, in some sense the `-X'
  251. mode also simplifies the analysis of slabs' behaviour, because its
  252. output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it
  253. pushes the analysis from looking through the numbers (tons of numbers)
  254. to something easier -- visual analysis.
  255. To generate plots:
  256. a) collect slabinfo extended records, for example::
  257. while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done
  258. b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script::
  259. slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN]
  260. The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records
  261. and generates 3 png files (and 3 pre-processing cache files) per STATS
  262. file:
  263. - Slabcache Totals: FOO_STATS-totals.png
  264. - Slabs sorted by size: FOO_STATS-slabs-by-size.png
  265. - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png
  266. Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you
  267. need to compare slabs' behaviour "prior to" and "after" some code
  268. modification. To help you out there, ``slabinfo-gnuplot.sh`` script
  269. can 'merge' the `Slabcache Totals` sections from different
  270. measurements. To visually compare N plots:
  271. a) Collect as many STATS1, STATS2, .. STATSN files as you need::
  272. while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done
  273. b) Pre-process those STATS files::
  274. slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN
  275. c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the
  276. generated pre-processed \*-totals::
  277. slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals
  278. This will produce a single plot (png file).
  279. Plots, expectedly, can be large so some fluctuations or small spikes
  280. can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two
  281. options to 'zoom-in'/'zoom-out':
  282. a) ``-s %d,%d`` -- overwrites the default image width and height
  283. b) ``-r %d,%d`` -- specifies a range of samples to use (for example,
  284. in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r
  285. 40,60`` range will plot only samples collected between 40th and
  286. 60th seconds).
  287. Christoph Lameter, May 30, 2007
  288. Sergey Senozhatsky, October 23, 2015