path-lookup.txt 18 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382
  1. Path walking and name lookup locking
  2. ====================================
  3. Path resolution is the finding a dentry corresponding to a path name string, by
  4. performing a path walk. Typically, for every open(), stat() etc., the path name
  5. will be resolved. Paths are resolved by walking the namespace tree, starting
  6. with the first component of the pathname (eg. root or cwd) with a known dentry,
  7. then finding the child of that dentry, which is named the next component in the
  8. path string. Then repeating the lookup from the child dentry and finding its
  9. child with the next element, and so on.
  10. Since it is a frequent operation for workloads like multiuser environments and
  11. web servers, it is important to optimize this code.
  12. Path walking synchronisation history:
  13. Prior to 2.5.10, dcache_lock was acquired in d_lookup (dcache hash lookup) and
  14. thus in every component during path look-up. Since 2.5.10 onwards, fast-walk
  15. algorithm changed this by holding the dcache_lock at the beginning and walking
  16. as many cached path component dentries as possible. This significantly
  17. decreases the number of acquisition of dcache_lock. However it also increases
  18. the lock hold time significantly and affects performance in large SMP machines.
  19. Since 2.5.62 kernel, dcache has been using a new locking model that uses RCU to
  20. make dcache look-up lock-free.
  21. All the above algorithms required taking a lock and reference count on the
  22. dentry that was looked up, so that may be used as the basis for walking the
  23. next path element. This is inefficient and unscalable. It is inefficient
  24. because of the locks and atomic operations required for every dentry element
  25. slows things down. It is not scalable because many parallel applications that
  26. are path-walk intensive tend to do path lookups starting from a common dentry
  27. (usually, the root "/" or current working directory). So contention on these
  28. common path elements causes lock and cacheline queueing.
  29. Since 2.6.38, RCU is used to make a significant part of the entire path walk
  30. (including dcache look-up) completely "store-free" (so, no locks, atomics, or
  31. even stores into cachelines of common dentries). This is known as "rcu-walk"
  32. path walking.
  33. Path walking overview
  34. =====================
  35. A name string specifies a start (root directory, cwd, fd-relative) and a
  36. sequence of elements (directory entry names), which together refer to a path in
  37. the namespace. A path is represented as a (dentry, vfsmount) tuple. The name
  38. elements are sub-strings, separated by '/'.
  39. Name lookups will want to find a particular path that a name string refers to
  40. (usually the final element, or parent of final element). This is done by taking
  41. the path given by the name's starting point (which we know in advance -- eg.
  42. current->fs->cwd or current->fs->root) as the first parent of the lookup. Then
  43. iteratively for each subsequent name element, look up the child of the current
  44. parent with the given name and if it is not the desired entry, make it the
  45. parent for the next lookup.
  46. A parent, of course, must be a directory, and we must have appropriate
  47. permissions on the parent inode to be able to walk into it.
  48. Turning the child into a parent for the next lookup requires more checks and
  49. procedures. Symlinks essentially substitute the symlink name for the target
  50. name in the name string, and require some recursive path walking. Mount points
  51. must be followed into (thus changing the vfsmount that subsequent path elements
  52. refer to), switching from the mount point path to the root of the particular
  53. mounted vfsmount. These behaviours are variously modified depending on the
  54. exact path walking flags.
  55. Path walking then must, broadly, do several particular things:
  56. - find the start point of the walk;
  57. - perform permissions and validity checks on inodes;
  58. - perform dcache hash name lookups on (parent, name element) tuples;
  59. - traverse mount points;
  60. - traverse symlinks;
  61. - lookup and create missing parts of the path on demand.
  62. Safe store-free look-up of dcache hash table
  63. ============================================
  64. Dcache name lookup
  65. ------------------
  66. In order to lookup a dcache (parent, name) tuple, we take a hash on the tuple
  67. and use that to select a bucket in the dcache-hash table. The list of entries
  68. in that bucket is then walked, and we do a full comparison of each entry
  69. against our (parent, name) tuple.
  70. The hash lists are RCU protected, so list walking is not serialised with
  71. concurrent updates (insertion, deletion from the hash). This is a standard RCU
  72. list application with the exception of renames, which will be covered below.
  73. Parent and name members of a dentry, as well as its membership in the dcache
  74. hash, and its inode are protected by the per-dentry d_lock spinlock. A
  75. reference is taken on the dentry (while the fields are verified under d_lock),
  76. and this stabilises its d_inode pointer and actual inode. This gives a stable
  77. point to perform the next step of our path walk against.
  78. These members are also protected by d_seq seqlock, although this offers
  79. read-only protection and no durability of results, so care must be taken when
  80. using d_seq for synchronisation (see seqcount based lookups, below).
  81. Renames
  82. -------
  83. Back to the rename case. In usual RCU protected lists, the only operations that
  84. will happen to an object is insertion, and then eventually removal from the
  85. list. The object will not be reused until an RCU grace period is complete.
  86. This ensures the RCU list traversal primitives can run over the object without
  87. problems (see RCU documentation for how this works).
  88. However when a dentry is renamed, its hash value can change, requiring it to be
  89. moved to a new hash list. Allocating and inserting a new alias would be
  90. expensive and also problematic for directory dentries. Latency would be far to
  91. high to wait for a grace period after removing the dentry and before inserting
  92. it in the new hash bucket. So what is done is to insert the dentry into the
  93. new list immediately.
  94. However, when the dentry's list pointers are updated to point to objects in the
  95. new list before waiting for a grace period, this can result in a concurrent RCU
  96. lookup of the old list veering off into the new (incorrect) list and missing
  97. the remaining dentries on the list.
  98. There is no fundamental problem with walking down the wrong list, because the
  99. dentry comparisons will never match. However it is fatal to miss a matching
  100. dentry. So a seqlock is used to detect when a rename has occurred, and so the
  101. lookup can be retried.
  102. 1 2 3
  103. +---+ +---+ +---+
  104. hlist-->| N-+->| N-+->| N-+->
  105. head <--+-P |<-+-P |<-+-P |
  106. +---+ +---+ +---+
  107. Rename of dentry 2 may require it deleted from the above list, and inserted
  108. into a new list. Deleting 2 gives the following list.
  109. 1 3
  110. +---+ +---+ (don't worry, the longer pointers do not
  111. hlist-->| N-+-------->| N-+-> impose a measurable performance overhead
  112. head <--+-P |<--------+-P | on modern CPUs)
  113. +---+ +---+
  114. ^ 2 ^
  115. | +---+ |
  116. | | N-+----+
  117. +----+-P |
  118. +---+
  119. This is a standard RCU-list deletion, which leaves the deleted object's
  120. pointers intact, so a concurrent list walker that is currently looking at
  121. object 2 will correctly continue to object 3 when it is time to traverse the
  122. next object.
  123. However, when inserting object 2 onto a new list, we end up with this:
  124. 1 3
  125. +---+ +---+
  126. hlist-->| N-+-------->| N-+->
  127. head <--+-P |<--------+-P |
  128. +---+ +---+
  129. 2
  130. +---+
  131. | N-+---->
  132. <----+-P |
  133. +---+
  134. Because we didn't wait for a grace period, there may be a concurrent lookup
  135. still at 2. Now when it follows 2's 'next' pointer, it will walk off into
  136. another list without ever having checked object 3.
  137. A related, but distinctly different, issue is that of rename atomicity versus
  138. lookup operations. If a file is renamed from 'A' to 'B', a lookup must only
  139. find either 'A' or 'B'. So if a lookup of 'A' returns NULL, a subsequent lookup
  140. of 'B' must succeed (note the reverse is not true).
  141. Between deleting the dentry from the old hash list, and inserting it on the new
  142. hash list, a lookup may find neither 'A' nor 'B' matching the dentry. The same
  143. rename seqlock is also used to cover this race in much the same way, by
  144. retrying a negative lookup result if a rename was in progress.
  145. Seqcount based lookups
  146. ----------------------
  147. In refcount based dcache lookups, d_lock is used to serialise access to
  148. the dentry, stabilising it while comparing its name and parent and then
  149. taking a reference count (the reference count then gives a stable place to
  150. start the next part of the path walk from).
  151. As explained above, we would like to do path walking without taking locks or
  152. reference counts on intermediate dentries along the path. To do this, a per
  153. dentry seqlock (d_seq) is used to take a "coherent snapshot" of what the dentry
  154. looks like (its name, parent, and inode). That snapshot is then used to start
  155. the next part of the path walk. When loading the coherent snapshot under d_seq,
  156. care must be taken to load the members up-front, and use those pointers rather
  157. than reloading from the dentry later on (otherwise we'd have interesting things
  158. like d_inode going NULL underneath us, if the name was unlinked).
  159. Also important is to avoid performing any destructive operations (pretty much:
  160. no non-atomic stores to shared data), and to recheck the seqcount when we are
  161. "done" with the operation. Retry or abort if the seqcount does not match.
  162. Avoiding destructive or changing operations means we can easily unwind from
  163. failure.
  164. What this means is that a caller, provided they are holding RCU lock to
  165. protect the dentry object from disappearing, can perform a seqcount based
  166. lookup which does not increment the refcount on the dentry or write to
  167. it in any way. This returned dentry can be used for subsequent operations,
  168. provided that d_seq is rechecked after that operation is complete.
  169. Inodes are also rcu freed, so the seqcount lookup dentry's inode may also be
  170. queried for permissions.
  171. With this two parts of the puzzle, we can do path lookups without taking
  172. locks or refcounts on dentry elements.
  173. RCU-walk path walking design
  174. ============================
  175. Path walking code now has two distinct modes, ref-walk and rcu-walk. ref-walk
  176. is the traditional[*] way of performing dcache lookups using d_lock to
  177. serialise concurrent modifications to the dentry and take a reference count on
  178. it. ref-walk is simple and obvious, and may sleep, take locks, etc while path
  179. walking is operating on each dentry. rcu-walk uses seqcount based dentry
  180. lookups, and can perform lookup of intermediate elements without any stores to
  181. shared data in the dentry or inode. rcu-walk can not be applied to all cases,
  182. eg. if the filesystem must sleep or perform non trivial operations, rcu-walk
  183. must be switched to ref-walk mode.
  184. [*] RCU is still used for the dentry hash lookup in ref-walk, but not the full
  185. path walk.
  186. Where ref-walk uses a stable, refcounted ``parent'' to walk the remaining
  187. path string, rcu-walk uses a d_seq protected snapshot. When looking up a
  188. child of this parent snapshot, we open d_seq critical section on the child
  189. before closing d_seq critical section on the parent. This gives an interlocking
  190. ladder of snapshots to walk down.
  191. proc 101
  192. /----------------\
  193. / comm: "vi" \
  194. / fs.root: dentry0 \
  195. \ fs.cwd: dentry2 /
  196. \ /
  197. \----------------/
  198. So when vi wants to open("/home/npiggin/test.c", O_RDWR), then it will
  199. start from current->fs->root, which is a pinned dentry. Alternatively,
  200. "./test.c" would start from cwd; both names refer to the same path in
  201. the context of proc101.
  202. dentry 0
  203. +---------------------+ rcu-walk begins here, we note d_seq, check the
  204. | name: "/" | inode's permission, and then look up the next
  205. | inode: 10 | path element which is "home"...
  206. | children:"home", ...|
  207. +---------------------+
  208. |
  209. dentry 1 V
  210. +---------------------+ ... which brings us here. We find dentry1 via
  211. | name: "home" | hash lookup, then note d_seq and compare name
  212. | inode: 678 | string and parent pointer. When we have a match,
  213. | children:"npiggin" | we now recheck the d_seq of dentry0. Then we
  214. +---------------------+ check inode and look up the next element.
  215. |
  216. dentry2 V
  217. +---------------------+ Note: if dentry0 is now modified, lookup is
  218. | name: "npiggin" | not necessarily invalid, so we need only keep a
  219. | inode: 543 | parent for d_seq verification, and grandparents
  220. | children:"a.c", ... | can be forgotten.
  221. +---------------------+
  222. |
  223. dentry3 V
  224. +---------------------+ At this point we have our destination dentry.
  225. | name: "a.c" | We now take its d_lock, verify d_seq of this
  226. | inode: 14221 | dentry. If that checks out, we can increment
  227. | children:NULL | its refcount because we're holding d_lock.
  228. +---------------------+
  229. Taking a refcount on a dentry from rcu-walk mode, by taking its d_lock,
  230. re-checking its d_seq, and then incrementing its refcount is called
  231. "dropping rcu" or dropping from rcu-walk into ref-walk mode.
  232. It is, in some sense, a bit of a house of cards. If the seqcount check of the
  233. parent snapshot fails, the house comes down, because we had closed the d_seq
  234. section on the grandparent, so we have nothing left to stand on. In that case,
  235. the path walk must be fully restarted (which we do in ref-walk mode, to avoid
  236. live locks). It is costly to have a full restart, but fortunately they are
  237. quite rare.
  238. When we reach a point where sleeping is required, or a filesystem callout
  239. requires ref-walk, then instead of restarting the walk, we attempt to drop rcu
  240. at the last known good dentry we have. Avoiding a full restart in ref-walk in
  241. these cases is fundamental for performance and scalability because blocking
  242. operations such as creates and unlinks are not uncommon.
  243. The detailed design for rcu-walk is like this:
  244. * LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
  245. * Take the RCU lock for the entire path walk, starting with the acquiring
  246. of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
  247. not required for dentry persistence.
  248. * synchronize_rcu is called when unregistering a filesystem, so we can
  249. access d_ops and i_ops during rcu-walk.
  250. * Similarly take the vfsmount lock for the entire path walk. So now mnt
  251. refcounts are not required for persistence. Also we are free to perform mount
  252. lookups, and to assume dentry mount points and mount roots are stable up and
  253. down the path.
  254. * Have a per-dentry seqlock to protect the dentry name, parent, and inode,
  255. so we can load this tuple atomically, and also check whether any of its
  256. members have changed.
  257. * Dentry lookups (based on parent, candidate string tuple) recheck the parent
  258. sequence after the child is found in case anything changed in the parent
  259. during the path walk.
  260. * inode is also RCU protected so we can load d_inode and use the inode for
  261. limited things.
  262. * i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
  263. * i_op can be loaded.
  264. * When the destination dentry is reached, drop rcu there (ie. take d_lock,
  265. verify d_seq, increment refcount).
  266. * If seqlock verification fails anywhere along the path, do a full restart
  267. of the path lookup in ref-walk mode. -ECHILD tends to be used (for want of
  268. a better errno) to signal an rcu-walk failure.
  269. The cases where rcu-walk cannot continue are:
  270. * NULL dentry (ie. any uncached path element)
  271. * Following links
  272. It may be possible eventually to make following links rcu-walk aware.
  273. Uncached path elements will always require dropping to ref-walk mode, at the
  274. very least because i_mutex needs to be grabbed, and objects allocated.
  275. Final note:
  276. "store-free" path walking is not strictly store free. We take vfsmount lock
  277. and refcounts (both of which can be made per-cpu), and we also store to the
  278. stack (which is essentially CPU-local), and we also have to take locks and
  279. refcount on final dentry.
  280. The point is that shared data, where practically possible, is not locked
  281. or stored into. The result is massive improvements in performance and
  282. scalability of path resolution.
  283. Interesting statistics
  284. ======================
  285. The following table gives rcu lookup statistics for a few simple workloads
  286. (2s12c24t Westmere, debian non-graphical system). Ungraceful are attempts to
  287. drop rcu that fail due to d_seq failure and requiring the entire path lookup
  288. again. Other cases are successful rcu-drops that are required before the final
  289. element, nodentry for missing dentry, revalidate for filesystem revalidate
  290. routine requiring rcu drop, permission for permission check requiring drop,
  291. and link for symlink traversal requiring drop.
  292. rcu-lookups restart nodentry link revalidate permission
  293. bootup 47121 0 4624 1010 10283 7852
  294. dbench 25386793 0 6778659(26.7%) 55 549 1156
  295. kbuild 2696672 10 64442(2.3%) 108764(4.0%) 1 1590
  296. git diff 39605 0 28 2 0 106
  297. vfstest 24185492 4945 708725(2.9%) 1076136(4.4%) 0 2651
  298. What this shows is that failed rcu-walk lookups, ie. ones that are restarted
  299. entirely with ref-walk, are quite rare. Even the "vfstest" case which
  300. specifically has concurrent renames/mkdir/rmdir/ creat/unlink/etc to exercise
  301. such races is not showing a huge amount of restarts.
  302. Dropping from rcu-walk to ref-walk mean that we have encountered a dentry where
  303. the reference count needs to be taken for some reason. This is either because
  304. we have reached the target of the path walk, or because we have encountered a
  305. condition that can't be resolved in rcu-walk mode. Ideally, we drop rcu-walk
  306. only when we have reached the target dentry, so the other statistics show where
  307. this does not happen.
  308. Note that a graceful drop from rcu-walk mode due to something such as the
  309. dentry not existing (which can be common) is not necessarily a failure of
  310. rcu-walk scheme, because some elements of the path may have been walked in
  311. rcu-walk mode. The further we get from common path elements (such as cwd or
  312. root), the less contended the dentry is likely to be. The closer we are to
  313. common path elements, the more likely they will exist in dentry cache.
  314. Papers and other documentation on dcache locking
  315. ================================================
  316. 1. Scaling dcache with RCU (https://linuxjournal.com/article.php?sid=7124).
  317. 2. http://lse.sourceforge.net/locking/dcache/dcache.html
  318. 3. path-lookup.md in this directory.