md-cluster.rst 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385
  1. ==========
  2. MD Cluster
  3. ==========
  4. The cluster MD is a shared-device RAID for a cluster, it supports
  5. two levels: raid1 and raid10 (limited support).
  6. 1. On-disk format
  7. =================
  8. Separate write-intent-bitmaps are used for each cluster node.
  9. The bitmaps record all writes that may have been started on that node,
  10. and may not yet have finished. The on-disk layout is::
  11. 0 4k 8k 12k
  12. -------------------------------------------------------------------
  13. | idle | md super | bm super [0] + bits |
  14. | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
  15. | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
  16. | bm bits [3, contd] | | |
  17. During "normal" functioning we assume the filesystem ensures that only
  18. one node writes to any given block at a time, so a write request will
  19. - set the appropriate bit (if not already set)
  20. - commit the write to all mirrors
  21. - schedule the bit to be cleared after a timeout.
  22. Reads are just handled normally. It is up to the filesystem to ensure
  23. one node doesn't read from a location where another node (or the same
  24. node) is writing.
  25. 2. DLM Locks for management
  26. ===========================
  27. There are three groups of locks for managing the device:
  28. 2.1 Bitmap lock resource (bm_lockres)
  29. -------------------------------------
  30. The bm_lockres protects individual node bitmaps. They are named in
  31. the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
  32. node joins the cluster, it acquires the lock in PW mode and it stays
  33. so during the lifetime the node is part of the cluster. The lock
  34. resource number is based on the slot number returned by the DLM
  35. subsystem. Since DLM starts node count from one and bitmap slots
  36. start from zero, one is subtracted from the DLM slot number to arrive
  37. at the bitmap slot number.
  38. The LVB of the bitmap lock for a particular node records the range
  39. of sectors that are being re-synced by that node. No other
  40. node may write to those sectors. This is used when a new nodes
  41. joins the cluster.
  42. 2.2 Message passing locks
  43. -------------------------
  44. Each node has to communicate with other nodes when starting or ending
  45. resync, and for metadata superblock updates. This communication is
  46. managed through three locks: "token", "message", and "ack", together
  47. with the Lock Value Block (LVB) of one of the "message" lock.
  48. 2.3 new-device management
  49. -------------------------
  50. A single lock: "no-new-dev" is used to co-ordinate the addition of
  51. new devices - this must be synchronized across the array.
  52. Normally all nodes hold a concurrent-read lock on this device.
  53. 3. Communication
  54. ================
  55. Messages can be broadcast to all nodes, and the sender waits for all
  56. other nodes to acknowledge the message before proceeding. Only one
  57. message can be processed at a time.
  58. 3.1 Message Types
  59. -----------------
  60. There are six types of messages which are passed:
  61. 3.1.1 METADATA_UPDATED
  62. ^^^^^^^^^^^^^^^^^^^^^^
  63. informs other nodes that the metadata has
  64. been updated, and the node must re-read the md superblock. This is
  65. performed synchronously. It is primarily used to signal device
  66. failure.
  67. 3.1.2 RESYNCING
  68. ^^^^^^^^^^^^^^^
  69. informs other nodes that a resync is initiated or
  70. ended so that each node may suspend or resume the region. Each
  71. RESYNCING message identifies a range of the devices that the
  72. sending node is about to resync. This overrides any previous
  73. notification from that node: only one ranged can be resynced at a
  74. time per-node.
  75. 3.1.3 NEWDISK
  76. ^^^^^^^^^^^^^
  77. informs other nodes that a device is being added to
  78. the array. Message contains an identifier for that device. See
  79. below for further details.
  80. 3.1.4 REMOVE
  81. ^^^^^^^^^^^^
  82. A failed or spare device is being removed from the
  83. array. The slot-number of the device is included in the message.
  84. 3.1.5 RE_ADD:
  85. A failed device is being re-activated - the assumption
  86. is that it has been determined to be working again.
  87. 3.1.6 BITMAP_NEEDS_SYNC:
  88. If a node is stopped locally but the bitmap
  89. isn't clean, then another node is informed to take the ownership of
  90. resync.
  91. 3.2 Communication mechanism
  92. ---------------------------
  93. The DLM LVB is used to communicate within nodes of the cluster. There
  94. are three resources used for the purpose:
  95. 3.2.1 token
  96. ^^^^^^^^^^^
  97. The resource which protects the entire communication
  98. system. The node having the token resource is allowed to
  99. communicate.
  100. 3.2.2 message
  101. ^^^^^^^^^^^^^
  102. The lock resource which carries the data to communicate.
  103. 3.2.3 ack
  104. ^^^^^^^^^
  105. The resource, acquiring which means the message has been
  106. acknowledged by all nodes in the cluster. The BAST of the resource
  107. is used to inform the receiving node that a node wants to
  108. communicate.
  109. The algorithm is:
  110. 1. receive status - all nodes have concurrent-reader lock on "ack"::
  111. sender receiver receiver
  112. "ack":CR "ack":CR "ack":CR
  113. 2. sender get EX on "token",
  114. sender get EX on "message"::
  115. sender receiver receiver
  116. "token":EX "ack":CR "ack":CR
  117. "message":EX
  118. "ack":CR
  119. Sender checks that it still needs to send a message. Messages
  120. received or other events that happened while waiting for the
  121. "token" may have made this message inappropriate or redundant.
  122. 3. sender writes LVB
  123. sender down-convert "message" from EX to CW
  124. sender try to get EX of "ack"
  125. ::
  126. [ wait until all receivers have *processed* the "message" ]
  127. [ triggered by bast of "ack" ]
  128. receiver get CR on "message"
  129. receiver read LVB
  130. receiver processes the message
  131. [ wait finish ]
  132. receiver releases "ack"
  133. receiver tries to get PR on "message"
  134. sender receiver receiver
  135. "token":EX "message":CR "message":CR
  136. "message":CW
  137. "ack":EX
  138. 4. triggered by grant of EX on "ack" (indicating all receivers
  139. have processed message)
  140. sender down-converts "ack" from EX to CR
  141. sender releases "message"
  142. sender releases "token"
  143. ::
  144. receiver upconvert to PR on "message"
  145. receiver get CR of "ack"
  146. receiver release "message"
  147. sender receiver receiver
  148. "ack":CR "ack":CR "ack":CR
  149. 4. Handling Failures
  150. ====================
  151. 4.1 Node Failure
  152. ----------------
  153. When a node fails, the DLM informs the cluster with the slot
  154. number. The node starts a cluster recovery thread. The cluster
  155. recovery thread:
  156. - acquires the bitmap<number> lock of the failed node
  157. - opens the bitmap
  158. - reads the bitmap of the failed node
  159. - copies the set bitmap to local node
  160. - cleans the bitmap of the failed node
  161. - releases bitmap<number> lock of the failed node
  162. - initiates resync of the bitmap on the current node
  163. md_check_recovery is invoked within recover_bitmaps,
  164. then md_check_recovery -> metadata_update_start/finish,
  165. it will lock the communication by lock_comm.
  166. Which means when one node is resyncing it blocks all
  167. other nodes from writing anywhere on the array.
  168. The resync process is the regular md resync. However, in a clustered
  169. environment when a resync is performed, it needs to tell other nodes
  170. of the areas which are suspended. Before a resync starts, the node
  171. send out RESYNCING with the (lo,hi) range of the area which needs to
  172. be suspended. Each node maintains a suspend_list, which contains the
  173. list of ranges which are currently suspended. On receiving RESYNCING,
  174. the node adds the range to the suspend_list. Similarly, when the node
  175. performing resync finishes, it sends RESYNCING with an empty range to
  176. other nodes and other nodes remove the corresponding entry from the
  177. suspend_list.
  178. A helper function, ->area_resyncing() can be used to check if a
  179. particular I/O range should be suspended or not.
  180. 4.2 Device Failure
  181. ==================
  182. Device failures are handled and communicated with the metadata update
  183. routine. When a node detects a device failure it does not allow
  184. any further writes to that device until the failure has been
  185. acknowledged by all other nodes.
  186. 5. Adding a new Device
  187. ----------------------
  188. For adding a new device, it is necessary that all nodes "see" the new
  189. device to be added. For this, the following algorithm is used:
  190. 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
  191. ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
  192. 2. Node 1 sends a NEWDISK message with uuid and slot number
  193. 3. Other nodes issue kobject_uevent_env with uuid and slot number
  194. (Steps 4,5 could be a udev rule)
  195. 4. In userspace, the node searches for the disk, perhaps
  196. using blkid -t SUB_UUID=""
  197. 5. Other nodes issue either of the following depending on whether
  198. the disk was found:
  199. ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
  200. disc.number set to slot number)
  201. ioctl(CLUSTERED_DISK_NACK)
  202. 6. Other nodes drop lock on "no-new-devs" (CR) if device is found
  203. 7. Node 1 attempts EX lock on "no-new-dev"
  204. 8. If node 1 gets the lock, it sends METADATA_UPDATED after
  205. unmarking the disk as SpareLocal
  206. 9. If not (get "no-new-dev" lock), it fails the operation and sends
  207. METADATA_UPDATED.
  208. 10. Other nodes get the information whether a disk is added or not
  209. by the following METADATA_UPDATED.
  210. 6. Module interface
  211. ===================
  212. There are 17 call-backs which the md core can make to the cluster
  213. module. Understanding these can give a good overview of the whole
  214. process.
  215. 6.1 join(nodes) and leave()
  216. ---------------------------
  217. These are called when an array is started with a clustered bitmap,
  218. and when the array is stopped. join() ensures the cluster is
  219. available and initializes the various resources.
  220. Only the first 'nodes' nodes in the cluster can use the array.
  221. 6.2 slot_number()
  222. -----------------
  223. Reports the slot number advised by the cluster infrastructure.
  224. Range is from 0 to nodes-1.
  225. 6.3 resync_info_update()
  226. ------------------------
  227. This updates the resync range that is stored in the bitmap lock.
  228. The starting point is updated as the resync progresses. The
  229. end point is always the end of the array.
  230. It does *not* send a RESYNCING message.
  231. 6.4 resync_start(), resync_finish()
  232. -----------------------------------
  233. These are called when resync/recovery/reshape starts or stops.
  234. They update the resyncing range in the bitmap lock and also
  235. send a RESYNCING message. resync_start reports the whole
  236. array as resyncing, resync_finish reports none of it.
  237. resync_finish() also sends a BITMAP_NEEDS_SYNC message which
  238. allows some other node to take over.
  239. 6.5 metadata_update_start(), metadata_update_finish(), metadata_update_cancel()
  240. -------------------------------------------------------------------------------
  241. metadata_update_start is used to get exclusive access to
  242. the metadata. If a change is still needed once that access is
  243. gained, metadata_update_finish() will send a METADATA_UPDATE
  244. message to all other nodes, otherwise metadata_update_cancel()
  245. can be used to release the lock.
  246. 6.6 area_resyncing()
  247. --------------------
  248. This combines two elements of functionality.
  249. Firstly, it will check if any node is currently resyncing
  250. anything in a given range of sectors. If any resync is found,
  251. then the caller will avoid writing or read-balancing in that
  252. range.
  253. Secondly, while node recovery is happening it reports that
  254. all areas are resyncing for READ requests. This avoids races
  255. between the cluster-filesystem and the cluster-RAID handling
  256. a node failure.
  257. 6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
  258. ---------------------------------------------------------------
  259. These are used to manage the new-disk protocol described above.
  260. When a new device is added, add_new_disk_start() is called before
  261. it is bound to the array and, if that succeeds, add_new_disk_finish()
  262. is called the device is fully added.
  263. When a device is added in acknowledgement to a previous
  264. request, or when the device is declared "unavailable",
  265. new_disk_ack() is called.
  266. 6.8 remove_disk()
  267. -----------------
  268. This is called when a spare or failed device is removed from
  269. the array. It causes a REMOVE message to be send to other nodes.
  270. 6.9 gather_bitmaps()
  271. --------------------
  272. This sends a RE_ADD message to all other nodes and then
  273. gathers bitmap information from all bitmaps. This combined
  274. bitmap is then used to recovery the re-added device.
  275. 6.10 lock_all_bitmaps() and unlock_all_bitmaps()
  276. ------------------------------------------------
  277. These are called when change bitmap to none. If a node plans
  278. to clear the cluster raid's bitmap, it need to make sure no other
  279. nodes are using the raid which is achieved by lock all bitmap
  280. locks within the cluster, and also those locks are unlocked
  281. accordingly.
  282. 7. Unsupported features
  283. =======================
  284. There are somethings which are not supported by cluster MD yet.
  285. - change array_sectors.