raid5-cache.rst 5.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111
  1. ================
  2. RAID 4/5/6 cache
  3. ================
  4. Raid 4/5/6 could include an extra disk for data cache besides normal RAID
  5. disks. The role of RAID disks isn't changed with the cache disk. The cache disk
  6. caches data to the RAID disks. The cache can be in write-through (supported
  7. since 4.4) or write-back mode (supported since 4.10). mdadm (supported since
  8. 3.4) has a new option '--write-journal' to create array with cache. Please
  9. refer to mdadm manual for details. By default (RAID array starts), the cache is
  10. in write-through mode. A user can switch it to write-back mode by::
  11. echo "write-back" > /sys/block/md0/md/journal_mode
  12. And switch it back to write-through mode by::
  13. echo "write-through" > /sys/block/md0/md/journal_mode
  14. In both modes, all writes to the array will hit cache disk first. This means
  15. the cache disk must be fast and sustainable.
  16. write-through mode
  17. ==================
  18. This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an unclean
  19. shutdown can cause data in some stripes to not be in consistent state, eg, data
  20. and parity don't match. The reason is that a stripe write involves several RAID
  21. disks and it's possible the writes don't hit all RAID disks yet before the
  22. unclean shutdown. We call an array degraded if it has inconsistent data. MD
  23. tries to resync the array to bring it back to normal state. But before the
  24. resync completes, any system crash will expose the chance of real data
  25. corruption in the RAID array. This problem is called 'write hole'.
  26. The write-through cache will cache all data on cache disk first. After the data
  27. is safe on the cache disk, the data will be flushed onto RAID disks. The
  28. two-step write will guarantee MD can recover correct data after unclean
  29. shutdown even the array is degraded. Thus the cache can close the 'write hole'.
  30. In write-through mode, MD reports IO completion to upper layer (usually
  31. filesystems) after the data is safe on RAID disks, so cache disk failure
  32. doesn't cause data loss. Of course cache disk failure means the array is
  33. exposed to 'write hole' again.
  34. In write-through mode, the cache disk isn't required to be big. Several
  35. hundreds megabytes are enough.
  36. write-back mode
  37. ===============
  38. write-back mode fixes the 'write hole' issue too, since all write data is
  39. cached on cache disk. But the main goal of 'write-back' cache is to speed up
  40. write. If a write crosses all RAID disks of a stripe, we call it full-stripe
  41. write. For non-full-stripe writes, MD must read old data before the new parity
  42. can be calculated. These synchronous reads hurt write throughput. Some writes
  43. which are sequential but not dispatched in the same time will suffer from this
  44. overhead too. Write-back cache will aggregate the data and flush the data to
  45. RAID disks only after the data becomes a full stripe write. This will
  46. completely avoid the overhead, so it's very helpful for some workloads. A
  47. typical workload which does sequential write followed by fsync is an example.
  48. In write-back mode, MD reports IO completion to upper layer (usually
  49. filesystems) right after the data hits cache disk. The data is flushed to raid
  50. disks later after specific conditions met. So cache disk failure will cause
  51. data loss.
  52. In write-back mode, MD also caches data in memory. The memory cache includes
  53. the same data stored on cache disk, so a power loss doesn't cause data loss.
  54. The memory cache size has performance impact for the array. It's recommended
  55. the size is big. A user can configure the size by::
  56. echo "2048" > /sys/block/md0/md/stripe_cache_size
  57. Too small cache disk will make the write aggregation less efficient in this
  58. mode depending on the workloads. It's recommended to use a cache disk with at
  59. least several gigabytes size in write-back mode.
  60. The implementation
  61. ==================
  62. The write-through and write-back cache use the same disk format. The cache disk
  63. is organized as a simple write log. The log consists of 'meta data' and 'data'
  64. pairs. The meta data describes the data. It also includes checksum and sequence
  65. ID for recovery identification. Data can be IO data and parity data. Data is
  66. checksumed too. The checksum is stored in the meta data ahead of the data. The
  67. checksum is an optimization because MD can write meta and data freely without
  68. worry about the order. MD superblock has a field pointed to the valid meta data
  69. of log head.
  70. The log implementation is pretty straightforward. The difficult part is the
  71. order in which MD writes data to cache disk and RAID disks. Specifically, in
  72. write-through mode, MD calculates parity for IO data, writes both IO data and
  73. parity to the log, writes the data and parity to RAID disks after the data and
  74. parity is settled down in log and finally the IO is finished. Read just reads
  75. from raid disks as usual.
  76. In write-back mode, MD writes IO data to the log and reports IO completion. The
  77. data is also fully cached in memory at that time, which means read must query
  78. memory cache. If some conditions are met, MD will flush the data to RAID disks.
  79. MD will calculate parity for the data and write parity into the log. After this
  80. is finished, MD will write both data and parity into RAID disks, then MD can
  81. release the memory cache. The flush conditions could be stripe becomes a full
  82. stripe write, free cache disk space is low or free in-kernel memory cache space
  83. is low.
  84. After an unclean shutdown, MD does recovery. MD reads all meta data and data
  85. from the log. The sequence ID and checksum will help us detect corrupted meta
  86. data and data. If MD finds a stripe with data and valid parities (1 parity for
  87. raid4/5 and 2 for raid6), MD will write the data and parities to RAID disks. If
  88. parities are incompleted, they are discarded. If part of data is corrupted,
  89. they are discarded too. MD then loads valid data and writes them to RAID disks
  90. in normal way.