data-integrity.rst 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291
  1. ==============
  2. Data Integrity
  3. ==============
  4. 1. Introduction
  5. ===============
  6. Modern filesystems feature checksumming of data and metadata to
  7. protect against data corruption. However, the detection of the
  8. corruption is done at read time which could potentially be months
  9. after the data was written. At that point the original data that the
  10. application tried to write is most likely lost.
  11. The solution is to ensure that the disk is actually storing what the
  12. application meant it to. Recent additions to both the SCSI family
  13. protocols (SBC Data Integrity Field, SCC protection proposal) as well
  14. as SATA/T13 (External Path Protection) try to remedy this by adding
  15. support for appending integrity metadata to an I/O. The integrity
  16. metadata (or protection information in SCSI terminology) includes a
  17. checksum for each sector as well as an incrementing counter that
  18. ensures the individual sectors are written in the right order. And
  19. for some protection schemes also that the I/O is written to the right
  20. place on disk.
  21. Current storage controllers and devices implement various protective
  22. measures, for instance checksumming and scrubbing. But these
  23. technologies are working in their own isolated domains or at best
  24. between adjacent nodes in the I/O path. The interesting thing about
  25. DIF and the other integrity extensions is that the protection format
  26. is well defined and every node in the I/O path can verify the
  27. integrity of the I/O and reject it if corruption is detected. This
  28. allows not only corruption prevention but also isolation of the point
  29. of failure.
  30. 2. The Data Integrity Extensions
  31. ================================
  32. As written, the protocol extensions only protect the path between
  33. controller and storage device. However, many controllers actually
  34. allow the operating system to interact with the integrity metadata
  35. (IMD). We have been working with several FC/SAS HBA vendors to enable
  36. the protection information to be transferred to and from their
  37. controllers.
  38. The SCSI Data Integrity Field works by appending 8 bytes of protection
  39. information to each sector. The data + integrity metadata is stored
  40. in 520 byte sectors on disk. Data + IMD are interleaved when
  41. transferred between the controller and target. The T13 proposal is
  42. similar.
  43. Because it is highly inconvenient for operating systems to deal with
  44. 520 (and 4104) byte sectors, we approached several HBA vendors and
  45. encouraged them to allow separation of the data and integrity metadata
  46. scatter-gather lists.
  47. The controller will interleave the buffers on write and split them on
  48. read. This means that Linux can DMA the data buffers to and from
  49. host memory without changes to the page cache.
  50. Also, the 16-bit CRC checksum mandated by both the SCSI and SATA specs
  51. is somewhat heavy to compute in software. Benchmarks found that
  52. calculating this checksum had a significant impact on system
  53. performance for a number of workloads. Some controllers allow a
  54. lighter-weight checksum to be used when interfacing with the operating
  55. system. Emulex, for instance, supports the TCP/IP checksum instead.
  56. The IP checksum received from the OS is converted to the 16-bit CRC
  57. when writing and vice versa. This allows the integrity metadata to be
  58. generated by Linux or the application at very low cost (comparable to
  59. software RAID5).
  60. The IP checksum is weaker than the CRC in terms of detecting bit
  61. errors. However, the strength is really in the separation of the data
  62. buffers and the integrity metadata. These two distinct buffers must
  63. match up for an I/O to complete.
  64. The separation of the data and integrity metadata buffers as well as
  65. the choice in checksums is referred to as the Data Integrity
  66. Extensions. As these extensions are outside the scope of the protocol
  67. bodies (T10, T13), Oracle and its partners are trying to standardize
  68. them within the Storage Networking Industry Association.
  69. 3. Kernel Changes
  70. =================
  71. The data integrity framework in Linux enables protection information
  72. to be pinned to I/Os and sent to/received from controllers that
  73. support it.
  74. The advantage to the integrity extensions in SCSI and SATA is that
  75. they enable us to protect the entire path from application to storage
  76. device. However, at the same time this is also the biggest
  77. disadvantage. It means that the protection information must be in a
  78. format that can be understood by the disk.
  79. Generally Linux/POSIX applications are agnostic to the intricacies of
  80. the storage devices they are accessing. The virtual filesystem switch
  81. and the block layer make things like hardware sector size and
  82. transport protocols completely transparent to the application.
  83. However, this level of detail is required when preparing the
  84. protection information to send to a disk. Consequently, the very
  85. concept of an end-to-end protection scheme is a layering violation.
  86. It is completely unreasonable for an application to be aware whether
  87. it is accessing a SCSI or SATA disk.
  88. The data integrity support implemented in Linux attempts to hide this
  89. from the application. As far as the application (and to some extent
  90. the kernel) is concerned, the integrity metadata is opaque information
  91. that's attached to the I/O.
  92. The current implementation allows the block layer to automatically
  93. generate the protection information for any I/O. Eventually the
  94. intent is to move the integrity metadata calculation to userspace for
  95. user data. Metadata and other I/O that originates within the kernel
  96. will still use the automatic generation interface.
  97. Some storage devices allow each hardware sector to be tagged with a
  98. 16-bit value. The owner of this tag space is the owner of the block
  99. device. I.e. the filesystem in most cases. The filesystem can use
  100. this extra space to tag sectors as they see fit. Because the tag
  101. space is limited, the block interface allows tagging bigger chunks by
  102. way of interleaving. This way, 8*16 bits of information can be
  103. attached to a typical 4KB filesystem block.
  104. This also means that applications such as fsck and mkfs will need
  105. access to manipulate the tags from user space. A passthrough
  106. interface for this is being worked on.
  107. 4. Block Layer Implementation Details
  108. =====================================
  109. 4.1 Bio
  110. -------
  111. The data integrity patches add a new field to struct bio when
  112. CONFIG_BLK_DEV_INTEGRITY is enabled. bio_integrity(bio) returns a
  113. pointer to a struct bip which contains the bio integrity payload.
  114. Essentially a bip is a trimmed down struct bio which holds a bio_vec
  115. containing the integrity metadata and the required housekeeping
  116. information (bvec pool, vector count, etc.)
  117. A kernel subsystem can enable data integrity protection on a bio by
  118. calling bio_integrity_alloc(bio). This will allocate and attach the
  119. bip to the bio.
  120. Individual pages containing integrity metadata can subsequently be
  121. attached using bio_integrity_add_page().
  122. bio_free() will automatically free the bip.
  123. 4.2 Block Device
  124. ----------------
  125. Because the format of the protection data is tied to the physical
  126. disk, each block device has been extended with a block integrity
  127. profile (struct blk_integrity). This optional profile is registered
  128. with the block layer using blk_integrity_register().
  129. The profile contains callback functions for generating and verifying
  130. the protection data, as well as getting and setting application tags.
  131. The profile also contains a few constants to aid in completing,
  132. merging and splitting the integrity metadata.
  133. Layered block devices will need to pick a profile that's appropriate
  134. for all subdevices. blk_integrity_compare() can help with that. DM
  135. and MD linear, RAID0 and RAID1 are currently supported. RAID4/5/6
  136. will require extra work due to the application tag.
  137. 5.0 Block Layer Integrity API
  138. =============================
  139. 5.1 Normal Filesystem
  140. ---------------------
  141. The normal filesystem is unaware that the underlying block device
  142. is capable of sending/receiving integrity metadata. The IMD will
  143. be automatically generated by the block layer at submit_bio() time
  144. in case of a WRITE. A READ request will cause the I/O integrity
  145. to be verified upon completion.
  146. IMD generation and verification can be toggled using the::
  147. /sys/block/<bdev>/integrity/write_generate
  148. and::
  149. /sys/block/<bdev>/integrity/read_verify
  150. flags.
  151. 5.2 Integrity-Aware Filesystem
  152. ------------------------------
  153. A filesystem that is integrity-aware can prepare I/Os with IMD
  154. attached. It can also use the application tag space if this is
  155. supported by the block device.
  156. `bool bio_integrity_prep(bio);`
  157. To generate IMD for WRITE and to set up buffers for READ, the
  158. filesystem must call bio_integrity_prep(bio).
  159. Prior to calling this function, the bio data direction and start
  160. sector must be set, and the bio should have all data pages
  161. added. It is up to the caller to ensure that the bio does not
  162. change while I/O is in progress.
  163. Complete bio with error if prepare failed for some reson.
  164. 5.3 Passing Existing Integrity Metadata
  165. ---------------------------------------
  166. Filesystems that either generate their own integrity metadata or
  167. are capable of transferring IMD from user space can use the
  168. following calls:
  169. `struct bip * bio_integrity_alloc(bio, gfp_mask, nr_pages);`
  170. Allocates the bio integrity payload and hangs it off of the bio.
  171. nr_pages indicate how many pages of protection data need to be
  172. stored in the integrity bio_vec list (similar to bio_alloc()).
  173. The integrity payload will be freed at bio_free() time.
  174. `int bio_integrity_add_page(bio, page, len, offset);`
  175. Attaches a page containing integrity metadata to an existing
  176. bio. The bio must have an existing bip,
  177. i.e. bio_integrity_alloc() must have been called. For a WRITE,
  178. the integrity metadata in the pages must be in a format
  179. understood by the target device with the notable exception that
  180. the sector numbers will be remapped as the request traverses the
  181. I/O stack. This implies that the pages added using this call
  182. will be modified during I/O! The first reference tag in the
  183. integrity metadata must have a value of bip->bip_sector.
  184. Pages can be added using bio_integrity_add_page() as long as
  185. there is room in the bip bio_vec array (nr_pages).
  186. Upon completion of a READ operation, the attached pages will
  187. contain the integrity metadata received from the storage device.
  188. It is up to the receiver to process them and verify data
  189. integrity upon completion.
  190. 5.4 Registering A Block Device As Capable Of Exchanging Integrity Metadata
  191. --------------------------------------------------------------------------
  192. To enable integrity exchange on a block device the gendisk must be
  193. registered as capable:
  194. `int blk_integrity_register(gendisk, blk_integrity);`
  195. The blk_integrity struct is a template and should contain the
  196. following::
  197. static struct blk_integrity my_profile = {
  198. .name = "STANDARDSBODY-TYPE-VARIANT-CSUM",
  199. .generate_fn = my_generate_fn,
  200. .verify_fn = my_verify_fn,
  201. .tuple_size = sizeof(struct my_tuple_size),
  202. .tag_size = <tag bytes per hw sector>,
  203. };
  204. 'name' is a text string which will be visible in sysfs. This is
  205. part of the userland API so chose it carefully and never change
  206. it. The format is standards body-type-variant.
  207. E.g. T10-DIF-TYPE1-IP or T13-EPP-0-CRC.
  208. 'generate_fn' generates appropriate integrity metadata (for WRITE).
  209. 'verify_fn' verifies that the data buffer matches the integrity
  210. metadata.
  211. 'tuple_size' must be set to match the size of the integrity
  212. metadata per sector. I.e. 8 for DIF and EPP.
  213. 'tag_size' must be set to identify how many bytes of tag space
  214. are available per hardware sector. For DIF this is either 2 or
  215. 0 depending on the value of the Control Mode Page ATO bit.
  216. ----------------------------------------------------------------------
  217. 2007-12-24 Martin K. Petersen <martin.petersen@oracle.com>