tcmu-design.rst 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405
  1. ====================
  2. TCM Userspace Design
  3. ====================
  4. .. Contents:
  5. 1) Design
  6. a) Background
  7. b) Benefits
  8. c) Design constraints
  9. d) Implementation overview
  10. i. Mailbox
  11. ii. Command ring
  12. iii. Data Area
  13. e) Device discovery
  14. f) Device events
  15. g) Other contingencies
  16. 2) Writing a user pass-through handler
  17. a) Discovering and configuring TCMU uio devices
  18. b) Waiting for events on the device(s)
  19. c) Managing the command ring
  20. 3) A final note
  21. Design
  22. ======
  23. TCM is another name for LIO, an in-kernel iSCSI target (server).
  24. Existing TCM targets run in the kernel. TCMU (TCM in Userspace)
  25. allows userspace programs to be written which act as iSCSI targets.
  26. This document describes the design.
  27. The existing kernel provides modules for different SCSI transport
  28. protocols. TCM also modularizes the data storage. There are existing
  29. modules for file, block device, RAM or using another SCSI device as
  30. storage. These are called "backstores" or "storage engines". These
  31. built-in modules are implemented entirely as kernel code.
  32. Background
  33. ----------
  34. In addition to modularizing the transport protocol used for carrying
  35. SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes
  36. the actual data storage as well. These are referred to as "backstores"
  37. or "storage engines". The target comes with backstores that allow a
  38. file, a block device, RAM, or another SCSI device to be used for the
  39. local storage needed for the exported SCSI LUN. Like the rest of LIO,
  40. these are implemented entirely as kernel code.
  41. These backstores cover the most common use cases, but not all. One new
  42. use case that other non-kernel target solutions, such as tgt, are able
  43. to support is using Gluster's GLFS or Ceph's RBD as a backstore. The
  44. target then serves as a translator, allowing initiators to store data
  45. in these non-traditional networked storage systems, while still only
  46. using standard protocols themselves.
  47. If the target is a userspace process, supporting these is easy. tgt,
  48. for example, needs only a small adapter module for each, because the
  49. modules just use the available userspace libraries for RBD and GLFS.
  50. Adding support for these backstores in LIO is considerably more
  51. difficult, because LIO is entirely kernel code. Instead of undertaking
  52. the significant work to port the GLFS or RBD APIs and protocols to the
  53. kernel, another approach is to create a userspace pass-through
  54. backstore for LIO, "TCMU".
  55. Benefits
  56. --------
  57. In addition to allowing relatively easy support for RBD and GLFS, TCMU
  58. will also allow easier development of new backstores. TCMU combines
  59. with the LIO loopback fabric to become something similar to FUSE
  60. (Filesystem in Userspace), but at the SCSI layer instead of the
  61. filesystem layer. A SUSE, if you will.
  62. The disadvantage is there are more distinct components to configure, and
  63. potentially to malfunction. This is unavoidable, but hopefully not
  64. fatal if we're careful to keep things as simple as possible.
  65. Design constraints
  66. ------------------
  67. - Good performance: high throughput, low latency
  68. - Cleanly handle if userspace:
  69. 1) never attaches
  70. 2) hangs
  71. 3) dies
  72. 4) misbehaves
  73. - Allow future flexibility in user & kernel implementations
  74. - Be reasonably memory-efficient
  75. - Simple to configure & run
  76. - Simple to write a userspace backend
  77. Implementation overview
  78. -----------------------
  79. The core of the TCMU interface is a memory region that is shared
  80. between kernel and userspace. Within this region is: a control area
  81. (mailbox); a lockless producer/consumer circular buffer for commands
  82. to be passed up, and status returned; and an in/out data buffer area.
  83. TCMU uses the pre-existing UIO subsystem. UIO allows device driver
  84. development in userspace, and this is conceptually very close to the
  85. TCMU use case, except instead of a physical device, TCMU implements a
  86. memory-mapped layout designed for SCSI commands. Using UIO also
  87. benefits TCMU by handling device introspection (e.g. a way for
  88. userspace to determine how large the shared region is) and signaling
  89. mechanisms in both directions.
  90. There are no embedded pointers in the memory region. Everything is
  91. expressed as an offset from the region's starting address. This allows
  92. the ring to still work if the user process dies and is restarted with
  93. the region mapped at a different virtual address.
  94. See target_core_user.h for the struct definitions.
  95. The Mailbox
  96. -----------
  97. The mailbox is always at the start of the shared memory region, and
  98. contains a version, details about the starting offset and size of the
  99. command ring, and head and tail pointers to be used by the kernel and
  100. userspace (respectively) to put commands on the ring, and indicate
  101. when the commands are completed.
  102. version - 1 (userspace should abort if otherwise)
  103. flags:
  104. - TCMU_MAILBOX_FLAG_CAP_OOOC:
  105. indicates out-of-order completion is supported.
  106. See "The Command Ring" for details.
  107. cmdr_off
  108. The offset of the start of the command ring from the start
  109. of the memory region, to account for the mailbox size.
  110. cmdr_size
  111. The size of the command ring. This does *not* need to be a
  112. power of two.
  113. cmd_head
  114. Modified by the kernel to indicate when a command has been
  115. placed on the ring.
  116. cmd_tail
  117. Modified by userspace to indicate when it has completed
  118. processing of a command.
  119. The Command Ring
  120. ----------------
  121. Commands are placed on the ring by the kernel incrementing
  122. mailbox.cmd_head by the size of the command, modulo cmdr_size, and
  123. then signaling userspace via uio_event_notify(). Once the command is
  124. completed, userspace updates mailbox.cmd_tail in the same way and
  125. signals the kernel via a 4-byte write(). When cmd_head equals
  126. cmd_tail, the ring is empty -- no commands are currently waiting to be
  127. processed by userspace.
  128. TCMU commands are 8-byte aligned. They start with a common header
  129. containing "len_op", a 32-bit value that stores the length, as well as
  130. the opcode in the lowest unused bits. It also contains cmd_id and
  131. flags fields for setting by the kernel (kflags) and userspace
  132. (uflags).
  133. Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.
  134. When the opcode is CMD, the entry in the command ring is a struct
  135. tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via
  136. tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the
  137. overall shared memory region, not the entry. The data in/out buffers
  138. are accessible via tht req.iov[] array. iov_cnt contains the number of
  139. entries in iov[] needed to describe either the Data-In or Data-Out
  140. buffers. For bidirectional commands, iov_cnt specifies how many iovec
  141. entries cover the Data-Out area, and iov_bidi_cnt specifies how many
  142. iovec entries immediately after that in iov[] cover the Data-In
  143. area. Just like other fields, iov.iov_base is an offset from the start
  144. of the region.
  145. When completing a command, userspace sets rsp.scsi_status, and
  146. rsp.sense_buffer if necessary. Userspace then increments
  147. mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the
  148. kernel via the UIO method, a 4-byte write to the file descriptor.
  149. If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is
  150. capable of handling out-of-order completions. In this case, userspace can
  151. handle command in different order other than original. Since kernel would
  152. still process the commands in the same order it appeared in the command
  153. ring, userspace need to update the cmd->id when completing the
  154. command(a.k.a steal the original command's entry).
  155. When the opcode is PAD, userspace only updates cmd_tail as above --
  156. it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry
  157. is contiguous within the command ring.)
  158. More opcodes may be added in the future. If userspace encounters an
  159. opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in
  160. hdr.uflags, update cmd_tail, and proceed with processing additional
  161. commands, if any.
  162. The Data Area
  163. -------------
  164. This is shared-memory space after the command ring. The organization
  165. of this area is not defined in the TCMU interface, and userspace
  166. should access only the parts referenced by pending iovs.
  167. Device Discovery
  168. ----------------
  169. Other devices may be using UIO besides TCMU. Unrelated user processes
  170. may also be handling different sets of TCMU devices. TCMU userspace
  171. processes must find their devices by scanning sysfs
  172. class/uio/uio*/name. For TCMU devices, these names will be of the
  173. format::
  174. tcm-user/<hba_num>/<device_name>/<subtype>/<path>
  175. where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num>
  176. and <device_name> allow userspace to find the device's path in the
  177. kernel target's configfs tree. Assuming the usual mount point, it is
  178. found at::
  179. /sys/kernel/config/target/core/user_<hba_num>/<device_name>
  180. This location contains attributes such as "hw_block_size", that
  181. userspace needs to know for correct operation.
  182. <subtype> will be a userspace-process-unique string to identify the
  183. TCMU device as expecting to be backed by a certain handler, and <path>
  184. will be an additional handler-specific string for the user process to
  185. configure the device, if needed. The name cannot contain ':', due to
  186. LIO limitations.
  187. For all devices so discovered, the user handler opens /dev/uioX and
  188. calls mmap()::
  189. mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
  190. where size must be equal to the value read from
  191. /sys/class/uio/uioX/maps/map0/size.
  192. Device Events
  193. -------------
  194. If a new device is added or removed, a notification will be broadcast
  195. over netlink, using a generic netlink family name of "TCM-USER" and a
  196. multicast group named "config". This will include the UIO name as
  197. described in the previous section, as well as the UIO minor
  198. number. This should allow userspace to identify both the UIO device and
  199. the LIO device, so that after determining the device is supported
  200. (based on subtype) it can take the appropriate action.
  201. Other contingencies
  202. -------------------
  203. Userspace handler process never attaches:
  204. - TCMU will post commands, and then abort them after a timeout period
  205. (30 seconds.)
  206. Userspace handler process is killed:
  207. - It is still possible to restart and re-connect to TCMU
  208. devices. Command ring is preserved. However, after the timeout period,
  209. the kernel will abort pending tasks.
  210. Userspace handler process hangs:
  211. - The kernel will abort pending tasks after a timeout period.
  212. Userspace handler process is malicious:
  213. - The process can trivially break the handling of devices it controls,
  214. but should not be able to access kernel memory outside its shared
  215. memory areas.
  216. Writing a user pass-through handler (with example code)
  217. =======================================================
  218. A user process handing a TCMU device must support the following:
  219. a) Discovering and configuring TCMU uio devices
  220. b) Waiting for events on the device(s)
  221. c) Managing the command ring: Parsing operations and commands,
  222. performing work as needed, setting response fields (scsi_status and
  223. possibly sense_buffer), updating cmd_tail, and notifying the kernel
  224. that work has been finished
  225. First, consider instead writing a plugin for tcmu-runner. tcmu-runner
  226. implements all of this, and provides a higher-level API for plugin
  227. authors.
  228. TCMU is designed so that multiple unrelated processes can manage TCMU
  229. devices separately. All handlers should make sure to only open their
  230. devices, based opon a known subtype string.
  231. a) Discovering and configuring TCMU UIO devices::
  232. /* error checking omitted for brevity */
  233. int fd, dev_fd;
  234. char buf[256];
  235. unsigned long long map_len;
  236. void *map;
  237. fd = open("/sys/class/uio/uio0/name", O_RDONLY);
  238. ret = read(fd, buf, sizeof(buf));
  239. close(fd);
  240. buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
  241. /* we only want uio devices whose name is a format we expect */
  242. if (strncmp(buf, "tcm-user", 8))
  243. exit(-1);
  244. /* Further checking for subtype also needed here */
  245. fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY);
  246. ret = read(fd, buf, sizeof(buf));
  247. close(fd);
  248. str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
  249. map_len = strtoull(buf, NULL, 0);
  250. dev_fd = open("/dev/uio0", O_RDWR);
  251. map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0);
  252. b) Waiting for events on the device(s)
  253. while (1) {
  254. char buf[4];
  255. int ret = read(dev_fd, buf, 4); /* will block */
  256. handle_device_events(dev_fd, map);
  257. }
  258. c) Managing the command ring::
  259. #include <linux/target_core_user.h>
  260. int handle_device_events(int fd, void *map)
  261. {
  262. struct tcmu_mailbox *mb = map;
  263. struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
  264. int did_some_work = 0;
  265. /* Process events from cmd ring until we catch up with cmd_head */
  266. while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) {
  267. if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) {
  268. uint8_t *cdb = (void *)mb + ent->req.cdb_off;
  269. bool success = true;
  270. /* Handle command here. */
  271. printf("SCSI opcode: 0x%x\n", cdb[0]);
  272. /* Set response fields */
  273. if (success)
  274. ent->rsp.scsi_status = SCSI_NO_SENSE;
  275. else {
  276. /* Also fill in rsp->sense_buffer here */
  277. ent->rsp.scsi_status = SCSI_CHECK_CONDITION;
  278. }
  279. }
  280. else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) {
  281. /* Tell the kernel we didn't handle unknown opcodes */
  282. ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP;
  283. }
  284. else {
  285. /* Do nothing for PAD entries except update cmd_tail */
  286. }
  287. /* update cmd_tail */
  288. mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size;
  289. ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
  290. did_some_work = 1;
  291. }
  292. /* Notify the kernel that work has been finished */
  293. if (did_some_work) {
  294. uint32_t buf = 0;
  295. write(fd, &buf, 4);
  296. }
  297. return 0;
  298. }
  299. A final note
  300. ============
  301. Please be careful to return codes as defined by the SCSI
  302. specifications. These are different than some values defined in the
  303. scsi/scsi.h include file. For example, CHECK CONDITION's status code
  304. is 2, not 1.