msg_zerocopy.rst 8.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256
  1. ============
  2. MSG_ZEROCOPY
  3. ============
  4. Intro
  5. =====
  6. The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
  7. The feature is currently implemented for TCP and UDP sockets.
  8. Opportunity and Caveats
  9. -----------------------
  10. Copying large buffers between user process and kernel can be
  11. expensive. Linux supports various interfaces that eschew copying,
  12. such as sendpage and splice. The MSG_ZEROCOPY flag extends the
  13. underlying copy avoidance mechanism to common socket send calls.
  14. Copy avoidance is not a free lunch. As implemented, with page pinning,
  15. it replaces per byte copy cost with page accounting and completion
  16. notification overhead. As a result, MSG_ZEROCOPY is generally only
  17. effective at writes over around 10 KB.
  18. Page pinning also changes system call semantics. It temporarily shares
  19. the buffer between process and network stack. Unlike with copying, the
  20. process cannot immediately overwrite the buffer after system call
  21. return without possibly modifying the data in flight. Kernel integrity
  22. is not affected, but a buggy program can possibly corrupt its own data
  23. stream.
  24. The kernel returns a notification when it is safe to modify data.
  25. Converting an existing application to MSG_ZEROCOPY is not always as
  26. trivial as just passing the flag, then.
  27. More Info
  28. ---------
  29. Much of this document was derived from a longer paper presented at
  30. netdev 2.1. For more in-depth information see that paper and talk,
  31. the excellent reporting over at LWN.net or read the original code.
  32. paper, slides, video
  33. https://netdevconf.org/2.1/session.html?debruijn
  34. LWN article
  35. https://lwn.net/Articles/726917/
  36. patchset
  37. [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY
  38. https://lkml.kernel.org/netdev/20170803202945.70750-1-willemdebruijn.kernel@gmail.com
  39. Interface
  40. =========
  41. Passing the MSG_ZEROCOPY flag is the most obvious step to enable copy
  42. avoidance, but not the only one.
  43. Socket Setup
  44. ------------
  45. The kernel is permissive when applications pass undefined flags to the
  46. send system call. By default it simply ignores these. To avoid enabling
  47. copy avoidance mode for legacy processes that accidentally already pass
  48. this flag, a process must first signal intent by setting a socket option:
  49. ::
  50. if (setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one)))
  51. error(1, errno, "setsockopt zerocopy");
  52. Transmission
  53. ------------
  54. The change to send (or sendto, sendmsg, sendmmsg) itself is trivial.
  55. Pass the new flag.
  56. ::
  57. ret = send(fd, buf, sizeof(buf), MSG_ZEROCOPY);
  58. A zerocopy failure will return -1 with errno ENOBUFS. This happens if
  59. the socket option was not set, the socket exceeds its optmem limit or
  60. the user exceeds its ulimit on locked pages.
  61. Mixing copy avoidance and copying
  62. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  63. Many workloads have a mixture of large and small buffers. Because copy
  64. avoidance is more expensive than copying for small packets, the
  65. feature is implemented as a flag. It is safe to mix calls with the flag
  66. with those without.
  67. Notifications
  68. -------------
  69. The kernel has to notify the process when it is safe to reuse a
  70. previously passed buffer. It queues completion notifications on the
  71. socket error queue, akin to the transmit timestamping interface.
  72. The notification itself is a simple scalar value. Each socket
  73. maintains an internal unsigned 32-bit counter. Each send call with
  74. MSG_ZEROCOPY that successfully sends data increments the counter. The
  75. counter is not incremented on failure or if called with length zero.
  76. The counter counts system call invocations, not bytes. It wraps after
  77. UINT_MAX calls.
  78. Notification Reception
  79. ~~~~~~~~~~~~~~~~~~~~~~
  80. The below snippet demonstrates the API. In the simplest case, each
  81. send syscall is followed by a poll and recvmsg on the error queue.
  82. Reading from the error queue is always a non-blocking operation. The
  83. poll call is there to block until an error is outstanding. It will set
  84. POLLERR in its output flags. That flag does not have to be set in the
  85. events field. Errors are signaled unconditionally.
  86. ::
  87. pfd.fd = fd;
  88. pfd.events = 0;
  89. if (poll(&pfd, 1, -1) != 1 || pfd.revents & POLLERR == 0)
  90. error(1, errno, "poll");
  91. ret = recvmsg(fd, &msg, MSG_ERRQUEUE);
  92. if (ret == -1)
  93. error(1, errno, "recvmsg");
  94. read_notification(msg);
  95. The example is for demonstration purpose only. In practice, it is more
  96. efficient to not wait for notifications, but read without blocking
  97. every couple of send calls.
  98. Notifications can be processed out of order with other operations on
  99. the socket. A socket that has an error queued would normally block
  100. other operations until the error is read. Zerocopy notifications have
  101. a zero error code, however, to not block send and recv calls.
  102. Notification Batching
  103. ~~~~~~~~~~~~~~~~~~~~~
  104. Multiple outstanding packets can be read at once using the recvmmsg
  105. call. This is often not needed. In each message the kernel returns not
  106. a single value, but a range. It coalesces consecutive notifications
  107. while one is outstanding for reception on the error queue.
  108. When a new notification is about to be queued, it checks whether the
  109. new value extends the range of the notification at the tail of the
  110. queue. If so, it drops the new notification packet and instead increases
  111. the range upper value of the outstanding notification.
  112. For protocols that acknowledge data in-order, like TCP, each
  113. notification can be squashed into the previous one, so that no more
  114. than one notification is outstanding at any one point.
  115. Ordered delivery is the common case, but not guaranteed. Notifications
  116. may arrive out of order on retransmission and socket teardown.
  117. Notification Parsing
  118. ~~~~~~~~~~~~~~~~~~~~
  119. The below snippet demonstrates how to parse the control message: the
  120. read_notification() call in the previous snippet. A notification
  121. is encoded in the standard error format, sock_extended_err.
  122. The level and type fields in the control data are protocol family
  123. specific, IP_RECVERR or IPV6_RECVERR.
  124. Error origin is the new type SO_EE_ORIGIN_ZEROCOPY. ee_errno is zero,
  125. as explained before, to avoid blocking read and write system calls on
  126. the socket.
  127. The 32-bit notification range is encoded as [ee_info, ee_data]. This
  128. range is inclusive. Other fields in the struct must be treated as
  129. undefined, bar for ee_code, as discussed below.
  130. ::
  131. struct sock_extended_err *serr;
  132. struct cmsghdr *cm;
  133. cm = CMSG_FIRSTHDR(msg);
  134. if (cm->cmsg_level != SOL_IP &&
  135. cm->cmsg_type != IP_RECVERR)
  136. error(1, 0, "cmsg");
  137. serr = (void *) CMSG_DATA(cm);
  138. if (serr->ee_errno != 0 ||
  139. serr->ee_origin != SO_EE_ORIGIN_ZEROCOPY)
  140. error(1, 0, "serr");
  141. printf("completed: %u..%u\n", serr->ee_info, serr->ee_data);
  142. Deferred copies
  143. ~~~~~~~~~~~~~~~
  144. Passing flag MSG_ZEROCOPY is a hint to the kernel to apply copy
  145. avoidance, and a contract that the kernel will queue a completion
  146. notification. It is not a guarantee that the copy is elided.
  147. Copy avoidance is not always feasible. Devices that do not support
  148. scatter-gather I/O cannot send packets made up of kernel generated
  149. protocol headers plus zerocopy user data. A packet may need to be
  150. converted to a private copy of data deep in the stack, say to compute
  151. a checksum.
  152. In all these cases, the kernel returns a completion notification when
  153. it releases its hold on the shared pages. That notification may arrive
  154. before the (copied) data is fully transmitted. A zerocopy completion
  155. notification is not a transmit completion notification, therefore.
  156. Deferred copies can be more expensive than a copy immediately in the
  157. system call, if the data is no longer warm in the cache. The process
  158. also incurs notification processing cost for no benefit. For this
  159. reason, the kernel signals if data was completed with a copy, by
  160. setting flag SO_EE_CODE_ZEROCOPY_COPIED in field ee_code on return.
  161. A process may use this signal to stop passing flag MSG_ZEROCOPY on
  162. subsequent requests on the same socket.
  163. Implementation
  164. ==============
  165. Loopback
  166. --------
  167. Data sent to local sockets can be queued indefinitely if the receive
  168. process does not read its socket. Unbound notification latency is not
  169. acceptable. For this reason all packets generated with MSG_ZEROCOPY
  170. that are looped to a local socket will incur a deferred copy. This
  171. includes looping onto packet sockets (e.g., tcpdump) and tun devices.
  172. Testing
  173. =======
  174. More realistic example code can be found in the kernel source under
  175. tools/testing/selftests/net/msg_zerocopy.c.
  176. Be cognizant of the loopback constraint. The test can be run between
  177. a pair of hosts. But if run between a local pair of processes, for
  178. instance when run with msg_zerocopy.sh between a veth pair across
  179. namespaces, the test will not show any improvement. For testing, the
  180. loopback restriction can be temporarily relaxed by making
  181. skb_orphan_frags_rx identical to skb_orphan_frags.