TechNotes.txt 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361
  1. ## @file
  2. #
  3. # Technical notes for the virtio-net driver.
  4. #
  5. # Copyright (C) 2013, Red Hat, Inc.
  6. #
  7. # SPDX-License-Identifier: BSD-2-Clause-Patent
  8. #
  9. ##
  10. Disclaimer
  11. ----------
  12. All statements concerning standards and specifications are informative and not
  13. normative. They are made in good faith. Corrections are most welcome on the
  14. edk2-devel mailing list.
  15. The following documents have been perused while writing the driver and this
  16. document:
  17. - Unified Extensible Firmware Interface Specification, Version 2.3.1, Errata C;
  18. June 27, 2012
  19. - Driver Writer's Guide for UEFI 2.3.1, 03/08/2012, Version 1.01;
  20. - Virtio PCI Card Specification, v0.9.5 DRAFT, 2012 May 7.
  21. Summary
  22. -------
  23. The VirtioNetDxe UEFI_DRIVER implements the Simple Network Protocol for
  24. virtio-net devices. Higher level protocols are automatically installed on top
  25. of it by the DXE Core / the ConnectController() boot service, enabling for
  26. virtio-net devices eg. DHCP configuration, TCP transfers with edk2 StdLib
  27. applications, and PXE booting in OVMF.
  28. UEFI driver structure
  29. ---------------------
  30. A driver instance, belonging to a given virtio-net device, can be in one of
  31. four states at any time. The states stack up as follows below. The state
  32. transitions are labeled with the primary function (and its important callees
  33. faithfully indented) that implement the transition.
  34. | ^
  35. | |
  36. [DriverBinding.c] | | [DriverBinding.c]
  37. VirtioNetDriverBindingStart | | VirtioNetDriverBindingStop
  38. VirtioNetSnpPopulate | | VirtioNetSnpEvacuate
  39. VirtioNetGetFeatures | |
  40. v |
  41. +-------------------------+
  42. | EfiSimpleNetworkStopped |
  43. +-------------------------+
  44. | ^
  45. [SnpStart.c] | | [SnpStop.c]
  46. VirtioNetStart | | VirtioNetStop
  47. | |
  48. v |
  49. +-------------------------+
  50. | EfiSimpleNetworkStarted |
  51. +-------------------------+
  52. | ^
  53. [SnpInitialize.c] | | [SnpShutdown.c]
  54. VirtioNetInitialize | | VirtioNetShutdown
  55. VirtioNetInitRing {Rx, Tx} | | VirtioNetShutdownRx [SnpSharedHelpers.c]
  56. VirtioRingInit | | VirtIo->UnmapSharedBuffer
  57. VirtioRingMap | | VirtIo->FreeSharedPages
  58. VirtioNetInitTx | | VirtioNetShutdownTx [SnpSharedHelpers.c]
  59. VirtIo->AllocateShare... | | VirtIo->UnmapSharedBuffer
  60. VirtioMapAllBytesInSh... | | VirtIo->FreeSharedPages
  61. VirtioNetInitRx | | VirtioNetUninitRing [SnpSharedHelpers.c]
  62. VirtIo->AllocateShare... | | {Tx, Rx}
  63. VirtioMapAllBytesInSh... | | VirtIo->UnmapSharedBuffer
  64. | | VirtioRingUninit
  65. v |
  66. +-----------------------------+
  67. | EfiSimpleNetworkInitialized |
  68. +-----------------------------+
  69. The state at the top means "nonexistent" and is hence unnamed on the diagram --
  70. a driver instance actually doesn't exist at that point. The transition
  71. functions out of and into that state implement the Driver Binding Protocol.
  72. The lower three states characterize an existent driver instance and are all
  73. states defined by the Simple Network Protocol. The transition functions between
  74. them are member functions of the Simple Network Protocol.
  75. Each transition function validates its expected source state and its
  76. parameters. For example, VirtioNetDriverBindingStop will refuse to disconnect
  77. from the controller unless it's in EfiSimpleNetworkStopped.
  78. Driver instance states (Simple Network Protocol)
  79. ------------------------------------------------
  80. In the EfiSimpleNetworkStopped state, the virtio-net device is (has been)
  81. re-set. No resources are allocated for networking / traffic purposes. The MAC
  82. address and other device attributes have been retrieved from the device (this
  83. is necessary for completing the VirtioNetDriverBindingStart transition).
  84. The EfiSimpleNetworkStarted is completely identical to the
  85. EfiSimpleNetworkStopped state for virtio-net, in the functional and
  86. resource-usage sense. This state is mandated / provided by the Simple Network
  87. Protocol for flexibility that the virtio-net driver doesn't exploit.
  88. In particular, the EfiSimpleNetworkStarted state is the target of the Shutdown
  89. SNP member function, and must therefore correspond to a hardware configuration
  90. where "[it] is safe for another driver to initialize". (Clearly another UEFI
  91. driver could not do that due to the exclusivity of the driver binding that
  92. VirtioNetDriverBindingStart() installs, but a later OS driver might qualify.)
  93. The EfiSimpleNetworkInitialized state is the live state of the virtio NIC / the
  94. driver instance. Virtio and other resources required for network traffic have
  95. been allocated, and the following SNP member functions are available (in
  96. addition to VirtioNetShutdown which leaves the state):
  97. - VirtioNetReceive [SnpReceive.c]: poll the virtio NIC for an Rx packet that
  98. may have arrived asynchronously;
  99. - VirtioNetTransmit [SnpTransmit.c]: queue a Tx packet for asynchronous
  100. transmission (meant to be used together with VirtioNetGetStatus);
  101. - VirtioNetGetStatus [SnpGetStatus.c]: query link status and status of pending
  102. Tx packets;
  103. - VirtioNetMcastIpToMac [SnpMcastIpToMac.c]: transform a multicast IPv4/IPv6
  104. address into a multicast MAC address;
  105. - VirtioNetReceiveFilters [SnpReceiveFilters.c]: emulate unicast / multicast /
  106. broadcast filter configuration (not their actual effect -- a more liberal
  107. filter setting than requested is allowed by the UEFI specification).
  108. The following SNP member functions are not supported [SnpUnsupported.c]:
  109. - VirtioNetReset: reinitialize the virtio NIC without shutting it down (a loop
  110. from/to EfiSimpleNetworkInitialized);
  111. - VirtioNetStationAddress: assign a new MAC address to the virtio NIC,
  112. - VirtioNetStatistics: collect statistics,
  113. - VirtioNetNvData: access non-volatile data on the virtio NIC.
  114. Missing support for these functions is allowed by the UEFI specification and
  115. doesn't seem to trip up higher level protocols.
  116. Events and task priority levels
  117. -------------------------------
  118. The UEFI specification defines a sophisticated mechanism for asynchronous
  119. events / callbacks (see "6.1 Event, Timer, and Task Priority Services" for
  120. details). Such callbacks work like software interrupts, and some notion of
  121. locking / masking is important to implement critical sections (atomic or
  122. exclusive access to data or a device). This notion is defined as Task Priority
  123. Levels.
  124. The virtio-net driver for OVMF must concern itself with events for two reasons:
  125. - The Simple Network Protocol provides its clients with a (non-optional) WAIT
  126. type event called WaitForPacket: it allows them to check or wait for Rx
  127. packets by polling or blocking on this event. (This functionality overlaps
  128. with the Receive member function.) The event is available to clients starting
  129. with EfiSimpleNetworkStopped (inclusive).
  130. The virtio-net driver is informed about such client polling or blockage by
  131. receiving an asynchronous callback (a software interrupt). In the callback
  132. function the driver must interrogate the driver instance state, and if it is
  133. EfiSimpleNetworkInitialized, access the Rx queue and see if any packets are
  134. available for consumption. If so, it must signal the WaitForPacket WAIT type
  135. event, waking the client.
  136. For simplicity and safety, all parts of the virtio-net driver that access any
  137. bit of the driver instance (data or device) run at the TPL_CALLBACK level.
  138. This is the highest level allowed for an SNP implementation, and all code
  139. protected in this manner satisfies even stricter non-blocking requirements
  140. than what's documented for TPL_CALLBACK.
  141. The task priority level for the WaitForPacket callback too is set by the
  142. driver, the choice is TPL_CALLBACK again. This in effect serializes the
  143. WaitForPacket callback (VirtioNetIsPacketAvailable [Events.c]) with "normal"
  144. parts of the driver.
  145. - According to the Driver Writer's Guide, a network driver should install a
  146. callback function for the global EXIT_BOOT_SERVICES event (a special NOTIFY
  147. type event). When the ExitBootServices() boot service has cleaned up internal
  148. firmware state and is about to pass control to the OS, any network driver has
  149. to stop any in-flight DMA transfers, lest it corrupts OS memory. For this
  150. reason EXIT_BOOT_SERVICES is emitted and the network driver must abort
  151. in-flight DMA transfers.
  152. This callback (VirtioNetExitBoot) is synchronized with the rest of the driver
  153. code just the same as explained for WaitForPacket. In
  154. EfiSimpleNetworkInitialized state it resets the virtio NIC, halting all data
  155. transfer. After the callback returns, no further driver code is expected to
  156. be scheduled.
  157. Virtio internals -- Rx
  158. ----------------------
  159. Requests (Rx and Tx alike) are always submitted by the guest and processed by
  160. the host. For Tx, processing means transmission. For Rx, processing means
  161. filling in the request with an incoming packet. Submitted requests exist on the
  162. "Available Ring", and answered (processed) requests show up on the "Used Ring".
  163. Packet data includes the media (Ethernet) header: destination MAC, source MAC,
  164. and Ethertype (14 bytes total).
  165. The following structures implement packet reception. Most of them are defined
  166. in the Virtio specification, the only driver-specific trait here is the static
  167. pre-configuration of the two-part descriptor chains, in VirtioNetInitRx. The
  168. diagram is simplified.
  169. Available Index Available Index
  170. last processed incremented
  171. by the host by the guest
  172. v -------> v
  173. Available +-------+-------+-------+-------+-------+
  174. Ring |DescIdx|DescIdx|DescIdx|DescIdx|DescIdx|
  175. +-------+-------+-------+-------+-------+
  176. =D6 =D2
  177. D2 D3 D4 D5 D6 D7
  178. Descr. +----------+----------++----------+----------++----------+----------+
  179. Table |Adr:Len:Nx|Adr:Len:Nx||Adr:Len:Nx|Adr:Len:Nx||Adr:Len:Nx|Adr:Len:Nx|
  180. +----------+----------++----------+----------++----------+----------+
  181. =A2 =D3 =A3 =A4 =D5 =A5 =A6 =D7 =A7
  182. A2 A3 A4 A5 A6 A7
  183. Receive +---------------+---------------+---------------+
  184. Destination |vnet hdr:packet|vnet hdr:packet|vnet hdr:packet|
  185. Area +---------------+---------------+---------------+
  186. Used Index Used Index incremented
  187. last processed by the guest by the host
  188. v -------> v
  189. Used +-----------+-----------+-----------+-----------+-----------+
  190. Ring |DescIdx:Len|DescIdx:Len|DescIdx:Len|DescIdx:Len|DescIdx:Len|
  191. +-----------+-----------+-----------+-----------+-----------+
  192. =D4
  193. In VirtioNetInitRx, the guest allocates the fixed size Receive Destination
  194. Area, which accommodates all packets delivered asynchronously by the host. To
  195. each packet, a slice of this area is dedicated; each slice is further
  196. subdivided into virtio-net request header and network packet data. The
  197. (device-physical) addresses of these sub-slices are denoted with A2, A3, A4 and
  198. so on. Importantly, an even-subscript "A" always belongs to a virtio-net
  199. request header, while an odd-subscript "A" always belongs to a packet
  200. sub-slice.
  201. Furthermore, the guest lays out a static pattern in the Descriptor Table. For
  202. each packet that can be in-flight or already arrived from the host,
  203. VirtioNetInitRx sets up a separate, two-part descriptor chain. For packet N,
  204. the Nth descriptor chain is set up as follows:
  205. - the first (=head) descriptor, with even index, points to the fixed-size
  206. sub-slice receiving the virtio-net request header,
  207. - the second descriptor (with odd index) points to the fixed (1514 byte) size
  208. sub-slice receiving the packet data,
  209. - a link from the first (head) descriptor in the chain is established to the
  210. second (tail) descriptor in the chain.
  211. Finally, the guest populates the Available Ring with the indices of the head
  212. descriptors. All descriptor indices on both the Available Ring and the Used
  213. Ring are even.
  214. Packet reception occurs as follows:
  215. - The host consumes a descriptor index off the Available Ring. This index is
  216. even (=2*N), and fingers the head descriptor of the chain belonging to packet
  217. N.
  218. - The host reads the descriptors D(2*N) and -- following the Next link there
  219. --- D(2*N+1), and stores the virtio-net request header at A(2*N), and the
  220. packet data at A(2*N+1).
  221. - The host places the index of the head descriptor, 2*N, onto the Used Ring,
  222. and sets the Len field in the same Used Ring Element to the total number of
  223. bytes transferred for the entire descriptor chain. This enables the guest to
  224. identify the length of Rx packets.
  225. - VirtioNetReceive polls the Used Ring. If a new Used Ring Element shows up, it
  226. copies the data out to the caller, and recycles the index of the head
  227. descriptor (ie. 2*N) to the Available Ring.
  228. - Because the host can process (answer) Rx requests in any order theoretically,
  229. the order of head descriptor indices on each of the Available Ring and the
  230. Used Ring is virtually random. (Except right after the initial population in
  231. VirtioNetInitRx, when the Available Ring is full and increasing, and the Used
  232. Ring is empty.)
  233. - If the Available Ring is empty, the host is forced to drop packets. If the
  234. Used Ring is empty, VirtioNetReceive returns EFI_NOT_READY (no packet
  235. available).
  236. Virtio internals -- Tx
  237. ----------------------
  238. The transmission structure erected by VirtioNetInitTx is similar, it differs
  239. in the following:
  240. - There is no Receive Destination Area.
  241. - Each head descriptor, D(2*N), points to a read-only virtio-net request header
  242. that is shared by all of the head descriptors. This virtio-net request header
  243. is never modified by the host.
  244. - Each tail descriptor is re-pointed to the device-mapped address of the
  245. caller-supplied packet buffer whenever VirtioNetTransmit places the
  246. corresponding head descriptor on the Available Ring. A reverse mapping, from
  247. the device-mapped address to the caller-supplied packet address, is saved in
  248. an associative data structure that belongs to the driver instance.
  249. - Per spec, the caller is responsible to hang on to the unmodified packet
  250. buffer until it is reported transmitted by VirtioNetGetStatus.
  251. Steps of packet transmission:
  252. - Client code calls VirtioNetTransmit. VirtioNetTransmit tracks free descriptor
  253. chains by keeping the indices of their head descriptors in a stack that is
  254. private to the driver instance. All elements of the stack are even.
  255. - If the stack is empty (that is, each descriptor chain, in isolation, is
  256. either pending transmission, or has been processed by the host but not
  257. yet recycled by a VirtioNetGetStatus call), then VirtioNetTransmit returns
  258. EFI_NOT_READY.
  259. - Otherwise the index of a free chain's head descriptor is popped from the
  260. stack. The linked tail descriptor is re-pointed as discussed above. The head
  261. descriptor's index is pushed on the Available Ring.
  262. - The host moves the head descriptor index from the Available Ring to the Used
  263. Ring when it transmits the packet.
  264. - Client code calls VirtioNetGetStatus. In case the Used Ring is empty, the
  265. function reports no Tx completion. Otherwise, a head descriptor's index is
  266. consumed from the Used Ring and recycled to the private stack. The client
  267. code's original packet buffer address is calculated by fetching the
  268. device-mapped address from the tail descriptor (where it has been stored at
  269. VirtioNetTransmit time), and by looking up the device-mapped address in the
  270. associative data structure. The reverse-mapped packet buffer address is
  271. returned to the caller.
  272. - The Len field of the Used Ring Element is not checked. The host is assumed to
  273. have transmitted the entire packet -- VirtioNetTransmit had forced it below
  274. 1514 bytes (inclusive). The Virtio specification suggests this packet size is
  275. always accepted (and a lower MTU could be encountered on any later hop as
  276. well). Additionally, there's no good way to report a short transmit via
  277. VirtioNetGetStatus; EFI_DEVICE_ERROR seems too serious from the specification
  278. and higher level protocols could interpret it as a fatal condition.
  279. - The host can theoretically reorder head descriptor indices when moving them
  280. from the Available Ring to the Used Ring (out of order transmission). Because
  281. of this (and the choice of a stack over a list for free descriptor chain
  282. tracking) the order of head descriptor indices on either Ring is
  283. unpredictable.