xillybus.rst 18 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379
  1. ==========================================
  2. Xillybus driver for generic FPGA interface
  3. ==========================================
  4. :Author: Eli Billauer, Xillybus Ltd. (http://xillybus.com)
  5. :Email: eli.billauer@gmail.com or as advertised on Xillybus' site.
  6. .. Contents:
  7. - Introduction
  8. -- Background
  9. -- Xillybus Overview
  10. - Usage
  11. -- User interface
  12. -- Synchronization
  13. -- Seekable pipes
  14. - Internals
  15. -- Source code organization
  16. -- Pipe attributes
  17. -- Host never reads from the FPGA
  18. -- Channels, pipes, and the message channel
  19. -- Data streaming
  20. -- Data granularity
  21. -- Probing
  22. -- Buffer allocation
  23. -- The "nonempty" message (supporting poll)
  24. Introduction
  25. ============
  26. Background
  27. ----------
  28. An FPGA (Field Programmable Gate Array) is a piece of logic hardware, which
  29. can be programmed to become virtually anything that is usually found as a
  30. dedicated chipset: For instance, a display adapter, network interface card,
  31. or even a processor with its peripherals. FPGAs are the LEGO of hardware:
  32. Based upon certain building blocks, you make your own toys the way you like
  33. them. It's usually pointless to reimplement something that is already
  34. available on the market as a chipset, so FPGAs are mostly used when some
  35. special functionality is needed, and the production volume is relatively low
  36. (hence not justifying the development of an ASIC).
  37. The challenge with FPGAs is that everything is implemented at a very low
  38. level, even lower than assembly language. In order to allow FPGA designers to
  39. focus on their specific project, and not reinvent the wheel over and over
  40. again, pre-designed building blocks, IP cores, are often used. These are the
  41. FPGA parallels of library functions. IP cores may implement certain
  42. mathematical functions, a functional unit (e.g. a USB interface), an entire
  43. processor (e.g. ARM) or anything that might come handy. Think of them as a
  44. building block, with electrical wires dangling on the sides for connection to
  45. other blocks.
  46. One of the daunting tasks in FPGA design is communicating with a fullblown
  47. operating system (actually, with the processor running it): Implementing the
  48. low-level bus protocol and the somewhat higher-level interface with the host
  49. (registers, interrupts, DMA etc.) is a project in itself. When the FPGA's
  50. function is a well-known one (e.g. a video adapter card, or a NIC), it can
  51. make sense to design the FPGA's interface logic specifically for the project.
  52. A special driver is then written to present the FPGA as a well-known interface
  53. to the kernel and/or user space. In that case, there is no reason to treat the
  54. FPGA differently than any device on the bus.
  55. It's however common that the desired data communication doesn't fit any well-
  56. known peripheral function. Also, the effort of designing an elegant
  57. abstraction for the data exchange is often considered too big. In those cases,
  58. a quicker and possibly less elegant solution is sought: The driver is
  59. effectively written as a user space program, leaving the kernel space part
  60. with just elementary data transport. This still requires designing some
  61. interface logic for the FPGA, and write a simple ad-hoc driver for the kernel.
  62. Xillybus Overview
  63. -----------------
  64. Xillybus is an IP core and a Linux driver. Together, they form a kit for
  65. elementary data transport between an FPGA and the host, providing pipe-like
  66. data streams with a straightforward user interface. It's intended as a low-
  67. effort solution for mixed FPGA-host projects, for which it makes sense to
  68. have the project-specific part of the driver running in a user-space program.
  69. Since the communication requirements may vary significantly from one FPGA
  70. project to another (the number of data pipes needed in each direction and
  71. their attributes), there isn't one specific chunk of logic being the Xillybus
  72. IP core. Rather, the IP core is configured and built based upon a
  73. specification given by its end user.
  74. Xillybus presents independent data streams, which resemble pipes or TCP/IP
  75. communication to the user. At the host side, a character device file is used
  76. just like any pipe file. On the FPGA side, hardware FIFOs are used to stream
  77. the data. This is contrary to a common method of communicating through fixed-
  78. sized buffers (even though such buffers are used by Xillybus under the hood).
  79. There may be more than a hundred of these streams on a single IP core, but
  80. also no more than one, depending on the configuration.
  81. In order to ease the deployment of the Xillybus IP core, it contains a simple
  82. data structure which completely defines the core's configuration. The Linux
  83. driver fetches this data structure during its initialization process, and sets
  84. up the DMA buffers and character devices accordingly. As a result, a single
  85. driver is used to work out of the box with any Xillybus IP core.
  86. The data structure just mentioned should not be confused with PCI's
  87. configuration space or the Flattened Device Tree.
  88. Usage
  89. =====
  90. User interface
  91. --------------
  92. On the host, all interface with Xillybus is done through /dev/xillybus_*
  93. device files, which are generated automatically as the drivers loads. The
  94. names of these files depend on the IP core that is loaded in the FPGA (see
  95. Probing below). To communicate with the FPGA, open the device file that
  96. corresponds to the hardware FIFO you want to send data or receive data from,
  97. and use plain write() or read() calls, just like with a regular pipe. In
  98. particular, it makes perfect sense to go::
  99. $ cat mydata > /dev/xillybus_thisfifo
  100. $ cat /dev/xillybus_thatfifo > hisdata
  101. possibly pressing CTRL-C as some stage, even though the xillybus_* pipes have
  102. the capability to send an EOF (but may not use it).
  103. The driver and hardware are designed to behave sensibly as pipes, including:
  104. * Supporting non-blocking I/O (by setting O_NONBLOCK on open() ).
  105. * Supporting poll() and select().
  106. * Being bandwidth efficient under load (using DMA) but also handle small
  107. pieces of data sent across (like TCP/IP) by autoflushing.
  108. A device file can be read only, write only or bidirectional. Bidirectional
  109. device files are treated like two independent pipes (except for sharing a
  110. "channel" structure in the implementation code).
  111. Synchronization
  112. ---------------
  113. Xillybus pipes are configured (on the IP core) to be either synchronous or
  114. asynchronous. For a synchronous pipe, write() returns successfully only after
  115. some data has been submitted and acknowledged by the FPGA. This slows down
  116. bulk data transfers, and is nearly impossible for use with streams that
  117. require data at a constant rate: There is no data transmitted to the FPGA
  118. between write() calls, in particular when the process loses the CPU.
  119. When a pipe is configured asynchronous, write() returns if there was enough
  120. room in the buffers to store any of the data in the buffers.
  121. For FPGA to host pipes, asynchronous pipes allow data transfer from the FPGA
  122. as soon as the respective device file is opened, regardless of if the data
  123. has been requested by a read() call. On synchronous pipes, only the amount
  124. of data requested by a read() call is transmitted.
  125. In summary, for synchronous pipes, data between the host and FPGA is
  126. transmitted only to satisfy the read() or write() call currently handled
  127. by the driver, and those calls wait for the transmission to complete before
  128. returning.
  129. Note that the synchronization attribute has nothing to do with the possibility
  130. that read() or write() completes less bytes than requested. There is a
  131. separate configuration flag ("allowpartial") that determines whether such a
  132. partial completion is allowed.
  133. Seekable pipes
  134. --------------
  135. A synchronous pipe can be configured to have the stream's position exposed
  136. to the user logic at the FPGA. Such a pipe is also seekable on the host API.
  137. With this feature, a memory or register interface can be attached on the
  138. FPGA side to the seekable stream. Reading or writing to a certain address in
  139. the attached memory is done by seeking to the desired address, and calling
  140. read() or write() as required.
  141. Internals
  142. =========
  143. Source code organization
  144. ------------------------
  145. The Xillybus driver consists of a core module, xillybus_core.c, and modules
  146. that depend on the specific bus interface (xillybus_of.c and xillybus_pcie.c).
  147. The bus specific modules are those probed when a suitable device is found by
  148. the kernel. Since the DMA mapping and synchronization functions, which are bus
  149. dependent by their nature, are used by the core module, a
  150. xilly_endpoint_hardware structure is passed to the core module on
  151. initialization. This structure is populated with pointers to wrapper functions
  152. which execute the DMA-related operations on the bus.
  153. Pipe attributes
  154. ---------------
  155. Each pipe has a number of attributes which are set when the FPGA component
  156. (IP core) is built. They are fetched from the IDT (the data structure which
  157. defines the core's configuration, see Probing below) by xilly_setupchannels()
  158. in xillybus_core.c as follows:
  159. * is_writebuf: The pipe's direction. A non-zero value means it's an FPGA to
  160. host pipe (the FPGA "writes").
  161. * channelnum: The pipe's identification number in communication between the
  162. host and FPGA.
  163. * format: The underlying data width. See Data Granularity below.
  164. * allowpartial: A non-zero value means that a read() or write() (whichever
  165. applies) may return with less than the requested number of bytes. The common
  166. choice is a non-zero value, to match standard UNIX behavior.
  167. * synchronous: A non-zero value means that the pipe is synchronous. See
  168. Synchronization above.
  169. * bufsize: Each DMA buffer's size. Always a power of two.
  170. * bufnum: The number of buffers allocated for this pipe. Always a power of two.
  171. * exclusive_open: A non-zero value forces exclusive opening of the associated
  172. device file. If the device file is bidirectional, and already opened only in
  173. one direction, the opposite direction may be opened once.
  174. * seekable: A non-zero value indicates that the pipe is seekable. See
  175. Seekable pipes above.
  176. * supports_nonempty: A non-zero value (which is typical) indicates that the
  177. hardware will send the messages that are necessary to support select() and
  178. poll() for this pipe.
  179. Host never reads from the FPGA
  180. ------------------------------
  181. Even though PCI Express is hotpluggable in general, a typical motherboard
  182. doesn't expect a card to go away all of the sudden. But since the PCIe card
  183. is based upon reprogrammable logic, a sudden disappearance from the bus is
  184. quite likely as a result of an accidental reprogramming of the FPGA while the
  185. host is up. In practice, nothing happens immediately in such a situation. But
  186. if the host attempts to read from an address that is mapped to the PCI Express
  187. device, that leads to an immediate freeze of the system on some motherboards,
  188. even though the PCIe standard requires a graceful recovery.
  189. In order to avoid these freezes, the Xillybus driver refrains completely from
  190. reading from the device's register space. All communication from the FPGA to
  191. the host is done through DMA. In particular, the Interrupt Service Routine
  192. doesn't follow the common practice of checking a status register when it's
  193. invoked. Rather, the FPGA prepares a small buffer which contains short
  194. messages, which inform the host what the interrupt was about.
  195. This mechanism is used on non-PCIe buses as well for the sake of uniformity.
  196. Channels, pipes, and the message channel
  197. ----------------------------------------
  198. Each of the (possibly bidirectional) pipes presented to the user is allocated
  199. a data channel between the FPGA and the host. The distinction between channels
  200. and pipes is necessary only because of channel 0, which is used for interrupt-
  201. related messages from the FPGA, and has no pipe attached to it.
  202. Data streaming
  203. --------------
  204. Even though a non-segmented data stream is presented to the user at both
  205. sides, the implementation relies on a set of DMA buffers which is allocated
  206. for each channel. For the sake of illustration, let's take the FPGA to host
  207. direction: As data streams into the respective channel's interface in the
  208. FPGA, the Xillybus IP core writes it to one of the DMA buffers. When the
  209. buffer is full, the FPGA informs the host about that (appending a
  210. XILLYMSG_OPCODE_RELEASEBUF message channel 0 and sending an interrupt if
  211. necessary). The host responds by making the data available for reading through
  212. the character device. When all data has been read, the host writes on the
  213. FPGA's buffer control register, allowing the buffer's overwriting. Flow
  214. control mechanisms exist on both sides to prevent underflows and overflows.
  215. This is not good enough for creating a TCP/IP-like stream: If the data flow
  216. stops momentarily before a DMA buffer is filled, the intuitive expectation is
  217. that the partial data in buffer will arrive anyhow, despite the buffer not
  218. being completed. This is implemented by adding a field in the
  219. XILLYMSG_OPCODE_RELEASEBUF message, through which the FPGA informs not just
  220. which buffer is submitted, but how much data it contains.
  221. But the FPGA will submit a partially filled buffer only if directed to do so
  222. by the host. This situation occurs when the read() method has been blocking
  223. for XILLY_RX_TIMEOUT jiffies (currently 10 ms), after which the host commands
  224. the FPGA to submit a DMA buffer as soon as it can. This timeout mechanism
  225. balances between bus bandwidth efficiency (preventing a lot of partially
  226. filled buffers being sent) and a latency held fairly low for tails of data.
  227. A similar setting is used in the host to FPGA direction. The handling of
  228. partial DMA buffers is somewhat different, though. The user can tell the
  229. driver to submit all data it has in the buffers to the FPGA, by issuing a
  230. write() with the byte count set to zero. This is similar to a flush request,
  231. but it doesn't block. There is also an autoflushing mechanism, which triggers
  232. an equivalent flush roughly XILLY_RX_TIMEOUT jiffies after the last write().
  233. This allows the user to be oblivious about the underlying buffering mechanism
  234. and yet enjoy a stream-like interface.
  235. Note that the issue of partial buffer flushing is irrelevant for pipes having
  236. the "synchronous" attribute nonzero, since synchronous pipes don't allow data
  237. to lay around in the DMA buffers between read() and write() anyhow.
  238. Data granularity
  239. ----------------
  240. The data arrives or is sent at the FPGA as 8, 16 or 32 bit wide words, as
  241. configured by the "format" attribute. Whenever possible, the driver attempts
  242. to hide this when the pipe is accessed differently from its natural alignment.
  243. For example, reading single bytes from a pipe with 32 bit granularity works
  244. with no issues. Writing single bytes to pipes with 16 or 32 bit granularity
  245. will also work, but the driver can't send partially completed words to the
  246. FPGA, so the transmission of up to one word may be held until it's fully
  247. occupied with user data.
  248. This somewhat complicates the handling of host to FPGA streams, because
  249. when a buffer is flushed, it may contain up to 3 bytes don't form a word in
  250. the FPGA, and hence can't be sent. To prevent loss of data, these leftover
  251. bytes need to be moved to the next buffer. The parts in xillybus_core.c
  252. that mention "leftovers" in some way are related to this complication.
  253. Probing
  254. -------
  255. As mentioned earlier, the number of pipes that are created when the driver
  256. loads and their attributes depend on the Xillybus IP core in the FPGA. During
  257. the driver's initialization, a blob containing configuration info, the
  258. Interface Description Table (IDT), is sent from the FPGA to the host. The
  259. bootstrap process is done in three phases:
  260. 1. Acquire the length of the IDT, so a buffer can be allocated for it. This
  261. is done by sending a quiesce command to the device, since the acknowledge
  262. for this command contains the IDT's buffer length.
  263. 2. Acquire the IDT itself.
  264. 3. Create the interfaces according to the IDT.
  265. Buffer allocation
  266. -----------------
  267. In order to simplify the logic that prevents illegal boundary crossings of
  268. PCIe packets, the following rule applies: If a buffer is smaller than 4kB,
  269. it must not cross a 4kB boundary. Otherwise, it must be 4kB aligned. The
  270. xilly_setupchannels() functions allocates these buffers by requesting whole
  271. pages from the kernel, and diving them into DMA buffers as necessary. Since
  272. all buffers' sizes are powers of two, it's possible to pack any set of such
  273. buffers, with a maximal waste of one page of memory.
  274. All buffers are allocated when the driver is loaded. This is necessary,
  275. since large continuous physical memory segments are sometimes requested,
  276. which are more likely to be available when the system is freshly booted.
  277. The allocation of buffer memory takes place in the same order they appear in
  278. the IDT. The driver relies on a rule that the pipes are sorted with decreasing
  279. buffer size in the IDT. If a requested buffer is larger or equal to a page,
  280. the necessary number of pages is requested from the kernel, and these are
  281. used for this buffer. If the requested buffer is smaller than a page, one
  282. single page is requested from the kernel, and that page is partially used.
  283. Or, if there already is a partially used page at hand, the buffer is packed
  284. into that page. It can be shown that all pages requested from the kernel
  285. (except possibly for the last) are 100% utilized this way.
  286. The "nonempty" message (supporting poll)
  287. ----------------------------------------
  288. In order to support the "poll" method (and hence select() ), there is a small
  289. catch regarding the FPGA to host direction: The FPGA may have filled a DMA
  290. buffer with some data, but not submitted that buffer. If the host waited for
  291. the buffer's submission by the FPGA, there would be a possibility that the
  292. FPGA side has sent data, but a select() call would still block, because the
  293. host has not received any notification about this. This is solved with
  294. XILLYMSG_OPCODE_NONEMPTY messages sent by the FPGA when a channel goes from
  295. completely empty to containing some data.
  296. These messages are used only to support poll() and select(). The IP core can
  297. be configured not to send them for a slight reduction of bandwidth.