dfl.rst 22 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509
  1. =================================================
  2. FPGA Device Feature List (DFL) Framework Overview
  3. =================================================
  4. Authors:
  5. - Enno Luebbers <enno.luebbers@intel.com>
  6. - Xiao Guangrong <guangrong.xiao@linux.intel.com>
  7. - Wu Hao <hao.wu@intel.com>
  8. The Device Feature List (DFL) FPGA framework (and drivers according to
  9. this framework) hides the very details of low layer hardwares and provides
  10. unified interfaces to userspace. Applications could use these interfaces to
  11. configure, enumerate, open and access FPGA accelerators on platforms which
  12. implement the DFL in the device memory. Besides this, the DFL framework
  13. enables system level management functions such as FPGA reconfiguration.
  14. Device Feature List (DFL) Overview
  15. ==================================
  16. Device Feature List (DFL) defines a linked list of feature headers within the
  17. device MMIO space to provide an extensible way of adding features. Software can
  18. walk through these predefined data structures to enumerate FPGA features:
  19. FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
  20. as illustrated below::
  21. Header Header Header Header
  22. +----------+ +-->+----------+ +-->+----------+ +-->+----------+
  23. | Type | | | Type | | | Type | | | Type |
  24. | FIU | | | Private | | | Private | | | Private |
  25. +----------+ | | Feature | | | Feature | | | Feature |
  26. | Next_DFH |--+ +----------+ | +----------+ | +----------+
  27. +----------+ | Next_DFH |--+ | Next_DFH |--+ | Next_DFH |--> NULL
  28. | ID | +----------+ +----------+ +----------+
  29. +----------+ | ID | | ID | | ID |
  30. | Next_AFU |--+ +----------+ +----------+ +----------+
  31. +----------+ | | Feature | | Feature | | Feature |
  32. | Header | | | Register | | Register | | Register |
  33. | Register | | | Set | | Set | | Set |
  34. | Set | | +----------+ +----------+ +----------+
  35. +----------+ | Header
  36. +-->+----------+
  37. | Type |
  38. | AFU |
  39. +----------+
  40. | Next_DFH |--> NULL
  41. +----------+
  42. | GUID |
  43. +----------+
  44. | Header |
  45. | Register |
  46. | Set |
  47. +----------+
  48. FPGA Interface Unit (FIU) represents a standalone functional unit for the
  49. interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
  50. descriptions on FME and Port in later sections).
  51. Accelerated Function Unit (AFU) represents a FPGA programmable region and
  52. always connects to a FIU (e.g. a Port) as its child as illustrated above.
  53. Private Features represent sub features of the FIU and AFU. They could be
  54. various function blocks with different IDs, but all private features which
  55. belong to the same FIU or AFU, must be linked to one list via the Next Device
  56. Feature Header (Next_DFH) pointer.
  57. Each FIU, AFU and Private Feature could implement its own functional registers.
  58. The functional register set for FIU and AFU, is named as Header Register Set,
  59. e.g. FME Header Register Set, and the one for Private Feature, is named as
  60. Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
  61. This Device Feature List provides a way of linking features together, it's
  62. convenient for software to locate each feature by walking through this list,
  63. and can be implemented in register regions of any FPGA device.
  64. FIU - FME (FPGA Management Engine)
  65. ==================================
  66. The FPGA Management Engine performs reconfiguration and other infrastructure
  67. functions. Each FPGA device only has one FME.
  68. User-space applications can acquire exclusive access to the FME using open(),
  69. and release it using close().
  70. The following functions are exposed through ioctls:
  71. - Get driver API version (DFL_FPGA_GET_API_VERSION)
  72. - Check for extensions (DFL_FPGA_CHECK_EXTENSION)
  73. - Program bitstream (DFL_FPGA_FME_PORT_PR)
  74. - Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
  75. - Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
  76. - Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM)
  77. - Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ)
  78. More functions are exposed through sysfs
  79. (/sys/class/fpga_region/regionX/dfl-fme.n/):
  80. Read bitstream ID (bitstream_id)
  81. bitstream_id indicates version of the static FPGA region.
  82. Read bitstream metadata (bitstream_metadata)
  83. bitstream_metadata includes detailed information of static FPGA region,
  84. e.g. synthesis date and seed.
  85. Read number of ports (ports_num)
  86. one FPGA device may have more than one port, this sysfs interface indicates
  87. how many ports the FPGA device has.
  88. Global error reporting management (errors/)
  89. error reporting sysfs interfaces allow user to read errors detected by the
  90. hardware, and clear the logged errors.
  91. Power management (dfl_fme_power hwmon)
  92. power management hwmon sysfs interfaces allow user to read power management
  93. information (power consumption, thresholds, threshold status, limits, etc.)
  94. and configure power thresholds for different throttling levels.
  95. Thermal management (dfl_fme_thermal hwmon)
  96. thermal management hwmon sysfs interfaces allow user to read thermal
  97. management information (current temperature, thresholds, threshold status,
  98. etc.).
  99. Performance reporting
  100. performance counters are exposed through perf PMU APIs. Standard perf tool
  101. can be used to monitor all available perf events. Please see performance
  102. counter section below for more detailed information.
  103. FIU - PORT
  104. ==========
  105. A port represents the interface between the static FPGA fabric and a partially
  106. reconfigurable region containing an AFU. It controls the communication from SW
  107. to the accelerator and exposes features such as reset and debug. Each FPGA
  108. device may have more than one port, but always one AFU per port.
  109. AFU
  110. ===
  111. An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
  112. used for accelerator-specific control registers.
  113. User-space applications can acquire exclusive access to an AFU attached to a
  114. port by using open() on the port device node and release it using close().
  115. The following functions are exposed through ioctls:
  116. - Get driver API version (DFL_FPGA_GET_API_VERSION)
  117. - Check for extensions (DFL_FPGA_CHECK_EXTENSION)
  118. - Get port info (DFL_FPGA_PORT_GET_INFO)
  119. - Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
  120. - Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
  121. - Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
  122. - Reset AFU (DFL_FPGA_PORT_RESET)
  123. - Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM)
  124. - Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ)
  125. - Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM)
  126. - Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ)
  127. DFL_FPGA_PORT_RESET:
  128. reset the FPGA Port and its AFU. Userspace can do Port
  129. reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
  130. never cause any system level issue, only functional failure (e.g. DMA or PR
  131. operation failure) and be recoverable from the failure.
  132. User-space applications can also mmap() accelerator MMIO regions.
  133. More functions are exposed through sysfs:
  134. (/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
  135. Read Accelerator GUID (afu_id)
  136. afu_id indicates which PR bitstream is programmed to this AFU.
  137. Error reporting (errors/)
  138. error reporting sysfs interfaces allow user to read port/afu errors
  139. detected by the hardware, and clear the logged errors.
  140. DFL Framework Overview
  141. ======================
  142. ::
  143. +----------+ +--------+ +--------+ +--------+
  144. | FME | | AFU | | AFU | | AFU |
  145. | Module | | Module | | Module | | Module |
  146. +----------+ +--------+ +--------+ +--------+
  147. +-----------------------+
  148. | FPGA Container Device | Device Feature List
  149. | (FPGA Base Region) | Framework
  150. +-----------------------+
  151. ------------------------------------------------------------------
  152. +----------------------------+
  153. | FPGA DFL Device Module |
  154. | (e.g. PCIE/Platform Device)|
  155. +----------------------------+
  156. +------------------------+
  157. | FPGA Hardware Device |
  158. +------------------------+
  159. DFL framework in kernel provides common interfaces to create container device
  160. (FPGA base region), discover feature devices and their private features from the
  161. given Device Feature Lists and create platform devices for feature devices
  162. (e.g. FME, Port and AFU) with related resources under the container device. It
  163. also abstracts operations for the private features and exposes common ops to
  164. feature device drivers.
  165. The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
  166. device and etc. Its driver module is always loaded first once the device is
  167. created by the system. This driver plays an infrastructural role in the
  168. driver architecture. It locates the DFLs in the device memory, handles them
  169. and related resources to common interfaces from DFL framework for enumeration.
  170. (Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
  171. The FPGA Management Engine (FME) driver is a platform driver which is loaded
  172. automatically after FME platform device creation from the DFL device module. It
  173. provides the key features for FPGA management, including:
  174. a) Expose static FPGA region information, e.g. version and metadata.
  175. Users can read related information via sysfs interfaces exposed
  176. by FME driver.
  177. b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
  178. bridges and FPGA regions during PR sub feature initialization. Once
  179. it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
  180. common interface function from FPGA Region to complete the partial
  181. reconfiguration of the PR bitstream to the given port.
  182. Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
  183. probed once the AFU platform device is created. The main function of this module
  184. is to provide an interface for userspace applications to access the individual
  185. accelerators, including basic reset control on port, AFU MMIO region export, dma
  186. buffer mapping service functions.
  187. After feature platform devices creation, matched platform drivers will be loaded
  188. automatically to handle different functionalities. Please refer to next sections
  189. for detailed information on functional units which have been already implemented
  190. under this DFL framework.
  191. Partial Reconfiguration
  192. =======================
  193. As mentioned above, accelerators can be reconfigured through partial
  194. reconfiguration of a PR bitstream file. The PR bitstream file must have been
  195. generated for the exact static FPGA region and targeted reconfigurable region
  196. (port) of the FPGA, otherwise, the reconfiguration operation will fail and
  197. possibly cause system instability. This compatibility can be checked by
  198. comparing the compatibility ID noted in the header of PR bitstream file against
  199. the compat_id exposed by the target FPGA region. This check is usually done by
  200. userspace before calling the reconfiguration IOCTL.
  201. FPGA virtualization - PCIe SRIOV
  202. ================================
  203. This section describes the virtualization support on DFL based FPGA device to
  204. enable accessing an accelerator from applications running in a virtual machine
  205. (VM). This section only describes the PCIe based FPGA device with SRIOV support.
  206. Features supported by the particular FPGA device are exposed through Device
  207. Feature Lists, as illustrated below:
  208. ::
  209. +-------------------------------+ +-------------+
  210. | PF | | VF |
  211. +-------------------------------+ +-------------+
  212. ^ ^ ^ ^
  213. | | | |
  214. +-----|------------|---------|--------------|-------+
  215. | | | | | |
  216. | +-----+ +-------+ +-------+ +-------+ |
  217. | | FME | | Port0 | | Port1 | | Port2 | |
  218. | +-----+ +-------+ +-------+ +-------+ |
  219. | ^ ^ ^ |
  220. | | | | |
  221. | +-------+ +------+ +-------+ |
  222. | | AFU | | AFU | | AFU | |
  223. | +-------+ +------+ +-------+ |
  224. | |
  225. | DFL based FPGA PCIe Device |
  226. +---------------------------------------------------+
  227. FME is always accessed through the physical function (PF).
  228. Ports (and related AFUs) are accessed via PF by default, but could be exposed
  229. through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
  230. 1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
  231. created via PCIe SRIOV interface, to virtual machines.
  232. The driver organization in virtualization case is illustrated below:
  233. ::
  234. +-------++------++------+ |
  235. | FME || FME || FME | |
  236. | FPGA || FPGA || FPGA | |
  237. |Manager||Bridge||Region| |
  238. +-------++------++------+ |
  239. +-----------------------+ +--------+ | +--------+
  240. | FME | | AFU | | | AFU |
  241. | Module | | Module | | | Module |
  242. +-----------------------+ +--------+ | +--------+
  243. +-----------------------+ | +-----------------------+
  244. | FPGA Container Device | | | FPGA Container Device |
  245. | (FPGA Base Region) | | | (FPGA Base Region) |
  246. +-----------------------+ | +-----------------------+
  247. +------------------+ | +------------------+
  248. | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
  249. +------------------+ Host | Machine +------------------+
  250. -------------------------------------- | ------------------------------
  251. +---------------+ | +---------------+
  252. | PCI PF Device | | | PCI VF Device |
  253. +---------------+ | +---------------+
  254. FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
  255. is detected. It:
  256. * Finishes enumeration on both FPGA PCIe PF and VF device using common
  257. interfaces from DFL framework.
  258. * Supports SRIOV.
  259. The FME device driver plays a management role in this driver architecture, it
  260. provides ioctls to release Port from PF and assign Port to PF. After release
  261. a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
  262. sysfs interface.
  263. To enable accessing an accelerator from applications running in a VM, the
  264. respective AFU's port needs to be assigned to a VF using the following steps:
  265. #. The PF owns all AFU ports by default. Any port that needs to be
  266. reassigned to a VF must first be released through the
  267. DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
  268. #. Once N ports are released from PF, then user can use command below
  269. to enable SRIOV and VFs. Each VF owns only one Port with AFU.
  270. ::
  271. echo N > $PCI_DEVICE_PATH/sriov_numvfs
  272. #. Pass through the VFs to VMs
  273. #. The AFU under VF is accessible from applications in VM (using the
  274. same driver inside the VF).
  275. Note that an FME can't be assigned to a VF, thus PR and other management
  276. functions are only available via the PF.
  277. Device enumeration
  278. ==================
  279. This section introduces how applications enumerate the fpga device from
  280. the sysfs hierarchy under /sys/class/fpga_region.
  281. In the example below, two DFL based FPGA devices are installed in the host. Each
  282. fpga device has one FME and two ports (AFUs).
  283. FPGA regions are created under /sys/class/fpga_region/::
  284. /sys/class/fpga_region/region0
  285. /sys/class/fpga_region/region1
  286. /sys/class/fpga_region/region2
  287. ...
  288. Application needs to search each regionX folder, if feature device is found,
  289. (e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
  290. fpga region which represents the FPGA device.
  291. Each base region has one FME and two ports (AFUs) as child devices::
  292. /sys/class/fpga_region/region0/dfl-fme.0
  293. /sys/class/fpga_region/region0/dfl-port.0
  294. /sys/class/fpga_region/region0/dfl-port.1
  295. ...
  296. /sys/class/fpga_region/region3/dfl-fme.1
  297. /sys/class/fpga_region/region3/dfl-port.2
  298. /sys/class/fpga_region/region3/dfl-port.3
  299. ...
  300. In general, the FME/AFU sysfs interfaces are named as follows::
  301. /sys/class/fpga_region/<regionX>/<dfl-fme.n>/
  302. /sys/class/fpga_region/<regionX>/<dfl-port.m>/
  303. with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
  304. ports.
  305. The device nodes used for ioctl() or mmap() can be referenced through::
  306. /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
  307. /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
  308. Performance Counters
  309. ====================
  310. Performance reporting is one private feature implemented in FME. It could
  311. supports several independent, system-wide, device counter sets in hardware to
  312. monitor and count for performance events, including "basic", "cache", "fabric",
  313. "vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor
  314. FPGA cache hit/miss rate, transaction number, interface clock counter of AFU
  315. and other FPGA performance events.
  316. Different FPGA devices may have different counter sets, depending on hardware
  317. implementation. E.g., some discrete FPGA cards don't have any cache. User could
  318. use "perf list" to check which perf events are supported by target hardware.
  319. In order to allow user to use standard perf API to access these performance
  320. counters, driver creates a perf PMU, and related sysfs interfaces in
  321. /sys/bus/event_source/devices/dfl_fme* to describe available perf events and
  322. configuration options.
  323. The "format" directory describes the format of the config field of struct
  324. perf_event_attr. There are 3 bitfields for config: "evtype" defines which type
  325. the perf event belongs to; "event" is the identity of the event within its
  326. category; "portid" is introduced to decide counters set to monitor on FPGA
  327. overall data or a specific port.
  328. The "events" directory describes the configuration templates for all available
  329. events which can be used with perf tool directly. For example, fab_mmio_read
  330. has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this
  331. event belongs to fabric type (0x02), the local event id is 0x06 and it is for
  332. overall monitoring (portid=0xff).
  333. Example usage of perf::
  334. $# perf list |grep dfl_fme
  335. dfl_fme0/fab_mmio_read/ [Kernel PMU event]
  336. <...>
  337. dfl_fme0/fab_port_mmio_read,portid=?/ [Kernel PMU event]
  338. <...>
  339. $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command>
  340. or
  341. $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command>
  342. or
  343. $# perf stat -a -e dfl_fme0/config=0xff2006/ <command>
  344. Another example, fab_port_mmio_read monitors mmio read of a specific port. So
  345. its configuration template is "event=0x06,evtype=0x01,portid=?". The portid
  346. should be explicitly set.
  347. Its usage of perf::
  348. $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command>
  349. or
  350. $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command>
  351. or
  352. $# perf stat -a -e dfl_fme0/config=0x2006/ <command>
  353. Please note for fabric counters, overall perf events (fab_*) and port perf
  354. events (fab_port_*) actually share one set of counters in hardware, so it can't
  355. monitor both at the same time. If this set of counters is configured to monitor
  356. overall data, then per port perf data is not supported. See below example::
  357. $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\
  358. portid=0/ sleep 1
  359. Performance counter stats for 'system wide':
  360. 3 dfl_fme0/fab_mmio_read/
  361. <not supported> dfl_fme0/fab_port_mmio_write,portid=0x0/
  362. 1.001750904 seconds time elapsed
  363. The driver also provides a "cpumask" sysfs attribute, which contains only one
  364. CPU id used to access these perf events. Counting on multiple CPU is not allowed
  365. since they are system-wide counters on FPGA device.
  366. The current driver does not support sampling. So "perf record" is unsupported.
  367. Interrupt support
  368. =================
  369. Some FME and AFU private features are able to generate interrupts. As mentioned
  370. above, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how
  371. many interrupts are supported for this private feature. Drivers also implement
  372. an eventfd based interrupt handling mechanism for users to get notified when
  373. interrupt happens. Users could set eventfds to driver via
  374. ioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for
  375. notification.
  376. In Current DFL, 3 sub features (Port error, FME global error and AFU interrupt)
  377. support interrupts.
  378. Add new FIUs support
  379. ====================
  380. It's possible that developers made some new function blocks (FIUs) under this
  381. DFL framework, then new platform device driver needs to be developed for the
  382. new feature dev (FIU) following the same way as existing feature dev drivers
  383. (e.g. FME and Port/AFU platform device driver). Besides that, it requires
  384. modification on DFL framework enumeration code too, for new FIU type detection
  385. and related platform devices creation.
  386. Add new private features support
  387. ================================
  388. In some cases, we may need to add some new private features to existing FIUs
  389. (e.g. FME or Port). Developers don't need to touch enumeration code in DFL
  390. framework, as each private feature will be parsed automatically and related
  391. mmio resources can be found under FIU platform device created by DFL framework.
  392. Developer only needs to provide a sub feature driver with matched feature id.
  393. FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
  394. could be a reference.
  395. Open discussion
  396. ===============
  397. FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
  398. to user now. In the future, if unified user interfaces for reconfiguration are
  399. added, FME driver should switch to them from ioctl interface.