rapidio.rst 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362
  1. ============
  2. Introduction
  3. ============
  4. The RapidIO standard is a packet-based fabric interconnect standard designed for
  5. use in embedded systems. Development of the RapidIO standard is directed by the
  6. RapidIO Trade Association (RTA). The current version of the RapidIO specification
  7. is publicly available for download from the RTA web-site [1].
  8. This document describes the basics of the Linux RapidIO subsystem and provides
  9. information on its major components.
  10. 1 Overview
  11. ==========
  12. Because the RapidIO subsystem follows the Linux device model it is integrated
  13. into the kernel similarly to other buses by defining RapidIO-specific device and
  14. bus types and registering them within the device model.
  15. The Linux RapidIO subsystem is architecture independent and therefore defines
  16. architecture-specific interfaces that provide support for common RapidIO
  17. subsystem operations.
  18. 2. Core Components
  19. ==================
  20. A typical RapidIO network is a combination of endpoints and switches.
  21. Each of these components is represented in the subsystem by an associated data
  22. structure. The core logical components of the RapidIO subsystem are defined
  23. in include/linux/rio.h file.
  24. 2.1 Master Port
  25. ---------------
  26. A master port (or mport) is a RapidIO interface controller that is local to the
  27. processor executing the Linux code. A master port generates and receives RapidIO
  28. packets (transactions). In the RapidIO subsystem each master port is represented
  29. by a rio_mport data structure. This structure contains master port specific
  30. resources such as mailboxes and doorbells. The rio_mport also includes a unique
  31. host device ID that is valid when a master port is configured as an enumerating
  32. host.
  33. RapidIO master ports are serviced by subsystem specific mport device drivers
  34. that provide functionality defined for this subsystem. To provide a hardware
  35. independent interface for RapidIO subsystem operations, rio_mport structure
  36. includes rio_ops data structure which contains pointers to hardware specific
  37. implementations of RapidIO functions.
  38. 2.2 Device
  39. ----------
  40. A RapidIO device is any endpoint (other than mport) or switch in the network.
  41. All devices are presented in the RapidIO subsystem by corresponding rio_dev data
  42. structure. Devices form one global device list and per-network device lists
  43. (depending on number of available mports and networks).
  44. 2.3 Switch
  45. ----------
  46. A RapidIO switch is a special class of device that routes packets between its
  47. ports towards their final destination. The packet destination port within a
  48. switch is defined by an internal routing table. A switch is presented in the
  49. RapidIO subsystem by rio_dev data structure expanded by additional rio_switch
  50. data structure, which contains switch specific information such as copy of the
  51. routing table and pointers to switch specific functions.
  52. The RapidIO subsystem defines the format and initialization method for subsystem
  53. specific switch drivers that are designed to provide hardware-specific
  54. implementation of common switch management routines.
  55. 2.4 Network
  56. -----------
  57. A RapidIO network is a combination of interconnected endpoint and switch devices.
  58. Each RapidIO network known to the system is represented by corresponding rio_net
  59. data structure. This structure includes lists of all devices and local master
  60. ports that form the same network. It also contains a pointer to the default
  61. master port that is used to communicate with devices within the network.
  62. 2.5 Device Drivers
  63. ------------------
  64. RapidIO device-specific drivers follow Linux Kernel Driver Model and are
  65. intended to support specific RapidIO devices attached to the RapidIO network.
  66. 2.6 Subsystem Interfaces
  67. ------------------------
  68. RapidIO interconnect specification defines features that may be used to provide
  69. one or more common service layers for all participating RapidIO devices. These
  70. common services may act separately from device-specific drivers or be used by
  71. device-specific drivers. Example of such service provider is the RIONET driver
  72. which implements Ethernet-over-RapidIO interface. Because only one driver can be
  73. registered for a device, all common RapidIO services have to be registered as
  74. subsystem interfaces. This allows to have multiple common services attached to
  75. the same device without blocking attachment of a device-specific driver.
  76. 3. Subsystem Initialization
  77. ===========================
  78. In order to initialize the RapidIO subsystem, a platform must initialize and
  79. register at least one master port within the RapidIO network. To register mport
  80. within the subsystem controller driver's initialization code calls function
  81. rio_register_mport() for each available master port.
  82. After all active master ports are registered with a RapidIO subsystem,
  83. an enumeration and/or discovery routine may be called automatically or
  84. by user-space command.
  85. RapidIO subsystem can be configured to be built as a statically linked or
  86. modular component of the kernel (see details below).
  87. 4. Enumeration and Discovery
  88. ============================
  89. 4.1 Overview
  90. ------------
  91. RapidIO subsystem configuration options allow users to build enumeration and
  92. discovery methods as statically linked components or loadable modules.
  93. An enumeration/discovery method implementation and available input parameters
  94. define how any given method can be attached to available RapidIO mports:
  95. simply to all available mports OR individually to the specified mport device.
  96. Depending on selected enumeration/discovery build configuration, there are
  97. several methods to initiate an enumeration and/or discovery process:
  98. (a) Statically linked enumeration and discovery process can be started
  99. automatically during kernel initialization time using corresponding module
  100. parameters. This was the original method used since introduction of RapidIO
  101. subsystem. Now this method relies on enumerator module parameter which is
  102. 'rio-scan.scan' for existing basic enumeration/discovery method.
  103. When automatic start of enumeration/discovery is used a user has to ensure
  104. that all discovering endpoints are started before the enumerating endpoint
  105. and are waiting for enumeration to be completed.
  106. Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering
  107. endpoint waits for enumeration to be completed. If the specified timeout
  108. expires the discovery process is terminated without obtaining RapidIO network
  109. information. NOTE: a timed out discovery process may be restarted later using
  110. a user-space command as it is described below (if the given endpoint was
  111. enumerated successfully).
  112. (b) Statically linked enumeration and discovery process can be started by
  113. a command from user space. This initiation method provides more flexibility
  114. for a system startup compared to the option (a) above. After all participating
  115. endpoints have been successfully booted, an enumeration process shall be
  116. started first by issuing a user-space command, after an enumeration is
  117. completed a discovery process can be started on all remaining endpoints.
  118. (c) Modular enumeration and discovery process can be started by a command from
  119. user space. After an enumeration/discovery module is loaded, a network scan
  120. process can be started by issuing a user-space command.
  121. Similar to the option (b) above, an enumerator has to be started first.
  122. (d) Modular enumeration and discovery process can be started by a module
  123. initialization routine. In this case an enumerating module shall be loaded
  124. first.
  125. When a network scan process is started it calls an enumeration or discovery
  126. routine depending on the configured role of a master port: host or agent.
  127. Enumeration is performed by a master port if it is configured as a host port by
  128. assigning a host destination ID greater than or equal to zero. The host
  129. destination ID can be assigned to a master port using various methods depending
  130. on RapidIO subsystem build configuration:
  131. (a) For a statically linked RapidIO subsystem core use command line parameter
  132. "rapidio.hdid=" with a list of destination ID assignments in order of mport
  133. device registration. For example, in a system with two RapidIO controllers
  134. the command line parameter "rapidio.hdid=-1,7" will result in assignment of
  135. the host destination ID=7 to the second RapidIO controller, while the first
  136. one will be assigned destination ID=-1.
  137. (b) If the RapidIO subsystem core is built as a loadable module, in addition
  138. to the method shown above, the host destination ID(s) can be specified using
  139. traditional methods of passing module parameter "hdid=" during its loading:
  140. - from command line: "modprobe rapidio hdid=-1,7", or
  141. - from modprobe configuration file using configuration command "options",
  142. like in this example: "options rapidio hdid=-1,7". An example of modprobe
  143. configuration file is provided in the section below.
  144. NOTES:
  145. (i) if "hdid=" parameter is omitted all available mport will be assigned
  146. destination ID = -1;
  147. (ii) the "hdid=" parameter in systems with multiple mports can have
  148. destination ID assignments omitted from the end of list (default = -1).
  149. If the host device ID for a specific master port is set to -1, the discovery
  150. process will be performed for it.
  151. The enumeration and discovery routines use RapidIO maintenance transactions
  152. to access the configuration space of devices.
  153. NOTE: If RapidIO switch-specific device drivers are built as loadable modules
  154. they must be loaded before enumeration/discovery process starts.
  155. This requirement is cased by the fact that enumeration/discovery methods invoke
  156. vendor-specific callbacks on early stages.
  157. 4.2 Automatic Start of Enumeration and Discovery
  158. ------------------------------------------------
  159. Automatic enumeration/discovery start method is applicable only to built-in
  160. enumeration/discovery RapidIO configuration selection. To enable automatic
  161. enumeration/discovery start by existing basic enumerator method set use boot
  162. command line parameter "rio-scan.scan=1".
  163. This configuration requires synchronized start of all RapidIO endpoints that
  164. form a network which will be enumerated/discovered. Discovering endpoints have
  165. to be started before an enumeration starts to ensure that all RapidIO
  166. controllers have been initialized and are ready to be discovered. Configuration
  167. parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which
  168. a discovering endpoint will wait for enumeration to be completed.
  169. When automatic enumeration/discovery start is selected, basic method's
  170. initialization routine calls rio_init_mports() to perform enumeration or
  171. discovery for all known mport devices.
  172. Depending on RapidIO network size and configuration this automatic
  173. enumeration/discovery start method may be difficult to use due to the
  174. requirement for synchronized start of all endpoints.
  175. 4.3 User-space Start of Enumeration and Discovery
  176. -------------------------------------------------
  177. User-space start of enumeration and discovery can be used with built-in and
  178. modular build configurations. For user-space controlled start RapidIO subsystem
  179. creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate
  180. an enumeration or discovery process on specific mport device, a user needs to
  181. write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a
  182. sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device
  183. registration. For example for machine with single RapidIO controller, mport_ID
  184. for that controller always will be 0.
  185. To initiate RapidIO enumeration/discovery on all available mports a user may
  186. write '-1' (or RIO_MPORT_ANY) into the scan attribute file.
  187. 4.4 Basic Enumeration Method
  188. ----------------------------
  189. This is an original enumeration/discovery method which is available since
  190. first release of RapidIO subsystem code. The enumeration process is
  191. implemented according to the enumeration algorithm outlined in the RapidIO
  192. Interconnect Specification: Annex I [1].
  193. This method can be configured as statically linked or loadable module.
  194. The method's single parameter "scan" allows to trigger the enumeration/discovery
  195. process from module initialization routine.
  196. This enumeration/discovery method can be started only once and does not support
  197. unloading if it is built as a module.
  198. The enumeration process traverses the network using a recursive depth-first
  199. algorithm. When a new device is found, the enumerator takes ownership of that
  200. device by writing into the Host Device ID Lock CSR. It does this to ensure that
  201. the enumerator has exclusive right to enumerate the device. If device ownership
  202. is successfully acquired, the enumerator allocates a new rio_dev structure and
  203. initializes it according to device capabilities.
  204. If the device is an endpoint, a unique device ID is assigned to it and its value
  205. is written into the device's Base Device ID CSR.
  206. If the device is a switch, the enumerator allocates an additional rio_switch
  207. structure to store switch specific information. Then the switch's vendor ID and
  208. device ID are queried against a table of known RapidIO switches. Each switch
  209. table entry contains a pointer to a switch-specific initialization routine that
  210. initializes pointers to the rest of switch specific operations, and performs
  211. hardware initialization if necessary. A RapidIO switch does not have a unique
  212. device ID; it relies on hopcount and routing for device ID of an attached
  213. endpoint if access to its configuration registers is required. If a switch (or
  214. chain of switches) does not have any endpoint (except enumerator) attached to
  215. it, a fake device ID will be assigned to configure a route to that switch.
  216. In the case of a chain of switches without endpoint, one fake device ID is used
  217. to configure a route through the entire chain and switches are differentiated by
  218. their hopcount value.
  219. For both endpoints and switches the enumerator writes a unique component tag
  220. into device's Component Tag CSR. That unique value is used by the error
  221. management notification mechanism to identify a device that is reporting an
  222. error management event.
  223. Enumeration beyond a switch is completed by iterating over each active egress
  224. port of that switch. For each active link, a route to a default device ID
  225. (0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written
  226. into the routing table. The algorithm recurs by calling itself with hopcount + 1
  227. and the default device ID in order to access the device on the active port.
  228. After the host has completed enumeration of the entire network it releases
  229. devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint
  230. in the system, it sets the Discovered bit in the Port General Control CSR
  231. to indicate that enumeration is completed and agents are allowed to execute
  232. passive discovery of the network.
  233. The discovery process is performed by agents and is similar to the enumeration
  234. process that is described above. However, the discovery process is performed
  235. without changes to the existing routing because agents only gather information
  236. about RapidIO network structure and are building an internal map of discovered
  237. devices. This way each Linux-based component of the RapidIO subsystem has
  238. a complete view of the network. The discovery process can be performed
  239. simultaneously by several agents. After initializing its RapidIO master port
  240. each agent waits for enumeration completion by the host for the configured wait
  241. time period. If this wait time period expires before enumeration is completed,
  242. an agent skips RapidIO discovery and continues with remaining kernel
  243. initialization.
  244. 4.5 Adding New Enumeration/Discovery Method
  245. -------------------------------------------
  246. RapidIO subsystem code organization allows addition of new enumeration/discovery
  247. methods as new configuration options without significant impact to the core
  248. RapidIO code.
  249. A new enumeration/discovery method has to be attached to one or more mport
  250. devices before an enumeration/discovery process can be started. Normally,
  251. method's module initialization routine calls rio_register_scan() to attach
  252. an enumerator to a specified mport device (or devices). The basic enumerator
  253. implementation demonstrates this process.
  254. 4.6 Using Loadable RapidIO Switch Drivers
  255. -----------------------------------------
  256. In the case when RapidIO switch drivers are built as loadable modules a user
  257. must ensure that they are loaded before the enumeration/discovery starts.
  258. This process can be automated by specifying pre- or post- dependencies in the
  259. RapidIO-specific modprobe configuration file as shown in the example below.
  260. File /etc/modprobe.d/rapidio.conf::
  261. # Configure RapidIO subsystem modules
  262. # Set enumerator host destination ID (overrides kernel command line option)
  263. options rapidio hdid=-1,2
  264. # Load RapidIO switch drivers immediately after rapidio core module was loaded
  265. softdep rapidio post: idt_gen2 idtcps tsi57x
  266. # OR :
  267. # Load RapidIO switch drivers just before rio-scan enumerator module is loaded
  268. softdep rio-scan pre: idt_gen2 idtcps tsi57x
  269. --------------------------
  270. NOTE:
  271. In the example above, one of "softdep" commands must be removed or
  272. commented out to keep required module loading sequence.
  273. 5. References
  274. =============
  275. [1] RapidIO Trade Association. RapidIO Interconnect Specifications.
  276. http://www.rapidio.org.
  277. [2] Rapidio TA. Technology Comparisons.
  278. http://www.rapidio.org/education/technology_comparisons/
  279. [3] RapidIO support for Linux.
  280. https://lwn.net/Articles/139118/
  281. [4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005
  282. https://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf