libsas.txt 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484
  1. SAS Layer
  2. ---------
  3. The SAS Layer is a management infrastructure which manages
  4. SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The
  5. layout is as follows: while SCSI Core is concerned with
  6. SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
  7. phy/OOB/link management, the SAS layer is concerned with:
  8. * SAS Phy/Port/HA event management (LLDD generates,
  9. SAS Layer processes),
  10. * SAS Port management (creation/destruction),
  11. * SAS Domain discovery and revalidation,
  12. * SAS Domain device management,
  13. * SCSI Host registration/unregistration,
  14. * Device registration with SCSI Core (SAS) or libata
  15. (SATA), and
  16. * Expander management and exporting expander control
  17. to user space.
  18. A SAS LLDD is a PCI device driver. It is concerned with
  19. phy/OOB management, and vendor specific tasks and generates
  20. events to the SAS layer.
  21. The SAS Layer does most SAS tasks as outlined in the SAS 1.1
  22. spec.
  23. The sas_ha_struct describes the SAS LLDD to the SAS layer.
  24. Most of it is used by the SAS Layer but a few fields need to
  25. be initialized by the LLDDs.
  26. After initializing your hardware, from the probe() function
  27. you call sas_register_ha(). It will register your LLDD with
  28. the SCSI subsystem, creating a SCSI host and it will
  29. register your SAS driver with the sysfs SAS tree it creates.
  30. It will then return. Then you enable your phys to actually
  31. start OOB (at which point your driver will start calling the
  32. notify_* event callbacks).
  33. Structure descriptions:
  34. struct sas_phy --------------------
  35. Normally this is statically embedded to your driver's
  36. phy structure:
  37. struct my_phy {
  38. blah;
  39. struct sas_phy sas_phy;
  40. bleh;
  41. };
  42. And then all the phys are an array of my_phy in your HA
  43. struct (shown below).
  44. Then as you go along and initialize your phys you also
  45. initialize the sas_phy struct, along with your own
  46. phy structure.
  47. In general, the phys are managed by the LLDD and the ports
  48. are managed by the SAS layer. So the phys are initialized
  49. and updated by the LLDD and the ports are initialized and
  50. updated by the SAS layer.
  51. There is a scheme where the LLDD can RW certain fields,
  52. and the SAS layer can only read such ones, and vice versa.
  53. The idea is to avoid unnecessary locking.
  54. enabled -- must be set (0/1)
  55. id -- must be set [0,MAX_PHYS)
  56. class, proto, type, role, oob_mode, linkrate -- must be set
  57. oob_mode -- you set this when OOB has finished and then notify
  58. the SAS Layer.
  59. sas_addr -- this normally points to an array holding the sas
  60. address of the phy, possibly somewhere in your my_phy
  61. struct.
  62. attached_sas_addr -- set this when you (LLDD) receive an
  63. IDENTIFY frame or a FIS frame, _before_ notifying the SAS
  64. layer. The idea is that sometimes the LLDD may want to fake
  65. or provide a different SAS address on that phy/port and this
  66. allows it to do this. At best you should copy the sas
  67. address from the IDENTIFY frame or maybe generate a SAS
  68. address for SATA directly attached devices. The Discover
  69. process may later change this.
  70. frame_rcvd -- this is where you copy the IDENTIFY/FIS frame
  71. when you get it; you lock, copy, set frame_rcvd_size and
  72. unlock the lock, and then call the event. It is a pointer
  73. since there's no way to know your hw frame size _exactly_,
  74. so you define the actual array in your phy struct and let
  75. this pointer point to it. You copy the frame from your
  76. DMAable memory to that area holding the lock.
  77. sas_prim -- this is where primitives go when they're
  78. received. See sas.h. Grab the lock, set the primitive,
  79. release the lock, notify.
  80. port -- this points to the sas_port if the phy belongs
  81. to a port -- the LLDD only reads this. It points to the
  82. sas_port this phy is part of. Set by the SAS Layer.
  83. ha -- may be set; the SAS layer sets it anyway.
  84. lldd_phy -- you should set this to point to your phy so you
  85. can find your way around faster when the SAS layer calls one
  86. of your callbacks and passes you a phy. If the sas_phy is
  87. embedded you can also use container_of -- whatever you
  88. prefer.
  89. struct sas_port --------------------
  90. The LLDD doesn't set any fields of this struct -- it only
  91. reads them. They should be self explanatory.
  92. phy_mask is 32 bit, this should be enough for now, as I
  93. haven't heard of a HA having more than 8 phys.
  94. lldd_port -- I haven't found use for that -- maybe other
  95. LLDD who wish to have internal port representation can make
  96. use of this.
  97. struct sas_ha_struct --------------------
  98. It normally is statically declared in your own LLDD
  99. structure describing your adapter:
  100. struct my_sas_ha {
  101. blah;
  102. struct sas_ha_struct sas_ha;
  103. struct my_phy phys[MAX_PHYS];
  104. struct sas_port sas_ports[MAX_PHYS]; /* (1) */
  105. bleh;
  106. };
  107. (1) If your LLDD doesn't have its own port representation.
  108. What needs to be initialized (sample function given below).
  109. pcidev
  110. sas_addr -- since the SAS layer doesn't want to mess with
  111. memory allocation, etc, this points to statically
  112. allocated array somewhere (say in your host adapter
  113. structure) and holds the SAS address of the host
  114. adapter as given by you or the manufacturer, etc.
  115. sas_port
  116. sas_phy -- an array of pointers to structures. (see
  117. note above on sas_addr).
  118. These must be set. See more notes below.
  119. num_phys -- the number of phys present in the sas_phy array,
  120. and the number of ports present in the sas_port
  121. array. There can be a maximum num_phys ports (one per
  122. port) so we drop the num_ports, and only use
  123. num_phys.
  124. The event interface:
  125. /* LLDD calls these to notify the class of an event. */
  126. void (*notify_ha_event)(struct sas_ha_struct *, enum ha_event);
  127. void (*notify_port_event)(struct sas_phy *, enum port_event);
  128. void (*notify_phy_event)(struct sas_phy *, enum phy_event);
  129. When sas_register_ha() returns, those are set and can be
  130. called by the LLDD to notify the SAS layer of such events
  131. the SAS layer.
  132. The port notification:
  133. /* The class calls these to notify the LLDD of an event. */
  134. void (*lldd_port_formed)(struct sas_phy *);
  135. void (*lldd_port_deformed)(struct sas_phy *);
  136. If the LLDD wants notification when a port has been formed
  137. or deformed it sets those to a function satisfying the type.
  138. A SAS LLDD should also implement at least one of the Task
  139. Management Functions (TMFs) described in SAM:
  140. /* Task Management Functions. Must be called from process context. */
  141. int (*lldd_abort_task)(struct sas_task *);
  142. int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
  143. int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
  144. int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
  145. int (*lldd_I_T_nexus_reset)(struct domain_device *);
  146. int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
  147. int (*lldd_query_task)(struct sas_task *);
  148. For more information please read SAM from T10.org.
  149. Port and Adapter management:
  150. /* Port and Adapter management */
  151. int (*lldd_clear_nexus_port)(struct sas_port *);
  152. int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
  153. A SAS LLDD should implement at least one of those.
  154. Phy management:
  155. /* Phy management */
  156. int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
  157. lldd_ha -- set this to point to your HA struct. You can also
  158. use container_of if you embedded it as shown above.
  159. A sample initialization and registration function
  160. can look like this (called last thing from probe())
  161. *but* before you enable the phys to do OOB:
  162. static int register_sas_ha(struct my_sas_ha *my_ha)
  163. {
  164. int i;
  165. static struct sas_phy *sas_phys[MAX_PHYS];
  166. static struct sas_port *sas_ports[MAX_PHYS];
  167. my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
  168. for (i = 0; i < MAX_PHYS; i++) {
  169. sas_phys[i] = &my_ha->phys[i].sas_phy;
  170. sas_ports[i] = &my_ha->sas_ports[i];
  171. }
  172. my_ha->sas_ha.sas_phy = sas_phys;
  173. my_ha->sas_ha.sas_port = sas_ports;
  174. my_ha->sas_ha.num_phys = MAX_PHYS;
  175. my_ha->sas_ha.lldd_port_formed = my_port_formed;
  176. my_ha->sas_ha.lldd_dev_found = my_dev_found;
  177. my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
  178. my_ha->sas_ha.lldd_max_execute_num = lldd_max_execute_num; (1)
  179. my_ha->sas_ha.lldd_queue_size = ha_can_queue;
  180. my_ha->sas_ha.lldd_execute_task = my_execute_task;
  181. my_ha->sas_ha.lldd_abort_task = my_abort_task;
  182. my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
  183. my_ha->sas_ha.lldd_clear_aca = my_clear_aca;
  184. my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
  185. my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
  186. my_ha->sas_ha.lldd_lu_reset = my_lu_reset;
  187. my_ha->sas_ha.lldd_query_task = my_query_task;
  188. my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
  189. my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
  190. my_ha->sas_ha.lldd_control_phy = my_control_phy;
  191. return sas_register_ha(&my_ha->sas_ha);
  192. }
  193. (1) This is normally a LLDD parameter, something of the
  194. lines of a task collector. What it tells the SAS Layer is
  195. whether the SAS layer should run in Direct Mode (default:
  196. value 0 or 1) or Task Collector Mode (value greater than 1).
  197. In Direct Mode, the SAS Layer calls Execute Task as soon as
  198. it has a command to send to the SDS, _and_ this is a single
  199. command, i.e. not linked.
  200. Some hardware (e.g. aic94xx) has the capability to DMA more
  201. than one task at a time (interrupt) from host memory. Task
  202. Collector Mode is an optional feature for HAs which support
  203. this in their hardware. (Again, it is completely optional
  204. even if your hardware supports it.)
  205. In Task Collector Mode, the SAS Layer would do _natural_
  206. coalescing of tasks and at the appropriate moment it would
  207. call your driver to DMA more than one task in a single HA
  208. interrupt. DMBS may want to use this by insmod/modprobe
  209. setting the lldd_max_execute_num to something greater than
  210. 1.
  211. (2) SAS 1.1 does not define I_T Nexus Reset TMF.
  212. Events
  213. ------
  214. Events are _the only way_ a SAS LLDD notifies the SAS layer
  215. of anything. There is no other method or way a LLDD to tell
  216. the SAS layer of anything happening internally or in the SAS
  217. domain.
  218. Phy events:
  219. PHYE_LOSS_OF_SIGNAL, (C)
  220. PHYE_OOB_DONE,
  221. PHYE_OOB_ERROR, (C)
  222. PHYE_SPINUP_HOLD.
  223. Port events, passed on a _phy_:
  224. PORTE_BYTES_DMAED, (M)
  225. PORTE_BROADCAST_RCVD, (E)
  226. PORTE_LINK_RESET_ERR, (C)
  227. PORTE_TIMER_EVENT, (C)
  228. PORTE_HARD_RESET.
  229. Host Adapter event:
  230. HAE_RESET
  231. A SAS LLDD should be able to generate
  232. - at least one event from group C (choice),
  233. - events marked M (mandatory) are mandatory (only one),
  234. - events marked E (expander) if it wants the SAS layer
  235. to handle domain revalidation (only one such).
  236. - Unmarked events are optional.
  237. Meaning:
  238. HAE_RESET -- when your HA got internal error and was reset.
  239. PORTE_BYTES_DMAED -- on receiving an IDENTIFY/FIS frame
  240. PORTE_BROADCAST_RCVD -- on receiving a primitive
  241. PORTE_LINK_RESET_ERR -- timer expired, loss of signal, loss
  242. of DWS, etc. (*)
  243. PORTE_TIMER_EVENT -- DWS reset timeout timer expired (*)
  244. PORTE_HARD_RESET -- Hard Reset primitive received.
  245. PHYE_LOSS_OF_SIGNAL -- the device is gone (*)
  246. PHYE_OOB_DONE -- OOB went fine and oob_mode is valid
  247. PHYE_OOB_ERROR -- Error while doing OOB, the device probably
  248. got disconnected. (*)
  249. PHYE_SPINUP_HOLD -- SATA is present, COMWAKE not sent.
  250. (*) should set/clear the appropriate fields in the phy,
  251. or alternatively call the inlined sas_phy_disconnected()
  252. which is just a helper, from their tasklet.
  253. The Execute Command SCSI RPC:
  254. int (*lldd_execute_task)(struct sas_task *, int num,
  255. unsigned long gfp_flags);
  256. Used to queue a task to the SAS LLDD. @task is the tasks to
  257. be executed. @num should be the number of tasks being
  258. queued at this function call (they are linked listed via
  259. task::list), @gfp_mask should be the gfp_mask defining the
  260. context of the caller.
  261. This function should implement the Execute Command SCSI RPC,
  262. or if you're sending a SCSI Task as linked commands, you
  263. should also use this function.
  264. That is, when lldd_execute_task() is called, the command(s)
  265. go out on the transport *immediately*. There is *no*
  266. queuing of any sort and at any level in a SAS LLDD.
  267. The use of task::list is two-fold, one for linked commands,
  268. the other discussed below.
  269. It is possible to queue up more than one task at a time, by
  270. initializing the list element of struct sas_task, and
  271. passing the number of tasks enlisted in this manner in num.
  272. Returns: -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
  273. 0, the task(s) were queued.
  274. If you want to pass num > 1, then either
  275. A) you're the only caller of this function and keep track
  276. of what you've queued to the LLDD, or
  277. B) you know what you're doing and have a strategy of
  278. retrying.
  279. As opposed to queuing one task at a time (function call),
  280. batch queuing of tasks, by having num > 1, greatly
  281. simplifies LLDD code, sequencer code, and _hardware design_,
  282. and has some performance advantages in certain situations
  283. (DBMS).
  284. The LLDD advertises if it can take more than one command at
  285. a time at lldd_execute_task(), by setting the
  286. lldd_max_execute_num parameter (controlled by "collector"
  287. module parameter in aic94xx SAS LLDD).
  288. You should leave this to the default 1, unless you know what
  289. you're doing.
  290. This is a function of the LLDD, to which the SAS layer can
  291. cater to.
  292. int lldd_queue_size
  293. The host adapter's queue size. This is the maximum
  294. number of commands the lldd can have pending to domain
  295. devices on behalf of all upper layers submitting through
  296. lldd_execute_task().
  297. You really want to set this to something (much) larger than
  298. 1.
  299. This _really_ has absolutely nothing to do with queuing.
  300. There is no queuing in SAS LLDDs.
  301. struct sas_task {
  302. dev -- the device this task is destined to
  303. list -- must be initialized (INIT_LIST_HEAD)
  304. task_proto -- _one_ of enum sas_proto
  305. scatter -- pointer to scatter gather list array
  306. num_scatter -- number of elements in scatter
  307. total_xfer_len -- total number of bytes expected to be transferred
  308. data_dir -- PCI_DMA_...
  309. task_done -- callback when the task has finished execution
  310. };
  311. When an external entity, entity other than the LLDD or the
  312. SAS Layer, wants to work with a struct domain_device, it
  313. _must_ call kobject_get() when getting a handle on the
  314. device and kobject_put() when it is done with the device.
  315. This does two things:
  316. A) implements proper kfree() for the device;
  317. B) increments/decrements the kref for all players:
  318. domain_device
  319. all domain_device's ... (if past an expander)
  320. port
  321. host adapter
  322. pci device
  323. and up the ladder, etc.
  324. DISCOVERY
  325. ---------
  326. The sysfs tree has the following purposes:
  327. a) It shows you the physical layout of the SAS domain at
  328. the current time, i.e. how the domain looks in the
  329. physical world right now.
  330. b) Shows some device parameters _at_discovery_time_.
  331. This is a link to the tree(1) program, very useful in
  332. viewing the SAS domain:
  333. ftp://mama.indstate.edu/linux/tree/
  334. I expect user space applications to actually create a
  335. graphical interface of this.
  336. That is, the sysfs domain tree doesn't show or keep state if
  337. you e.g., change the meaning of the READY LED MEANING
  338. setting, but it does show you the current connection status
  339. of the domain device.
  340. Keeping internal device state changes is responsibility of
  341. upper layers (Command set drivers) and user space.
  342. When a device or devices are unplugged from the domain, this
  343. is reflected in the sysfs tree immediately, and the device(s)
  344. removed from the system.
  345. The structure domain_device describes any device in the SAS
  346. domain. It is completely managed by the SAS layer. A task
  347. points to a domain device, this is how the SAS LLDD knows
  348. where to send the task(s) to. A SAS LLDD only reads the
  349. contents of the domain_device structure, but it never creates
  350. or destroys one.
  351. Expander management from User Space
  352. -----------------------------------
  353. In each expander directory in sysfs, there is a file called
  354. "smp_portal". It is a binary sysfs attribute file, which
  355. implements an SMP portal (Note: this is *NOT* an SMP port),
  356. to which user space applications can send SMP requests and
  357. receive SMP responses.
  358. Functionality is deceptively simple:
  359. 1. Build the SMP frame you want to send. The format and layout
  360. is described in the SAS spec. Leave the CRC field equal 0.
  361. open(2)
  362. 2. Open the expander's SMP portal sysfs file in RW mode.
  363. write(2)
  364. 3. Write the frame you built in 1.
  365. read(2)
  366. 4. Read the amount of data you expect to receive for the frame you built.
  367. If you receive different amount of data you expected to receive,
  368. then there was some kind of error.
  369. close(2)
  370. All this process is shown in detail in the function do_smp_func()
  371. and its callers, in the file "expander_conf.c".
  372. The kernel functionality is implemented in the file
  373. "sas_expander.c".
  374. The program "expander_conf.c" implements this. It takes one
  375. argument, the sysfs file name of the SMP portal to the
  376. expander, and gives expander information, including routing
  377. tables.
  378. The SMP portal gives you complete control of the expander,
  379. so please be careful.