scsi_eh.txt 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479
  1. SCSI EH
  2. ======================================
  3. This document describes SCSI midlayer error handling infrastructure.
  4. Please refer to Documentation/scsi/scsi_mid_low_api.txt for more
  5. information regarding SCSI midlayer.
  6. TABLE OF CONTENTS
  7. [1] How SCSI commands travel through the midlayer and to EH
  8. [1-1] struct scsi_cmnd
  9. [1-2] How do scmd's get completed?
  10. [1-2-1] Completing a scmd w/ scsi_done
  11. [1-2-2] Completing a scmd w/ timeout
  12. [1-3] How EH takes over
  13. [2] How SCSI EH works
  14. [2-1] EH through fine-grained callbacks
  15. [2-1-1] Overview
  16. [2-1-2] Flow of scmds through EH
  17. [2-1-3] Flow of control
  18. [2-2] EH through transportt->eh_strategy_handler()
  19. [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
  20. [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
  21. [2-2-3] Things to consider
  22. [1] How SCSI commands travel through the midlayer and to EH
  23. [1-1] struct scsi_cmnd
  24. Each SCSI command is represented with struct scsi_cmnd (== scmd). A
  25. scmd has two list_head's to link itself into lists. The two are
  26. scmd->list and scmd->eh_entry. The former is used for free list or
  27. per-device allocated scmd list and not of much interest to this EH
  28. discussion. The latter is used for completion and EH lists and unless
  29. otherwise stated scmds are always linked using scmd->eh_entry in this
  30. discussion.
  31. [1-2] How do scmd's get completed?
  32. Once LLDD gets hold of a scmd, either the LLDD will complete the
  33. command by calling scsi_done callback passed from midlayer when
  34. invoking hostt->queuecommand() or SCSI midlayer will time it out.
  35. [1-2-1] Completing a scmd w/ scsi_done
  36. For all non-EH commands, scsi_done() is the completion callback. It
  37. does the following.
  38. 1. Delete timeout timer. If it fails, it means that timeout timer
  39. has expired and is going to finish the command. Just return.
  40. 2. Link scmd to per-cpu scsi_done_q using scmd->en_entry
  41. 3. Raise SCSI_SOFTIRQ
  42. SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to
  43. determine what to do with the command. scsi_decide_disposition()
  44. looks at the scmd->result value and sense data to determine what to do
  45. with the command.
  46. - SUCCESS
  47. scsi_finish_command() is invoked for the command. The
  48. function does some maintenance choirs and notify completion by
  49. calling scmd->done() callback, which, for fs requests, would
  50. be HLD completion callback - sd:sd_rw_intr, sr:rw_intr,
  51. st:st_intr.
  52. - NEEDS_RETRY
  53. - ADD_TO_MLQUEUE
  54. scmd is requeued to blk queue.
  55. - otherwise
  56. scsi_eh_scmd_add(scmd, 0) is invoked for the command. See
  57. [1-3] for details of this function.
  58. [1-2-2] Completing a scmd w/ timeout
  59. The timeout handler is scsi_times_out(). When a timeout occurs, this
  60. function
  61. 1. invokes optional hostt->eh_timed_out() callback. Return value can
  62. be one of
  63. - EH_HANDLED
  64. This indicates that eh_timed_out() dealt with the timeout. The
  65. scmd is passed to __scsi_done() and thus linked into per-cpu
  66. scsi_done_q. Normal command completion described in [1-2-1]
  67. follows.
  68. - EH_RESET_TIMER
  69. This indicates that more time is required to finish the
  70. command. Timer is restarted. This action is counted as a
  71. retry and only allowed scmd->allowed + 1(!) times. Once the
  72. limit is reached, action for EH_NOT_HANDLED is taken instead.
  73. *NOTE* This action is racy as the LLDD could finish the scmd
  74. after the timeout has expired but before it's added back. In
  75. such cases, scsi_done() would think that timeout has occurred
  76. and return without doing anything. We lose completion and the
  77. command will time out again.
  78. - EH_NOT_HANDLED
  79. This is the same as when eh_timed_out() callback doesn't exist.
  80. Step #2 is taken.
  81. 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the
  82. command. See [1-3] for more information.
  83. [1-3] How EH takes over
  84. scmds enter EH via scsi_eh_scmd_add(), which does the following.
  85. 1. Turns on scmd->eh_eflags as requested. It's 0 for error
  86. completions and SCSI_EH_CANCEL_CMD for timeouts.
  87. 2. Links scmd->eh_entry to shost->eh_cmd_q
  88. 3. Sets SHOST_RECOVERY bit in shost->shost_state
  89. 4. Increments shost->host_failed
  90. 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
  91. As can be seen above, once any scmd is added to shost->eh_cmd_q,
  92. SHOST_RECOVERY shost_state bit is turned on. This prevents any new
  93. scmd to be issued from blk queue to the host; eventually, all scmds on
  94. the host either complete normally, fail and get added to eh_cmd_q, or
  95. time out and get added to shost->eh_cmd_q.
  96. If all scmds either complete or fail, the number of in-flight scmds
  97. becomes equal to the number of failed scmds - i.e. shost->host_busy ==
  98. shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
  99. SCSI EH thread can expect that all in-flight commands have failed and
  100. are linked on shost->eh_cmd_q.
  101. Note that this does not mean lower layers are quiescent. If a LLDD
  102. completed a scmd with error status, the LLDD and lower layers are
  103. assumed to forget about the scmd at that point. However, if a scmd
  104. has timed out, unless hostt->eh_timed_out() made lower layers forget
  105. about the scmd, which currently no LLDD does, the command is still
  106. active as long as lower layers are concerned and completion could
  107. occur at any time. Of course, all such completions are ignored as the
  108. timer has already expired.
  109. We'll talk about how SCSI EH takes actions to abort - make LLDD
  110. forget about - timed out scmds later.
  111. [2] How SCSI EH works
  112. LLDD's can implement SCSI EH actions in one of the following two
  113. ways.
  114. - Fine-grained EH callbacks
  115. LLDD can implement fine-grained EH callbacks and let SCSI
  116. midlayer drive error handling and call appropriate callbacks.
  117. This will be discussed further in [2-1].
  118. - eh_strategy_handler() callback
  119. This is one big callback which should perform whole error
  120. handling. As such, it should do all choirs SCSI midlayer
  121. performs during recovery. This will be discussed in [2-2].
  122. Once recovery is complete, SCSI EH resumes normal operation by
  123. calling scsi_restart_operations(), which
  124. 1. Checks if door locking is needed and locks door.
  125. 2. Clears SHOST_RECOVERY shost_state bit
  126. 3. Wakes up waiters on shost->host_wait. This occurs if someone
  127. calls scsi_block_when_processing_errors() on the host.
  128. (*QUESTION* why is it needed? All operations will be blocked
  129. anyway after it reaches blk queue.)
  130. 4. Kicks queues in all devices on the host in the asses
  131. [2-1] EH through fine-grained callbacks
  132. [2-1-1] Overview
  133. If eh_strategy_handler() is not present, SCSI midlayer takes charge
  134. of driving error handling. EH's goals are two - make LLDD, host and
  135. device forget about timed out scmds and make them ready for new
  136. commands. A scmd is said to be recovered if the scmd is forgotten by
  137. lower layers and lower layers are ready to process or fail the scmd
  138. again.
  139. To achieve these goals, EH performs recovery actions with increasing
  140. severity. Some actions are performed by issuing SCSI commands and
  141. others are performed by invoking one of the following fine-grained
  142. hostt EH callbacks. Callbacks may be omitted and omitted ones are
  143. considered to fail always.
  144. int (* eh_abort_handler)(struct scsi_cmnd *);
  145. int (* eh_device_reset_handler)(struct scsi_cmnd *);
  146. int (* eh_bus_reset_handler)(struct scsi_cmnd *);
  147. int (* eh_host_reset_handler)(struct scsi_cmnd *);
  148. Higher-severity actions are taken only when lower-severity actions
  149. cannot recover some of failed scmds. Also, note that failure of the
  150. highest-severity action means EH failure and results in offlining of
  151. all unrecovered devices.
  152. During recovery, the following rules are followed
  153. - Recovery actions are performed on failed scmds on the to do list,
  154. eh_work_q. If a recovery action succeeds for a scmd, recovered
  155. scmds are removed from eh_work_q.
  156. Note that single recovery action on a scmd can recover multiple
  157. scmds. e.g. resetting a device recovers all failed scmds on the
  158. device.
  159. - Higher severity actions are taken iff eh_work_q is not empty after
  160. lower severity actions are complete.
  161. - EH reuses failed scmds to issue commands for recovery. For
  162. timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
  163. before reusing it for EH commands.
  164. When a scmd is recovered, the scmd is moved from eh_work_q to EH
  165. local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
  166. recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
  167. either retry or error-finish (notify upper layer of failure) recovered
  168. scmds.
  169. scmds are retried iff its sdev is still online (not offlined during
  170. EH), REQ_FAILFAST is not set and ++scmd->retries is less than
  171. scmd->allowed.
  172. [2-1-2] Flow of scmds through EH
  173. 1. Error completion / time out
  174. ACTION: scsi_eh_scmd_add() is invoked for scmd
  175. - set scmd->eh_eflags
  176. - add scmd to shost->eh_cmd_q
  177. - set SHOST_RECOVERY
  178. - shost->host_failed++
  179. LOCKING: shost->host_lock
  180. 2. EH starts
  181. ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
  182. is cleared.
  183. LOCKING: shost->host_lock (not strictly necessary, just for
  184. consistency)
  185. 3. scmd recovered
  186. ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
  187. - shost->host_failed--
  188. - clear scmd->eh_eflags
  189. - scsi_setup_cmd_retry()
  190. - move from local eh_work_q to local eh_done_q
  191. LOCKING: none
  192. 4. EH completes
  193. ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
  194. layer of failure.
  195. - scmd is removed from eh_done_q and scmd->eh_entry is cleared
  196. - if retry is necessary, scmd is requeued using
  197. scsi_queue_insert()
  198. - otherwise, scsi_finish_command() is invoked for scmd
  199. LOCKING: queue or finish function performs appropriate locking
  200. [2-1-3] Flow of control
  201. EH through fine-grained callbacks start from scsi_unjam_host().
  202. <<scsi_unjam_host>>
  203. 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
  204. eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
  205. cleared by this action.
  206. 2. Invoke scsi_eh_get_sense.
  207. <<scsi_eh_get_sense>>
  208. This action is taken for each error-completed
  209. (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most
  210. SCSI transports/LLDDs automatically acquire sense data on
  211. command failures (autosense). Autosense is recommended for
  212. performance reasons and as sense information could get out of
  213. sync inbetween occurrence of CHECK CONDITION and this action.
  214. Note that if autosense is not supported, scmd->sense_buffer
  215. contains invalid sense data when error-completing the scmd
  216. with scsi_done(). scsi_decide_disposition() always returns
  217. FAILED in such cases thus invoking SCSI EH. When the scmd
  218. reaches here, sense data is acquired and
  219. scsi_decide_disposition() is called again.
  220. 1. Invoke scsi_request_sense() which issues REQUEST_SENSE
  221. command. If fails, no action. Note that taking no action
  222. causes higher-severity recovery to be taken for the scmd.
  223. 2. Invoke scsi_decide_disposition() on the scmd
  224. - SUCCESS
  225. scmd->retries is set to scmd->allowed preventing
  226. scsi_eh_flush_done_q() from retrying the scmd and
  227. scsi_eh_finish_cmd() is invoked.
  228. - NEEDS_RETRY
  229. scsi_eh_finish_cmd() invoked
  230. - otherwise
  231. No action.
  232. 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
  233. <<scsi_eh_abort_cmds>>
  234. This action is taken for each timed out command.
  235. hostt->eh_abort_handler() is invoked for each scmd. The
  236. handler returns SUCCESS if it has succeeded to make LLDD and
  237. all related hardware forget about the scmd.
  238. If a timedout scmd is successfully aborted and the sdev is
  239. either offline or ready, scsi_eh_finish_cmd() is invoked for
  240. the scmd. Otherwise, the scmd is left in eh_work_q for
  241. higher-severity actions.
  242. Note that both offline and ready status mean that the sdev is
  243. ready to process new scmds, where processing also implies
  244. immediate failing; thus, if a sdev is in one of the two
  245. states, no further recovery action is needed.
  246. Device readiness is tested using scsi_eh_tur() which issues
  247. TEST_UNIT_READY command. Note that the scmd must have been
  248. aborted successfully before reusing it for TEST_UNIT_READY.
  249. 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
  250. <<scsi_eh_ready_devs>>
  251. This function takes four increasingly more severe measures to
  252. make failed sdevs ready for new commands.
  253. 1. Invoke scsi_eh_stu()
  254. <<scsi_eh_stu>>
  255. For each sdev which has failed scmds with valid sense data
  256. of which scsi_check_sense()'s verdict is FAILED,
  257. START_STOP_UNIT command is issued w/ start=1. Note that
  258. as we explicitly choose error-completed scmds, it is known
  259. that lower layers have forgotten about the scmd and we can
  260. reuse it for STU.
  261. If STU succeeds and the sdev is either offline or ready,
  262. all failed scmds on the sdev are EH-finished with
  263. scsi_eh_finish_cmd().
  264. *NOTE* If hostt->eh_abort_handler() isn't implemented or
  265. failed, we may still have timed out scmds at this point
  266. and STU doesn't make lower layers forget about those
  267. scmds. Yet, this function EH-finish all scmds on the sdev
  268. if STU succeeds leaving lower layers in an inconsistent
  269. state. It seems that STU action should be taken only when
  270. a sdev has no timed out scmd.
  271. 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
  272. <<scsi_eh_bus_device_reset>>
  273. This action is very similar to scsi_eh_stu() except that,
  274. instead of issuing STU, hostt->eh_device_reset_handler()
  275. is used. Also, as we're not issuing SCSI commands and
  276. resetting clears all scmds on the sdev, there is no need
  277. to choose error-completed scmds.
  278. 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
  279. <<scsi_eh_bus_reset>>
  280. hostt->eh_bus_reset_handler() is invoked for each channel
  281. with failed scmds. If bus reset succeeds, all failed
  282. scmds on all ready or offline sdevs on the channel are
  283. EH-finished.
  284. 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
  285. <<scsi_eh_host_reset>>
  286. This is the last resort. hostt->eh_host_reset_handler()
  287. is invoked. If host reset succeeds, all failed scmds on
  288. all ready or offline sdevs on the host are EH-finished.
  289. 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
  290. <<scsi_eh_offline_sdevs>>
  291. Take all sdevs which still have unrecovered scmds offline
  292. and EH-finish the scmds.
  293. 5. Invoke scsi_eh_flush_done_q().
  294. <<scsi_eh_flush_done_q>>
  295. At this point all scmds are recovered (or given up) and
  296. put on eh_done_q by scsi_eh_finish_cmd(). This function
  297. flushes eh_done_q by either retrying or notifying upper
  298. layer of failure of the scmds.
  299. [2-2] EH through transportt->eh_strategy_handler()
  300. transportt->eh_strategy_handler() is invoked in the place of
  301. scsi_unjam_host() and it is responsible for whole recovery process.
  302. On completion, the handler should have made lower layers forget about
  303. all failed scmds and either ready for new commands or offline. Also,
  304. it should perform SCSI EH maintenance choirs to maintain integrity of
  305. SCSI midlayer. IOW, of the steps described in [2-1-2], all steps
  306. except for #1 must be implemented by eh_strategy_handler().
  307. [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
  308. The following conditions are true on entry to the handler.
  309. - Each failed scmd's eh_flags field is set appropriately.
  310. - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
  311. - SHOST_RECOVERY is set.
  312. - shost->host_failed == shost->host_busy
  313. [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
  314. The following conditions must be true on exit from the handler.
  315. - shost->host_failed is zero.
  316. - Each scmd's eh_eflags field is cleared.
  317. - Each scmd is in such a state that scsi_setup_cmd_retry() on the
  318. scmd doesn't make any difference.
  319. - shost->eh_cmd_q is cleared.
  320. - Each scmd->eh_entry is cleared.
  321. - Either scsi_queue_insert() or scsi_finish_command() is called on
  322. each scmd. Note that the handler is free to use scmd->retries and
  323. ->allowed to limit the number of retries.
  324. [2-2-3] Things to consider
  325. - Know that timed out scmds are still active on lower layers. Make
  326. lower layers forget about them before doing anything else with
  327. those scmds.
  328. - For consistency, when accessing/modifying shost data structure,
  329. grab shost->host_lock.
  330. - On completion, each failed sdev must have forgotten about all
  331. active scmds.
  332. - On completion, each failed sdev must be ready for new commands or
  333. offline.
  334. --
  335. Tejun Heo
  336. htejun@gmail.com
  337. 11th September 2005