分类： LINUX
2009-08-06 19:21:41
When can work
Linux have a error handling thread called scsi_error_handler, it will call eh_strategy_handler function pointer if have otherwise it call scsi_unjam_host to do error handling.
scsi_unjam_host
The below is an important function scsi_error_handler that handle error handling, the marked red part shows some policy related to error handling
The above two conditions meet any one error handling can start. host_failed means the failed scmd request count, the host_busy means working request count. So the necessary condition is all working requests failed it can start error handling. The other two is if host_failed is not equal 0 or host_eh_scheduled is not 0.
host_eh_scheduled is a variable that to let driver to motivate scsi_error_handler by call function scsi_schedule_eh. When user call scsi_scheduler_eh function the host_eh_sceduled would increase one. So the aim it to let scsi_error_handler thread can run if failed request number equal to busy request number.
When doing recovery the normal command also can’t be send to LLDD from mid layer. scsi_request_fn function would check the shost status. when doing recovery the host status set to
1) SHOST_RECOVERY
2) SHOST_CANCEL_RECOVERY
3) SHOST_DEL_RECOVERY
int scsi_error_handler(void *data)
* We use TASK_INTERRUPTIBLE so that the thread is not
* counted against the load average as a running process.
* We never actually get interrupted because kthread_run
* disables singal delivery for the created thread.
if ((shost->host_failed == 0 && shost->host_eh_scheduled == 0) ||
shost->host_failed != shost->host_busy) {
printk("Error handler scsi_eh_%d sleeping\n",
printk("Error handler scsi_eh_%d waking up\n",
* We have a host that is failing for some reason. Figure out
* what we need to do to get it up and online again (if we can).
* If we fail, we end up taking the thing offline.
if (shost->transportt->eh_strategy_handler)
* Note - if the above fails completely, the action is to take
* individual devices offline and flush the queue of any
* outstanding requests that may have been pending. When we
* restart, we restart any I/O to any other devices on the bus
printk("Error handler scsi_eh_%d exiting\n", shost->host_no));
What to do
The linux scsi error handler thread just call a function pointer eh_strategy_handler or scsi_unjam_host. The eh_strategy_handler function pointer common not implement by LLDD. So the scsi_unjam_host stand out to do the job. This function is important so we would talk about it later.
Because when doing failed request recovery the normal operation is blocked so after doing recovery need to restart the operation so function scsi_restart_operation function called.
In Linux document have a detail description for this function. The link is
./Document/scsi/scsi_eh.txt
Below I also refer the function implement and the kernel document to describe the function. I think it is very detail. I don’t want to talk about the thing that the document already talked. However there also may something else to talk.
In host template we have some function pointer to implement that related to error handling. It includes
1) int (* eh_abort_handler)(struct scsi_cmnd *);
2) int (* eh_device_reset_handler)(struct scsi_cmnd *);
3) int (* eh_bus_reset_handler)(struct scsi_cmnd *);
4) int (* eh_host_reset_handler)(struct scsi_cmnd *);
These 4 function is different level error handling from easy to complex. When doing error handling while no normal request handling, it tries below layer error handler, if it can succeed handle all error request no need more level handler stand out. And error handling can finish. Otherwise it need to using higher level error handling function to handle.
Here we should follow the function. Simply saying that there are two type error request.
1) time out request
2) no succeed handle by LLDD request but returned with error sense information.
For different type error request, though it all need be handled in scsi_unjam_host function, however there error mark is different so there error handle way is different.
The SCSI_EH_CANCEL_CMD mark now is only add to time out request. The sense error request is related the scmd that return with not correct sense.
Below would talked about that under paragraph. First let’s make sure on concept that what error handling aim. Just like two type error request so error handling aim existed 2.
1) clear time out scmd in LLDD. Because time out request is handling in LLDD, the scmd is still remembered in LLDD driver. So the error handling want the LLDD forgot the time out scmd in LLDD. And cancel the jobs that doing for handling the time out scmd in LLDD driver.
2) Found hardware have error, so reset hardware to let it work normally.
static void scsi_unjam_host(struct Scsi_Host *shost)
spin_lock_irqsave(shost->host_lock, flags);
list_splice_init(&shost->eh_cmd_q, &eh_work_q);
spin_unlock_irqrestore(shost->host_lock, flags);
SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));
if (!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);
1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
This action is taken for each error-completed
(!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most
SCSI transports/LLDDs automatically acquire sense data on
performance reasons and as sense information could get out of
sync inbetween occurrence of CHECK CONDITION and this action.
Note that if autosense is not supported, scmd->sense_buffer
contains invalid sense data when error-completing the scmd
FAILED in such cases thus invoking SCSI EH. When the scmd
scsi_decide_disposition() is called again.
1. Invoke scsi_request_sense() which issues REQUEST_SENSE
causes higher-severity recovery to be taken for the scmd.
2. Invoke scsi_decide_disposition() on the scmd
- SUCCESS
scmd->retries is set to scmd->allowed preventing
scsi_eh_flush_done_q() from retrying the scmd and
- NEEDS_RETRY
- otherwise
3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
This action is taken for each timed out command.
hostt->eh_abort_handler() is invoked for each scmd. The
handler returns SUCCESS if it has succeeded to make LLDD and
all related hardware forget about the scmd.
If a timedout scmd is successfully aborted and the sdev is
either offline or ready, scsi_eh_finish_cmd() is invoked for
the scmd. Otherwise, the scmd is left in eh_work_q for
Note that both offline and ready status mean that the sdev is
ready to process new scmds, where processing also implies
immediate failing; thus, if a sdev is in one of the two
states, no further recovery action is needed.
Device readiness is tested using scsi_eh_tur() which issues
aborted successfully before reusing it for TEST_UNIT_READY.
4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
This function takes four increasingly more severe measures to
For each sdev which has failed scmds with valid sense data
of which scsi_check_sense()'s verdict is FAILED,
START_STOP_UNIT command is issued w/ start=1. Note that
as we explicitly choose error-completed scmds, it is known
that lower layers have forgotten about the scmd and we can
reuse it for STU.
If STU succeeds and the sdev is either offline or ready,
all failed scmds on the sdev are EH-finished with
scsi_eh_finish_cmd().
*NOTE* If hostt->eh_abort_handler() isn't implemented or
failed, we may still have timed out scmds at this point
and STU doesn't make lower layers forget about those
scmds. Yet, this function EH-finish all scmds on the sdev
if STU succeeds leaving lower layers in an inconsistent
state. It seems that STU action should be taken only when
a sdev has no timed out scmd.
2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
This action is very similar to scsi_eh_stu() except that,
instead of issuing STU, hostt->eh_device_reset_handler()
is used. Also, as we're not issuing SCSI commands and
resetting clears all scmds on the sdev, there is no need
to choose error-completed scmds.
3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
hostt->eh_bus_reset_handler() is invoked for each channel
with failed scmds. If bus reset succeeds, all failed
scmds on all ready or offline sdevs on the channel are
EH-finished.
4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
This is the last resort. hostt->eh_host_reset_handler()
is invoked. If host reset succeeds, all failed scmds on
all ready or offline sdevs on the host are EH-finished.
5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
Take all sdevs which still have unrecovered scmds offline
and EH-finish the scmds.
At this point all scmds are recovered (or given up) and
put on eh_done_q by scsi_eh_finish_cmd(). This function
flushes eh_done_q by either retrying or notifying upper
layer of failure of the scmds.
Previously I have said that error handle aim for time out request is to let LLDD to forgot that scmd after an time out scmd insert to error queue. (By the way time out request also may be handled by other way, I would describe later. Here only talked error handling thread how to handle time out request) the error handling aim is to let LLDD to forgot. So it mark SCSI_EH_CANCEL_CMD to the time out scmd. And insert in to error queue.
2) try to reset device to make hardware forgot scmd. ( because scmd->done function already be ignore, so let hardware forgot that is the left thing ), if failed
The other important function related to time out request eh_timed_out function
The error return request would not try to abort request because it already return by LLDD driver. So it just try to get correct sense.
3) try to reset device to make hardware forgot scmd. ( because scmd->done function already be ignore, so let hardware forgot that is the left thing ), if failed
The two path is only different at previous time out scmd need to let LLDD forgot and sense error request need to re-get sense.
If failed, they all try to reset hardware until reset return succeed. Because they thinks if abort or get sense fail the hardware may have problem so reset. hardware
此处缺少图,若要看图请下载 pdf
此处缺少图,若要看图请下载 pdf
After the error request handled finished, it would insert to an done_q. When scsi_eh_flush_done_q called, all the scmd in the done_q would be handled. The scmd in done_q have two results. one is to reinsert to request queue to retry. If it retry count didn’t run out and the device still on line. The other result is just finish the request.
So the error handling aim in that thread we could see is just make hardware works fine, then may retry or fail the error request.
1) the scmd to do error handing such as send START_STOP_UNIT cmd is reusing the failed scmd.
Time out is an very common error situation. Above talked about how to handle time out scmd in scsi_error_handler thread. In fact not all time out request would be handled by scsi_error_handler thread handle. Many of them would handled by others.
In linux scsi host and transport templates structure both have time out scmd handler function pointer. both name eh_timed_out. Both of them are optional.
1) eh_timed_out function if not available it would return EH_NOT_HANDLED. And insert to time out scmd into error queue.
2) if eh_timed_out avail, it have 3 return value, EH_HANDLED,EH_RESET_TIMER,EH_NOT_HANDLED
3) EH_HANDLED_HANDLED lead to finish to command
4) EH_ RESET_TIMER lead to enlarger the time out time value.
5) EH_NOT_HANDLED lead to the scmd request insert to error queue
And when I check code I found EH_RESET_TIMER is not cause retry count increase not like kernel Document saying
So here shows that only EH_NOT_HANDLED time out scmd would enter in error queue.
Below reference paragraphs have detail introduction.
[1-2-2] Completing a scmd w/ timeout
1. invokes optional hostt->eh_timed_out() callback. Return value can
This indicates that eh_timed_out() dealt with the timeout. The
scmd is passed to __scsi_done() and thus linked into per-cpu
scsi_done_q. Normal command completion described in [1-2-1]
This indicates that more time is required to finish the
command. Timer is restarted. This action is counted as a
retry and only allowed scmd->allowed + 1(!) times. Once the
limit is reached, action for EH_NOT_HANDLED is taken instead.
*NOTE* This action is racy as the LLDD could finish the scmd
after the timeout has expired but before it's added back. In
such cases, scsi_done() would think that timeout has occurred
This is the same as when eh_timed_out() callback doesn't exist.
2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the
command. See [1-3] for more information.
LSI mpt2SAS driver is for the LSI message passing technology related controller driver. I thinks it just like our Loki, it includes embedded cpus.
For error handling part it have below functions.
reference lsi error handling related functions 3
2) if the scmd have hardware related, get a message and set it to MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK and send it to hba firmware. to cancel related command.
3) and get a message and set it to MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET and send it to hba firmware to reset the device.
1) reset the hba controller.
During hba hardware reset. It would block by using completion mechanism to wait for hba restart.
 |
|
