요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
=======
SCSI EH
=======
This document describes SCSI midlayer error handling infrastructure.
Please refer to Documentation/scsi/scsi_mid_low_api.rst for more
information regarding SCSI midlayer.
.. TABLE OF CONTENTS
[1] How SCSI commands travel through the midlayer and to EH
[1-1] struct scsi_cmnd
[1-2] How do scmd's get completed?
[1-2-1] Completing a scmd w/ scsi_done
[1-2-2] Completing a scmd w/ timeout
[1-3] How EH takes over
[2] How SCSI EH works
[2-1] EH through fine-grained callbacks
[2-1-1] Overview
[2-1-2] Flow of scmds through EH
[2-1-3] Flow of control
[2-2] EH through transportt->eh_strategy_handler()
[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
[2-2-3] Things to consider
1. How SCSI commands travel through the midlayer and to EH
==========================================================
1.1 struct scsi_cmnd
--------------------
Each SCSI command is represented with struct scsi_cmnd (== scmd). A
scmd has two list_head's to link itself into lists. The two are
scmd->list and scmd->eh_entry. The former is used for free list or
per-device allocated scmd list and not of much interest to this EH
discussion. The latter is used for completion and EH lists and unless
otherwise stated scmds are always linked using scmd->eh_entry in this
discussion.
1.2 How do scmd's get completed?
--------------------------------
Once LLDD gets hold of a scmd, either the LLDD will complete the
command by calling scsi_done callback passed from midlayer when
invoking hostt->queuecommand() or the block layer will time it out.
1.2.1 Completing a scmd w/ scsi_done
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For all non-EH commands, scsi_done() is the completion callback. It
just calls blk_mq_complete_request() to delete the block layer timer and
raise BLOCK_SOFTIRQ.
The BLOCK_SOFTIRQ indirectly calls scsi_complete(), which calls
scsi_decide_disposition() to determine what to do with the command.
scsi_decide_disposition() looks at the scmd->result value and sense
data to determine what to do with the command.
- SUCCESS
scsi_finish_command() is invoked for the command. The
function does some maintenance chores and then calls
scsi_io_completion() to finish the I/O.
scsi_io_completion() then notifies the block layer on
the completed request by calling blk_end_request and
friends or figures out what to do with the remainder
of the data in case of an error.
- NEEDS_RETRY
- ADD_TO_MLQUEUE
scmd is requeued to blk queue.
- otherwise
scsi_eh_scmd_add(scmd) is invoked for the command. See
[1-3] for details of this function.
1.2.2 Completing a scmd w/ timeout
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The timeout handler is scsi_timeout(). When a timeout occurs, this function
1. invokes optional hostt->eh_timed_out() callback. Return value can
be one of
- SCSI_EH_RESET_TIMER
This indicates that more time is required to finish the
command. Timer is restarted.
- SCSI_EH_NOT_HANDLED
eh_timed_out() callback did not handle the command.
Step #2 is taken.
- SCSI_EH_DONE
eh_timed_out() completed the command.
2. scsi_abort_command() is invoked to schedule an asynchronous abort which may
issue a retry scmd->allowed + 1 times. Asynchronous aborts are not invoked
for commands for which the SCSI_EH_ABORT_SCHEDULED flag is set (this
indicates that the command already had been aborted once, and this is a
retry which failed), when retries are exceeded, or when the EH deadline is
expired. In these cases Step #3 is taken.
3. scsi_eh_scmd_add(scmd) is invoked for the
command. See [1-4] for more information.
1.3 Asynchronous command aborts
-------------------------------
After a timeout occurs a command abort is scheduled from
scsi_abort_command(). If the abort is successful the command
will either be retried (if the number of retries is not exhausted)
or terminated with DID_TIME_OUT.
Otherwise scsi_eh_scmd_add() is invoked for the command.
See [1-4] for more information.
1.4 How EH takes over
---------------------
scmds enter EH via scsi_eh_scmd_add(), which does the following.
1. Links scmd->eh_entry to shost->eh_cmd_q
2. Sets SHOST_RECOVERY bit in shost->shost_state
3. Increments shost->host_failed
4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
As can be seen above, once any scmd is added to shost->eh_cmd_q,
SHOST_RECOVERY shost_state bit is turned on. This prevents any new
scmd to be issued from blk queue to the host; eventually, all scmds on
the host either complete normally, fail and get added to eh_cmd_q, or
time out and get added to shost->eh_cmd_q.
If all scmds either complete or fail, the number of in-flight scmds
becomes equal to the number of failed scmds - i.e. shost->host_busy ==
shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
SCSI EH thread can expect that all in-flight commands have failed and
are linked on shost->eh_cmd_q.
Note that this does not mean lower layers are quiescent. If a LLDD
completed a scmd with error status, the LLDD and lower layers are
assumed to forget about the scmd at that point. However, if a scmd
has timed out, unless hostt->eh_timed_out() made lower layers forget
about the scmd, which currently no LLDD does, the command is still
active as long as lower layers are concerned and completion could
occur at any time. Of course, all such completions are ignored as the
timer has already expired.
We'll talk about how SCSI EH takes actions to abort - make LLDD
forget about - timed out scmds later.
2. How SCSI EH works
====================
LLDD's can implement SCSI EH actions in one of the following two
ways.
- Fine-grained EH callbacks
LLDD can implement fine-grained EH callbacks and let SCSI
midlayer drive error handling and call appropriate callbacks.
This will be discussed further in [2-1].
- eh_strategy_handler() callback
This is one big callback which should perform whole error
handling. As such, it should do all chores the SCSI midlayer
performs during recovery. This will be discussed in [2-2].
Once recovery is complete, SCSI EH resumes normal operation by
calling scsi_restart_operations(), which
1. Checks if door locking is needed and locks door.
2. Clears SHOST_RECOVERY shost_state bit
3. Wakes up waiters on shost->host_wait. This occurs if someone
calls scsi_block_when_processing_errors() on the host.
(*QUESTION* why is it needed? All operations will be blocked
anyway after it reaches blk queue.)
4. Kicks queues in all devices on the host in the asses
2.1 EH through fine-grained callbacks
-------------------------------------
2.1.1 Overview
^^^^^^^^^^^^^^
If eh_strategy_handler() is not present, SCSI midlayer takes charge
of driving error handling. EH's goals are two - make LLDD, host and
device forget about timed out scmds and make them ready for new
commands. A scmd is said to be recovered if the scmd is forgotten by
lower layers and lower layers are ready to process or fail the scmd
again.
To achieve these goals, EH performs recovery actions with increasing
severity. Some actions are performed by issuing SCSI commands and
others are performed by invoking one of the following fine-grained
hostt EH callbacks. Callbacks may be omitted and omitted ones are
considered to fail always.
::
int (* eh_abort_handler)(struct scsi_cmnd *);
int (* eh_device_reset_handler)(struct scsi_cmnd *);
int (* eh_bus_reset_handler)(struct scsi_cmnd *);
int (* eh_host_reset_handler)(struct scsi_cmnd *);
Higher-severity actions are taken only when lower-severity actions
cannot recover some of failed scmds. Also, note that failure of the
highest-severity action means EH failure and results in offlining of
all unrecovered devices.
During recovery, the following rules are followed
- Recovery actions are performed on failed scmds on the to do list,
eh_work_q. If a recovery action succeeds for a scmd, recovered
scmds are removed from eh_work_q.
Note that single recovery action on a scmd can recover multiple
scmds. e.g. resetting a device recovers all failed scmds on the
device.
- Higher severity actions are taken iff eh_work_q is not empty after
lower severity actions are complete.
- EH reuses failed scmds to issue commands for recovery. For
timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
before reusing it for EH commands.
When a scmd is recovered, the scmd is moved from eh_work_q to EH
local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
either retry or error-finish (notify upper layer of failure) recovered
scmds.
scmds are retried iff its sdev is still online (not offlined during
EH), REQ_FAILFAST is not set and ++scmd->retries is less than
scmd->allowed.
2.1.2 Flow of scmds through EH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Error completion / time out
:ACTION: scsi_eh_scmd_add() is invoked for scmd
- add scmd to shost->eh_cmd_q
- set SHOST_RECOVERY
- shost->host_failed++
:LOCKING: shost->host_lock
2. EH starts
:ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
is cleared.
:LOCKING: shost->host_lock (not strictly necessary, just for
consistency)
3. scmd recovered
:ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
- move from local eh_work_q to local eh_done_q
:LOCKING: none
:CONCURRENCY: at most one thread per separate eh_work_q to
keep queue manipulation lockless
4. EH completes
:ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
layer of failure. May be called concurrently but must have
a no more than one thread per separate eh_work_q to
manipulate the queue locklessly
- scmd is removed from eh_done_q and scmd->eh_entry is cleared
- if retry is necessary, scmd is requeued using
scsi_queue_insert()
- otherwise, scsi_finish_command() is invoked for scmd
- zero shost->host_failed
:LOCKING: queue or finish function performs appropriate locking
2.1.3 Flow of control
^^^^^^^^^^^^^^^^^^^^^^
EH through fine-grained callbacks start from scsi_unjam_host().
``scsi_unjam_host``
1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
cleared by this action.
2. Invoke scsi_eh_get_sense.
``scsi_eh_get_sense``
This action is taken for each error-completed
command without valid sense data. Most
SCSI transports/LLDDs automatically acquire sense data on
command failures (autosense). Autosense is recommended for
performance reasons and as sense information could get out of
sync between occurrence of CHECK CONDITION and this action.
Note that if autosense is not supported, scmd->sense_buffer
contains invalid sense data when error-completing the scmd
with scsi_done(). scsi_decide_disposition() always returns
FAILED in such cases thus invoking SCSI EH. When the scmd
reaches here, sense data is acquired and
scsi_decide_disposition() is called again.
1. Invoke scsi_request_sense() which issues REQUEST_SENSE
command. If fails, no action. Note that taking no action
causes higher-severity recovery to be taken for the scmd.
2. Invoke scsi_decide_disposition() on the scmd
- SUCCESS
scmd->retries is set to scmd->allowed preventing
scsi_eh_flush_done_q() from retrying the scmd and
scsi_eh_finish_cmd() is invoked.
- NEEDS_RETRY
scsi_eh_finish_cmd() invoked
- otherwise
No action.
4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
``scsi_eh_ready_devs``
This function takes four increasingly more severe measures to
make failed sdevs ready for new commands.
1. Invoke scsi_eh_stu()
``scsi_eh_stu``
For each sdev which has failed scmds with valid sense data
of which scsi_check_sense()'s verdict is FAILED,
START STOP UNIT command is issued w/ start=1. Note that
as we explicitly choose error-completed scmds, it is known
that lower layers have forgotten about the scmd and we can
reuse it for STU.
If STU succeeds and the sdev is either offline or ready,
all failed scmds on the sdev are EH-finished with
scsi_eh_finish_cmd().
*NOTE* If hostt->eh_abort_handler() isn't implemented or
failed, we may still have timed out scmds at this point
and STU doesn't make lower layers forget about those
scmds. Yet, this function EH-finish all scmds on the sdev
if STU succeeds leaving lower layers in an inconsistent
state. It seems that STU action should be taken only when
a sdev has no timed out scmd.
2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
``scsi_eh_bus_device_reset``
This action is very similar to scsi_eh_stu() except that,
instead of issuing STU, hostt->eh_device_reset_handler()
is used. Also, as we're not issuing SCSI commands and
resetting clears all scmds on the sdev, there is no need
to choose error-completed scmds.
3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
``scsi_eh_bus_reset``
hostt->eh_bus_reset_handler() is invoked for each channel
with failed scmds. If bus reset succeeds, all failed
scmds on all ready or offline sdevs on the channel are
EH-finished.
4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
``scsi_eh_host_reset``
This is the last resort. hostt->eh_host_reset_handler()
is invoked. If host reset succeeds, all failed scmds on
all ready or offline sdevs on the host are EH-finished.
5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
``scsi_eh_offline_sdevs``
Take all sdevs which still have unrecovered scmds offline
and EH-finish the scmds.
5. Invoke scsi_eh_flush_done_q().
``scsi_eh_flush_done_q``
At this point all scmds are recovered (or given up) and
put on eh_done_q by scsi_eh_finish_cmd(). This function
flushes eh_done_q by either retrying or notifying upper
layer of failure of the scmds.
2.2 EH through transportt->eh_strategy_handler()
------------------------------------------------
transportt->eh_strategy_handler() is invoked in the place of
scsi_unjam_host() and it is responsible for whole recovery process.
On completion, the handler should have made lower layers forget about
all failed scmds and either ready for new commands or offline. Also,
it should perform SCSI EH maintenance chores to maintain integrity of
SCSI midlayer. IOW, of the steps described in [2-1-2], all steps
except for #1 must be implemented by eh_strategy_handler().
2.2.1 Pre transportt->eh_strategy_handler() SCSI midlayer conditions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following conditions are true on entry to the handler.
- Each failed scmd's eh_flags field is set appropriately.
- Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
- SHOST_RECOVERY is set.
- shost->host_failed == shost->host_busy
2.2.2 Post transportt->eh_strategy_handler() SCSI midlayer conditions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following conditions must be true on exit from the handler.
- shost->host_failed is zero.
- shost->eh_cmd_q is cleared.
- Each scmd->eh_entry is cleared.
- Either scsi_queue_insert() or scsi_finish_command() is called on
each scmd. Note that the handler is free to use scmd->retries and
->allowed to limit the number of retries.
2.2.3 Things to consider
^^^^^^^^^^^^^^^^^^^^^^^^
- Know that timed out scmds are still active on lower layers. Make
lower layers forget about them before doing anything else with
those scmds.
- For consistency, when accessing/modifying shost data structure,
grab shost->host_lock.
- On completion, each failed sdev must have forgotten about all
active scmds.
- On completion, each failed sdev must be ready for new commands or
offline.
Tejun Heo
11th September 2005
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
문서 목적과 구성
1-29이 문서는 SCSI 중간 계층의 오류 처리(Error Handling, EH) 기반 구조를 설명한다. 중간 계층 전반은 `Documentation/scsi/scsi_mid_low_api.rst`를 함께 참고한다.
내용은 `struct scsi_cmnd`, 정상 완료와 시간 초과, EH 인수 과정, 세분화된 콜백, `transportt->eh_strategy_handler()` 방식의 진입 및 종료 조건으로 이어진다.
.. SPDX-License-Identifier: GPL-2.0
=======
SCSI EH
=======
This document describes SCSI midlayer error handling infrastructure.
Please refer to Documentation/scsi/scsi_mid_low_api.rst for more
information regarding SCSI midlayer.
.. TABLE OF CONTENTS
[1] How SCSI commands travel through the midlayer and to EH
[1-1] struct scsi_cmnd
[1-2] How do scmd's get completed?
[1-2-1] Completing a scmd w/ scsi_done
[1-2-2] Completing a scmd w/ timeout
[1-3] How EH takes over
[2] How SCSI EH works
[2-1] EH through fine-grained callbacks
[2-1-1] Overview
[2-1-2] Flow of scmds through EH
[2-1-3] Flow of control
[2-2] EH through transportt->eh_strategy_handler()
[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions
[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions
[2-2-3] Things to consider
scsi_cmnd와 완료 경로
30-52각 SCSI 명령은 `struct scsi_cmnd`, 줄여서 `scmd`로 표현된다. 두 `list_head` 가운데 `scmd->list`는 자유 목록 또는 장치별 할당 목록에 쓰이고, EH 논의에서 중요한 `scmd->eh_entry`는 완료 목록과 EH 목록을 연결한다.
LLDD가 `scmd`를 받으면 `hostt->queuecommand()` 호출 때 중간 계층이 넘긴 `scsi_done` 콜백으로 완료하거나, 완료되지 못한 명령은 블록 계층의 시간 제한에 걸린다.
1. How SCSI commands travel through the midlayer and to EH
==========================================================
1.1 struct scsi_cmnd
--------------------
Each SCSI command is represented with struct scsi_cmnd (== scmd). A
scmd has two list_head's to link itself into lists. The two are
scmd->list and scmd->eh_entry. The former is used for free list or
per-device allocated scmd list and not of much interest to this EH
discussion. The latter is used for completion and EH lists and unless
otherwise stated scmds are always linked using scmd->eh_entry in this
discussion.
1.2 How do scmd's get completed?
--------------------------------
Once LLDD gets hold of a scmd, either the LLDD will complete the
command by calling scsi_done callback passed from midlayer when
invoking hostt->queuecommand() or the block layer will time it out.
scsi_done에 의한 완료
53-86EH 명령이 아닌 모든 명령의 완료 콜백은 `scsi_done()`이다. 이 함수는 `blk_mq_complete_request()`를 호출해 블록 계층 타이머를 제거하고 `BLOCK_SOFTIRQ`를 발생시킨다. 소프트 IRQ는 간접적으로 `scsi_complete()`를 호출하고, 이어 `scsi_decide_disposition()`이 `scmd->result`와 sense 데이터를 검사해 처리 방식을 정한다.
결과가 `SUCCESS`이면 `scsi_finish_command()`가 유지 작업을 수행하고 `scsi_io_completion()`으로 I/O를 끝낸다. `scsi_io_completion()`은 블록 계층에 완료를 알리거나 오류 뒤 남은 데이터를 처리한다. `NEEDS_RETRY` 또는 `ADD_TO_MLQUEUE`이면 명령을 블록 큐에 다시 넣는다. 그 밖의 결과는 `scsi_eh_scmd_add(scmd)`로 EH에 넘긴다.
scsi_done 이후 disposition에 따른 경로다.
1.2.1 Completing a scmd w/ scsi_done
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For all non-EH commands, scsi_done() is the completion callback. It
just calls blk_mq_complete_request() to delete the block layer timer and
raise BLOCK_SOFTIRQ.
The BLOCK_SOFTIRQ indirectly calls scsi_complete(), which calls
scsi_decide_disposition() to determine what to do with the command.
scsi_decide_disposition() looks at the scmd->result value and sense
data to determine what to do with the command.
- SUCCESS
scsi_finish_command() is invoked for the command. The
function does some maintenance chores and then calls
scsi_io_completion() to finish the I/O.
scsi_io_completion() then notifies the block layer on
the completed request by calling blk_end_request and
friends or figures out what to do with the remainder
of the data in case of an error.
- NEEDS_RETRY
- ADD_TO_MLQUEUE
scmd is requeued to blk queue.
- otherwise
scsi_eh_scmd_add(scmd) is invoked for the command. See
[1-3] for details of this function.
시간 초과 처리
87-115시간 초과 처리기는 `scsi_timeout()`이다. 먼저 선택적인 `hostt->eh_timed_out()`을 호출한다. `SCSI_EH_RESET_TIMER`이면 시간이 더 필요하므로 타이머를 다시 시작하고, `SCSI_EH_NOT_HANDLED`이면 다음 단계로 진행하며, `SCSI_EH_DONE`이면 콜백이 명령을 완료한 것이다.
처리되지 않은 경우 `scsi_abort_command()`가 비동기 abort를 예약하며 최대 `scmd->allowed + 1`회 재시도할 수 있다. 이미 `SCSI_EH_ABORT_SCHEDULED`인 명령, 재시도 횟수를 넘긴 명령, EH 기한이 만료된 명령에는 비동기 abort를 다시 예약하지 않고 `scsi_eh_scmd_add(scmd)`로 넘긴다.
시간 초과 콜백의 세 가지 의미다.
1.2.2 Completing a scmd w/ timeout
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The timeout handler is scsi_timeout(). When a timeout occurs, this function
1. invokes optional hostt->eh_timed_out() callback. Return value can
be one of
- SCSI_EH_RESET_TIMER
This indicates that more time is required to finish the
command. Timer is restarted.
- SCSI_EH_NOT_HANDLED
eh_timed_out() callback did not handle the command.
Step #2 is taken.
- SCSI_EH_DONE
eh_timed_out() completed the command.
2. scsi_abort_command() is invoked to schedule an asynchronous abort which may
issue a retry scmd->allowed + 1 times. Asynchronous aborts are not invoked
for commands for which the SCSI_EH_ABORT_SCHEDULED flag is set (this
indicates that the command already had been aborted once, and this is a
retry which failed), when retries are exceeded, or when the EH deadline is
expired. In these cases Step #3 is taken.
3. scsi_eh_scmd_add(scmd) is invoked for the
command. See [1-4] for more information.
비동기 abort와 EH 인수
116-164시간 초과 뒤 `scsi_abort_command()`가 명령 abort를 예약한다. 성공하면 재시도 횟수가 남은 명령은 다시 실행하고, 모두 소진했으면 `DID_TIME_OUT`으로 끝낸다. abort가 실패하면 `scsi_eh_scmd_add()`가 명령을 EH로 넘긴다.
`scsi_eh_scmd_add()`는 `scmd->eh_entry`를 `shost->eh_cmd_q`에 연결하고, `shost->shost_state`의 `SHOST_RECOVERY` 비트를 세우며, `shost->host_failed`를 증가시킨다. `shost->host_busy == shost->host_failed`가 되면 SCSI EH 스레드를 깨운다.
`SHOST_RECOVERY`가 설정되면 블록 큐에서 호스트로 새 명령을 보내지 않는다. 결국 진행 중인 모든 명령이 정상 완료되거나 실패 또는 시간 초과로 `eh_cmd_q`에 들어간다. busy와 failed가 같아질 때 EH는 모든 진행 중 명령이 실패 목록에 있다고 기대할 수 있다.
다만 하위 계층이 정지했다는 뜻은 아니다. 오류 상태로 완료한 LLDD는 해당 `scmd`를 잊었다고 가정하지만, 시간 초과 명령은 `hostt->eh_timed_out()`이 하위 계층에서 제거하지 않는 한 여전히 활성일 수 있다. 뒤늦은 완료는 타이머가 만료되었으므로 무시된다.
1.3 Asynchronous command aborts
-------------------------------
After a timeout occurs a command abort is scheduled from
scsi_abort_command(). If the abort is successful the command
will either be retried (if the number of retries is not exhausted)
or terminated with DID_TIME_OUT.
Otherwise scsi_eh_scmd_add() is invoked for the command.
See [1-4] for more information.
1.4 How EH takes over
---------------------
scmds enter EH via scsi_eh_scmd_add(), which does the following.
1. Links scmd->eh_entry to shost->eh_cmd_q
2. Sets SHOST_RECOVERY bit in shost->shost_state
3. Increments shost->host_failed
4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
As can be seen above, once any scmd is added to shost->eh_cmd_q,
SHOST_RECOVERY shost_state bit is turned on. This prevents any new
scmd to be issued from blk queue to the host; eventually, all scmds on
the host either complete normally, fail and get added to eh_cmd_q, or
time out and get added to shost->eh_cmd_q.
If all scmds either complete or fail, the number of in-flight scmds
becomes equal to the number of failed scmds - i.e. shost->host_busy ==
shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
SCSI EH thread can expect that all in-flight commands have failed and
are linked on shost->eh_cmd_q.
Note that this does not mean lower layers are quiescent. If a LLDD
completed a scmd with error status, the LLDD and lower layers are
assumed to forget about the scmd at that point. However, if a scmd
has timed out, unless hostt->eh_timed_out() made lower layers forget
about the scmd, which currently no LLDD does, the command is still
active as long as lower layers are concerned and completion could
occur at any time. Of course, all such completions are ignored as the
timer has already expired.
We'll talk about how SCSI EH takes actions to abort - make LLDD
forget about - timed out scmds later.
두 EH 방식과 정상 동작 재개
165-195LLDD는 세분화된 EH 콜백을 제공해 중간 계층이 복구를 이끌게 하거나, 전체 오류 처리를 맡는 단일 `eh_strategy_handler()`를 제공할 수 있다. 후자는 중간 계층이 보통 수행하는 복구 유지 작업까지 모두 책임져야 한다.
복구 뒤 `scsi_restart_operations()`는 필요하면 도어를 잠그고, `SHOST_RECOVERY`를 지우고, `shost->host_wait` 대기자를 깨운 다음 호스트의 모든 장치 큐를 다시 가동한다.
2. How SCSI EH works
====================
LLDD's can implement SCSI EH actions in one of the following two
ways.
- Fine-grained EH callbacks
LLDD can implement fine-grained EH callbacks and let SCSI
midlayer drive error handling and call appropriate callbacks.
This will be discussed further in [2-1].
- eh_strategy_handler() callback
This is one big callback which should perform whole error
handling. As such, it should do all chores the SCSI midlayer
performs during recovery. This will be discussed in [2-2].
Once recovery is complete, SCSI EH resumes normal operation by
calling scsi_restart_operations(), which
1. Checks if door locking is needed and locks door.
2. Clears SHOST_RECOVERY shost_state bit
3. Wakes up waiters on shost->host_wait. This occurs if someone
calls scsi_block_when_processing_errors() on the host.
(*QUESTION* why is it needed? All operations will be blocked
anyway after it reaches blk queue.)
4. Kicks queues in all devices on the host in the asses
세분화된 콜백의 복구 원칙
196-254`eh_strategy_handler()`가 없으면 SCSI 중간 계층이 오류 처리를 주도한다. 목표는 하위 계층과 호스트, 장치가 시간 초과 `scmd`를 잊게 하고 새 명령을 처리할 준비를 갖추게 하는 것이다. 하위 계층이 명령을 잊고 해당 명령을 다시 처리하거나 실패시킬 준비가 되면 복구된 것으로 본다.
복구 동작은 심각도가 증가하는 순서로 진행한다. `eh_abort_handler`, `eh_device_reset_handler`, `eh_bus_reset_handler`, `eh_host_reset_handler`를 사용할 수 있고, 생략한 콜백은 항상 실패한 것으로 본다. 낮은 단계에서 남은 명령만 더 높은 단계로 올리며, 최상위 단계도 실패하면 복구되지 않은 장치를 오프라인으로 전환한다.
복구 대상은 `eh_work_q`에 있다. 한 동작이 성공한 `scmd`는 `scsi_eh_finish_cmd()`로 `eh_done_q`로 옮긴다. 장치 reset 하나가 그 장치의 여러 실패 명령을 한꺼번에 복구할 수 있다. 시간 초과 명령을 EH 명령으로 재사용하기 전에는 LLDD가 그 명령을 잊었는지 보장해야 한다.
`eh_work_q`가 비면 `scsi_eh_flush_done_q()`가 복구 명령을 재시도하거나 오류 완료로 상위 계층에 알린다. 장치가 온라인이고 `REQ_FAILFAST`가 없으며 증가한 `scmd->retries`가 `scmd->allowed`보다 작을 때만 재시도한다.
남은 실패 명령이 있을 때만 다음 단계로 진행한다.
2.1 EH through fine-grained callbacks
-------------------------------------
2.1.1 Overview
^^^^^^^^^^^^^^
If eh_strategy_handler() is not present, SCSI midlayer takes charge
of driving error handling. EH's goals are two - make LLDD, host and
device forget about timed out scmds and make them ready for new
commands. A scmd is said to be recovered if the scmd is forgotten by
lower layers and lower layers are ready to process or fail the scmd
again.
To achieve these goals, EH performs recovery actions with increasing
severity. Some actions are performed by issuing SCSI commands and
others are performed by invoking one of the following fine-grained
hostt EH callbacks. Callbacks may be omitted and omitted ones are
considered to fail always.
::
int (* eh_abort_handler)(struct scsi_cmnd *);
int (* eh_device_reset_handler)(struct scsi_cmnd *);
int (* eh_bus_reset_handler)(struct scsi_cmnd *);
int (* eh_host_reset_handler)(struct scsi_cmnd *);
Higher-severity actions are taken only when lower-severity actions
cannot recover some of failed scmds. Also, note that failure of the
highest-severity action means EH failure and results in offlining of
all unrecovered devices.
During recovery, the following rules are followed
- Recovery actions are performed on failed scmds on the to do list,
eh_work_q. If a recovery action succeeds for a scmd, recovered
scmds are removed from eh_work_q.
Note that single recovery action on a scmd can recover multiple
scmds. e.g. resetting a device recovers all failed scmds on the
device.
- Higher severity actions are taken iff eh_work_q is not empty after
lower severity actions are complete.
- EH reuses failed scmds to issue commands for recovery. For
timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
before reusing it for EH commands.
When a scmd is recovered, the scmd is moved from eh_work_q to EH
local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
either retry or error-finish (notify upper layer of failure) recovered
scmds.
scmds are retried iff its sdev is still online (not offlined during
EH), REQ_FAILFAST is not set and ++scmd->retries is less than
scmd->allowed.
EH 큐를 지나는 scmd 흐름
255-302오류 완료 또는 시간 초과 시 `scsi_eh_scmd_add()`는 호스트 잠금 아래 명령을 `shost->eh_cmd_q`에 넣고 `SHOST_RECOVERY`를 세우며 `host_failed`를 증가시킨다. EH가 시작되면 모든 명령을 로컬 `eh_work_q`로 옮겨 호스트 큐를 비운다.
명령이 복구되면 잠금 없이 `scsi_eh_finish_cmd()`가 `eh_work_q`에서 `eh_done_q`로 옮긴다. 큐 조작을 잠금 없이 유지하려면 각 독립 `eh_work_q`를 다루는 스레드는 최대 하나여야 한다.
완료 단계의 `scsi_eh_flush_done_q()`는 `eh_done_q`에서 명령을 제거하고 `scmd->eh_entry`를 지운다. 재시도가 필요하면 `scsi_queue_insert()`, 아니면 `scsi_finish_command()`를 호출하고 `shost->host_failed`를 0으로 만든다. 큐 삽입 또는 완료 함수가 필요한 잠금을 담당한다.
각 단계의 목록과 핵심 동작이다.
2.1.2 Flow of scmds through EH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Error completion / time out
:ACTION: scsi_eh_scmd_add() is invoked for scmd
- add scmd to shost->eh_cmd_q
- set SHOST_RECOVERY
- shost->host_failed++
:LOCKING: shost->host_lock
2. EH starts
:ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
is cleared.
:LOCKING: shost->host_lock (not strictly necessary, just for
consistency)
3. scmd recovered
:ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
- move from local eh_work_q to local eh_done_q
:LOCKING: none
:CONCURRENCY: at most one thread per separate eh_work_q to
keep queue manipulation lockless
4. EH completes
:ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
layer of failure. May be called concurrently but must have
a no more than one thread per separate eh_work_q to
manipulate the queue locklessly
- scmd is removed from eh_done_q and scmd->eh_entry is cleared
- if retry is necessary, scmd is requeued using
scsi_queue_insert()
- otherwise, scsi_finish_command() is invoked for scmd
- zero shost->host_failed
:LOCKING: queue or finish function performs appropriate locking
scsi_unjam_host와 sense 획득
303-348세분화된 콜백 방식은 `scsi_unjam_host()`에서 시작한다. `shost->host_lock`을 잡고 `shost->eh_cmd_q`를 로컬 `eh_work_q`로 splice한 뒤 잠금을 해제한다. 이 동작으로 호스트 EH 큐가 비워진다.
`scsi_eh_get_sense()`는 유효한 sense 데이터 없이 오류 완료된 각 명령에 `REQUEST_SENSE`를 보낸다. 대부분의 전송 계층과 LLDD는 성능과 sense 정보의 시점 일치를 위해 autosense를 제공하는 것이 권장된다.
autosense가 없으면 `scsi_done()`으로 오류 완료할 때 `scmd->sense_buffer`가 유효하지 않아 `scsi_decide_disposition()`이 `FAILED`를 반환한다. EH에서 sense를 얻은 뒤 다시 disposition을 판정한다. `SUCCESS`이면 retries를 allowed로 맞춰 추가 재시도를 막고 완료 큐로 옮기며, `NEEDS_RETRY`도 완료 큐로 옮긴다. 그 밖의 결과에는 동작하지 않아 더 강한 복구가 이어진다.
2.1.3 Flow of control
^^^^^^^^^^^^^^^^^^^^^^
EH through fine-grained callbacks start from scsi_unjam_host().
``scsi_unjam_host``
1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
cleared by this action.
2. Invoke scsi_eh_get_sense.
``scsi_eh_get_sense``
This action is taken for each error-completed
command without valid sense data. Most
SCSI transports/LLDDs automatically acquire sense data on
command failures (autosense). Autosense is recommended for
performance reasons and as sense information could get out of
sync between occurrence of CHECK CONDITION and this action.
Note that if autosense is not supported, scmd->sense_buffer
contains invalid sense data when error-completing the scmd
with scsi_done(). scsi_decide_disposition() always returns
FAILED in such cases thus invoking SCSI EH. When the scmd
reaches here, sense data is acquired and
scsi_decide_disposition() is called again.
1. Invoke scsi_request_sense() which issues REQUEST_SENSE
command. If fails, no action. Note that taking no action
causes higher-severity recovery to be taken for the scmd.
2. Invoke scsi_decide_disposition() on the scmd
- SUCCESS
scmd->retries is set to scmd->allowed preventing
scsi_eh_flush_done_q() from retrying the scmd and
scsi_eh_finish_cmd() is invoked.
- NEEDS_RETRY
scsi_eh_finish_cmd() invoked
- otherwise
No action.
장치를 준비시키는 단계
349-422`eh_work_q`가 비지 않았으면 `scsi_eh_ready_devs()`가 점점 강한 조치를 취한다. 먼저 유효한 sense가 있고 `scsi_check_sense()`가 `FAILED`로 판정한 오류 완료 명령마다 `START STOP UNIT`을 `start=1`로 보낸다. 이 경우 하위 계층이 이미 명령을 잊었다고 알 수 있어 `scmd`를 재사용할 수 있다.
STU가 성공하고 장치가 준비되었거나 오프라인이면 해당 장치의 모든 실패 명령을 EH 완료 처리한다. 원문은 abort 콜백이 없거나 실패한 경우 시간 초과 명령이 아직 하위 계층에 남을 수 있으므로, 그런 장치에는 STU를 제한해야 한다는 일관성 문제를 지적한다.
다음은 `eh_device_reset_handler()`를 쓰는 device reset이다. SCSI 명령을 발행하지 않고 reset이 장치의 모든 명령을 지우므로 오류 완료 명령만 고를 필요가 없다. 이어 채널별 `eh_bus_reset_handler()`, 호스트 전체의 `eh_host_reset_handler()`를 시도한다. 성공하면 해당 범위에서 준비되었거나 오프라인인 장치의 실패 명령을 EH 완료한다.
마지막까지 남은 명령이 있으면 `scsi_eh_offline_sdevs()`가 장치를 오프라인으로 만들고 명령을 완료 큐로 옮긴다. 모든 명령이 복구되거나 포기된 뒤 `scsi_eh_flush_done_q()`가 재시도 또는 상위 계층 오류 통지를 수행한다.
작업 큐가 남아 있을 때의 실제 호출 순서다.
4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
``scsi_eh_ready_devs``
This function takes four increasingly more severe measures to
make failed sdevs ready for new commands.
1. Invoke scsi_eh_stu()
``scsi_eh_stu``
For each sdev which has failed scmds with valid sense data
of which scsi_check_sense()'s verdict is FAILED,
START STOP UNIT command is issued w/ start=1. Note that
as we explicitly choose error-completed scmds, it is known
that lower layers have forgotten about the scmd and we can
reuse it for STU.
If STU succeeds and the sdev is either offline or ready,
all failed scmds on the sdev are EH-finished with
scsi_eh_finish_cmd().
*NOTE* If hostt->eh_abort_handler() isn't implemented or
failed, we may still have timed out scmds at this point
and STU doesn't make lower layers forget about those
scmds. Yet, this function EH-finish all scmds on the sdev
if STU succeeds leaving lower layers in an inconsistent
state. It seems that STU action should be taken only when
a sdev has no timed out scmd.
2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
``scsi_eh_bus_device_reset``
This action is very similar to scsi_eh_stu() except that,
instead of issuing STU, hostt->eh_device_reset_handler()
is used. Also, as we're not issuing SCSI commands and
resetting clears all scmds on the sdev, there is no need
to choose error-completed scmds.
3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
``scsi_eh_bus_reset``
hostt->eh_bus_reset_handler() is invoked for each channel
with failed scmds. If bus reset succeeds, all failed
scmds on all ready or offline sdevs on the channel are
EH-finished.
4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
``scsi_eh_host_reset``
This is the last resort. hostt->eh_host_reset_handler()
is invoked. If host reset succeeds, all failed scmds on
all ready or offline sdevs on the host are EH-finished.
5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
``scsi_eh_offline_sdevs``
Take all sdevs which still have unrecovered scmds offline
and EH-finish the scmds.
5. Invoke scsi_eh_flush_done_q().
``scsi_eh_flush_done_q``
At this point all scmds are recovered (or given up) and
put on eh_done_q by scsi_eh_finish_cmd(). This function
flushes eh_done_q by either retrying or notifying upper
layer of failure of the scmds.
eh_strategy_handler 책임과 진입 조건
423-448`transportt->eh_strategy_handler()`는 `scsi_unjam_host()` 대신 호출되며 전체 복구 과정을 책임진다. 반환할 때 하위 계층은 모든 실패 `scmd`를 잊어야 하고, 새 명령을 받을 준비가 되었거나 오프라인이어야 한다. 앞의 EH 흐름에서 오류를 큐에 넣는 최초 단계 외의 모든 작업을 구현해야 한다.
진입 시 각 실패 명령의 `eh_flags`가 적절히 설정되어 있고, `scmd->eh_entry`로 `shost->eh_cmd_q`에 연결되어 있다. `SHOST_RECOVERY`가 설정되어 있으며 `shost->host_failed == shost->host_busy`가 성립한다.
2.2 EH through transportt->eh_strategy_handler()
------------------------------------------------
transportt->eh_strategy_handler() is invoked in the place of
scsi_unjam_host() and it is responsible for whole recovery process.
On completion, the handler should have made lower layers forget about
all failed scmds and either ready for new commands or offline. Also,
it should perform SCSI EH maintenance chores to maintain integrity of
SCSI midlayer. IOW, of the steps described in [2-1-2], all steps
except for #1 must be implemented by eh_strategy_handler().
2.2.1 Pre transportt->eh_strategy_handler() SCSI midlayer conditions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following conditions are true on entry to the handler.
- Each failed scmd's eh_flags field is set appropriately.
- Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
- SHOST_RECOVERY is set.
- shost->host_failed == shost->host_busy
eh_strategy_handler 종료 조건
449-464핸들러가 반환할 때 `shost->host_failed`는 0, `shost->eh_cmd_q`는 빈 상태여야 하고 각 `scmd->eh_entry`도 지워져야 한다. 모든 실패 명령에 `scsi_queue_insert()` 또는 `scsi_finish_command()` 중 하나를 호출해야 한다. 핸들러는 `scmd->retries`와 `scmd->allowed`로 자체 재시도 횟수를 제한할 수 있다.
중간 계층 무결성을 지키기 위한 계약이다.
2.2.2 Post transportt->eh_strategy_handler() SCSI midlayer conditions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following conditions must be true on exit from the handler.
- shost->host_failed is zero.
- shost->eh_cmd_q is cleared.
- Each scmd->eh_entry is cleared.
- Either scsi_queue_insert() or scsi_finish_command() is called on
each scmd. Note that the handler is free to use scmd->retries and
->allowed to limit the number of retries.
구현 시 주의 사항과 문서 정보
465-485시간 초과 `scmd`는 하위 계층에서 여전히 활성일 수 있으므로 다른 작업이나 재사용 전에 반드시 하위 계층이 잊게 해야 한다. `shost` 데이터를 읽거나 바꿀 때는 일관성을 위해 `shost->host_lock`을 잡는다.
완료 시 실패한 각 `sdev`는 활성 `scmd`를 모두 잊은 상태여야 하며, 새 명령을 받을 준비가 되었거나 오프라인이어야 한다. 문서는 Tejun Heo가 작성했고 날짜는 2005년 9월 11일이다.
2.2.3 Things to consider
^^^^^^^^^^^^^^^^^^^^^^^^
- Know that timed out scmds are still active on lower layers. Make
lower layers forget about them before doing anything else with
those scmds.
- For consistency, when accessing/modifying shost data structure,
grab shost->host_lock.
- On completion, each failed sdev must have forgotten about all
active scmds.
- On completion, each failed sdev must be ready for new commands or
offline.
Tejun Heo
[email protected]
11th September 2005
요약·해설
scsi_eh.rst:1-485SCSI scsi_cmnd 완료, 시간 초과, 복구 큐와 LLDD 오류 처리 콜백의 전체 흐름을 설명합니다.