PCI error recovery

목적과 적용 범위

저자는 Linas Vepstas, Richard Lary, Mike Mason입니다.

많은 PCI bus controller는 data·address bus의 parity error와 SERR·PERR 같은 hardware PCI error를 감지할 수 있습니다. PCIe chipset과 IBM Power4·Power5·Power6 기반 pSeries의 PCI host bridge 같은 고급 chipset은 이러한 error를 처리할 수 있습니다.

일반적인 조치는 영향받은 device를 disconnect하여 모든 I/O를 중단하는 것입니다. 이는 잘못된 address로 향하는 DMA처럼 system memory를 손상시킬 수 있는 상황을 막습니다.

대개 영향받은 PCI device를 reset하고 다시 동작시키는 reconnect mechanism도 제공합니다. Reset 단계에는 device driver와 PCI controller chip의 협력이 필요합니다.

이 문서는 bus disconnect를 device driver에 알리고 error recovery를 수행하는 generic API를 설명합니다. 이 API는 kernel 2.6.16 이후에 구현되어 있습니다.

.. SPDX-License-Identifier: GPL-2.0

==================
PCI Error Recovery
==================


:Authors: - Linas Vepstas <[email protected]>
          - Richard Lary <[email protected]>
          - Mike Mason <[email protected]>


Many PCI bus controllers are able to detect a variety of hardware
PCI errors on the bus, such as parity errors on the data and address
buses, as well as SERR and PERR errors.  Some of the more advanced
chipsets are able to deal with these errors; these include PCIe chipsets,
and the PCI-host bridges found on IBM Power4, Power5 and Power6-based
pSeries boxes. A typical action taken is to disconnect the affected device,
halting all I/O to it.  The goal of a disconnection is to avoid system
corruption; for example, to halt system memory corruption due to DMAs
to "wild" addresses. Typically, a reconnection mechanism is also
offered, so that the affected PCI device(s) are reset and put back
into working condition. The reset phase requires coordination
between the affected device drivers and the PCI controller chip.
This document describes a generic API for notifying device drivers
of a bus disconnection, and then performing error recovery.
This API is currently implemented in the 2.6.16 and later kernels.

보고와 복구 단계

PCI hardware error가 bus disconnect를 일으키면 multi-function card의 여러 driver instance를 포함해 영향받은 모든 device driver에 가능한 빨리 보고합니다.

이 알림으로 driver는 영원히 바뀌지 않을 I/O-space register를 기다리며 spinloop에서 deadlock되는 일을 피하고, 들어오는 I/O를 필요한 만큼 미룰 수 있습니다.

복구가 여러 단계인 가장 큰 이유는 여러 device driver가 연결된 multi-function device를 처리해야 하기 때문입니다. 첫 단계에서 각 driver는 단순 I/O 재활성화 또는 slot reset 중 원하는 reset 유형을 표시합니다.

Driver 하나라도 slot reset을 요청하면 slot reset을 수행합니다. Reset 또는 I/O 재활성화 뒤에는 모든 driver에 다시 알려 필요한 device setup·configuration을 수행하게 하고, 모두 끝나면 마지막으로 정상 동작 재개 event를 보냅니다.

Reporting and recovery is performed in several steps. First, when
a PCI hardware error has resulted in a bus disconnect, that event
is reported as soon as possible to all affected device drivers,
including multiple instances of a device driver on multi-function
cards. This allows device drivers to avoid deadlocking in spinloops,
waiting for some i/o-space register to change, when it never will.
It also gives the drivers a chance to defer incoming I/O as
needed.

Next, recovery is performed in several stages. Most of the complexity
is forced by the need to handle multi-function devices, that is,
devices that have multiple device drivers associated with them.
In the first stage, each driver is allowed to indicate what type
of reset it desires, the choices being a simple re-enabling of I/O
or requesting a slot reset.

If any driver requests a slot reset, that is what will be done.

After a reset and/or a re-enabling of I/O, all drivers are
again notified, so that they may then perform any device setup/config
that may be required.  After these have all completed, a final
"resume normal operations" event is sent out.

Kernel 기반 구현을 선택한 이유

User space가 아니라 kernel 기반 구현을 선택한 가장 큰 이유는 storage media에 연결된 PCI device, 특히 root filesystem을 가진 device의 bus disconnect를 처리해야 하기 때문입니다.

Root filesystem이 끊기면 user-space mechanism은 복구를 완료하기 위해 매우 복잡한 우회가 필요합니다. 현재 Linux filesystem 대부분은 기반 block device의 disconnect와 reconnect를 견디지 못합니다.

반면 bus error는 device driver에서 다루기 쉽고, 많은 driver가 이미 비슷한 복구 절차를 처리합니다. 예를 들어 SCSI generic layer에는 SCSI bus error와 bus reset을 다루는 상당한 mechanism이 있습니다.

The biggest reason for choosing a kernel-based implementation rather
than a user-space implementation was the need to deal with bus
disconnects of PCI devices attached to storage media, and, in particular,
disconnects from devices holding the root file system.  If the root
file system is disconnected, a user-space mechanism would have to go
through a large number of contortions to complete recovery. Almost all
of the current Linux file systems are not tolerant of disconnection
from/reconnection to their underlying block device. By contrast,
bus errors are easy to manage in the device driver. Indeed, most
device drivers already handle very similar recovery procedures;
for example, the SCSI-generic layer already provides significant
mechanisms for dealing with SCSI bus errors and SCSI bus resets.

pci_error_handlers callback

아래 설계와 구현 상세는 2005년 4월 5일 무렵 Ben Herrenschmidt와의 공개 email 논의를 바탕으로 합니다.

Error recovery API는 `struct pci_driver`의 새 field가 가리키는 function pointer 구조체로 driver에 노출됩니다. 이 구조체를 제공하지 않는 driver는 non-aware이며 실제 복구 단계는 platform에 따라 달라집니다. `arch/powerpc` 구현은 PCI hotplug remove/add를 모의합니다.

struct pci_error_handlers {
        int (*error_detected)(struct pci_dev *dev, pci_channel_state_t);
        int (*mmio_enabled)(struct pci_dev *dev);
        int (*slot_reset)(struct pci_dev *dev);
        void (*resume)(struct pci_dev *dev);
        void (*cor_error_detected)(struct pci_dev *dev);
};

Detailed Design
===============

Design and implementation details below, based on a chain of
public email discussions with Ben Herrenschmidt, circa 5 April 2005.

The error recovery API support is exposed to the driver in the form of
a structure of function pointers pointed to by a new field in struct
pci_driver. A driver that fails to provide the structure is "non-aware",
and the actual recovery steps taken are platform dependent.  The
arch/powerpc implementation will simulate a PCI hotplug remove/add.

This structure has the form::

        struct pci_error_handlers
        {
                int (*error_detected)(struct pci_dev *dev, pci_channel_state_t);
                int (*mmio_enabled)(struct pci_dev *dev);
                int (*slot_reset)(struct pci_dev *dev);
                void (*resume)(struct pci_dev *dev);
                void (*cor_error_detected)(struct pci_dev *dev);
        };

Channel state와 result code

`pci_channel_state_t`는 I/O channel이 정상인지, 차단됐는지, card가 완전히 죽었는지를 나타냅니다.

`enum pci_ers_result`는 callback이 복구 가능성과 필요한 조치를 platform에 반환하는 값입니다.

Driver가 모든 callback을 구현할 필요는 없지만 하나라도 구현하면 `error_detected()`는 반드시 구현해야 합니다. 구현하지 않은 callback의 기능은 미지원으로 간주합니다. 예를 들어 `mmio_enabled()`와 `resume()`이 없으면 복구에 필요하지 않은 것으로 봅니다. 보통 driver는 `slot_reset()`을 알고 싶어 합니다.

실제 platform 동작은 platform별로 다르지만 아래의 일반 순서를 따릅니다.

The possible channel states are::

        typedef enum {
                pci_channel_io_normal,  /* I/O channel is in normal state */
                pci_channel_io_frozen,  /* I/O to channel is blocked */
                pci_channel_io_perm_failure, /* PCI card is dead */
        } pci_channel_state_t;

Possible return values are::

        enum pci_ers_result {
                PCI_ERS_RESULT_NONE,        /* no result/none/not supported in device driver */
                PCI_ERS_RESULT_CAN_RECOVER, /* Device driver can recover without slot reset */
                PCI_ERS_RESULT_NEED_RESET,  /* Device driver wants slot to be reset. */
                PCI_ERS_RESULT_DISCONNECT,  /* Device has completely failed, is unrecoverable */
                PCI_ERS_RESULT_RECOVERED,   /* Device driver is fully recovered and operational */
        };

A driver does not have to implement all of these callbacks; however,
if it implements any, it must implement error_detected(). If a callback
is not implemented, the corresponding feature is considered unsupported.
For example, if mmio_enabled() and resume() aren't there, then it
is assumed that the driver does not need these callbacks
for recovery.  Typically a driver will want to know about
a slot_reset().

The actual steps taken by a platform to recover from a PCI error
event will be platform-dependent, but will follow the general
sequence described below.

STEP 0: Error event

PCI hardware가 PCI bus error를 감지합니다. PowerPC에서는 slot을 isolate하여 모든 I/O를 차단하고, 모든 read는 `0xffffffff`, 모든 write는 무시됩니다.

Downstream Port Containment를 지원하는 platform에서는 faulting device가 속한 sub-hierarchy의 link를 disable합니다. 그 sub-hierarchy의 모든 device가 접근 불가능해집니다. 규격 위치는 PCIe r7.0 section 6.2.11입니다.

STEP 0: Error Event
-------------------
A PCI bus error is detected by the PCI hardware.  On powerpc, the slot
is isolated, in that all I/O is blocked: all reads return 0xffffffff,
all writes are ignored.

Similarly, on platforms supporting Downstream Port Containment
(PCIe r7.0 sec 6.2.11), the link to the sub-hierarchy with the
faulting device is disabled. Any device in the sub-hierarchy
becomes inaccessible.

STEP 1: Notification

Platform은 error의 영향을 받는 모든 driver instance에 `error_detected()`를 호출합니다.

Platform에 따라 device가 이미 접근 불가능할 수 있습니다. Driver가 실패한 I/O로 error를 먼저 알아챘더라도 이 callback이 올바른 synchronization point입니다.

Driver는 pending timer 등 작업이 끝나기를 기다리고 cleanup할 수 있으며 semaphore를 잡거나 schedule할 수 있지만 device에는 접근하면 안 됩니다. Callback 안과 반환 뒤에는 새 I/O를 시작하지 않아야 합니다. Task context에서 호출되며 일종의 quiesce 지점입니다. Interrupt에 관한 주의는 문서 끝에 있습니다.

참여하는 모든 driver는 이 callback을 구현해야 합니다. Device가 error에도 사용 가능하면 `PCI_ERS_RESULT_RECOVERED`, I/O를 시도해 hardware를 복구하거나 diagnostic 정보를 얻고 싶으면 `PCI_ERS_RESULT_CAN_RECOVER`, slot reset 없이는 복구할 수 없으면 `PCI_ERS_RESULT_NEED_RESET`, 전혀 복구하지 않으려면 `PCI_ERS_RESULT_DISCONNECT`를 반환합니다.

Segment 또는 slot의 모든 driver가 `CAN_RECOVER`를 반환하면 platform이 slot I/O를 다시 enable하고 STEP 2로 갑니다. 하나라도 `NEED_RESET`이면 STEP 4로 가며, platform이 slot을 복구할 수 없으면 STEP 6으로 갑니다.

STEP 1: Notification
--------------------
Platform calls the error_detected() callback on every instance of
every driver affected by the error.

At this point, the device might not be accessible anymore, depending on
the platform (the slot will be isolated on powerpc). The driver may
already have "noticed" the error because of a failing I/O, but this
is the proper "synchronization point", that is, it gives the driver
a chance to cleanup, waiting for pending stuff (timers, whatever, etc...)
to complete; it can take semaphores, schedule, etc... everything but
touch the device. Within this function and after it returns, the driver
shouldn't do any new IOs. Called in task context. This is sort of a
"quiesce" point. See note about interrupts at the end of this doc.

All drivers participating in this system must implement this call.
The driver must return one of the following result codes:

  - PCI_ERS_RESULT_RECOVERED
      Driver returns this if it thinks the device is usable despite
      the error and does not need further intervention.
  - PCI_ERS_RESULT_CAN_RECOVER
      Driver returns this if it thinks it might be able to recover
      the HW by just banging IOs or if it wants to be given
      a chance to extract some diagnostic information (see
      mmio_enable, below).
  - PCI_ERS_RESULT_NEED_RESET
      Driver returns this if it can't recover without a
      slot reset.
  - PCI_ERS_RESULT_DISCONNECT
      Driver returns this if it doesn't want to recover at all.

The next step taken will depend on the result codes returned by the
drivers.

If all drivers on the segment/slot return PCI_ERS_RESULT_CAN_RECOVER,
then the platform should re-enable IOs on the slot (or do nothing in
particular, if the platform doesn't isolate slots), and recovery
proceeds to STEP 2 (MMIO Enable).

If any driver requested a slot reset (by returning PCI_ERS_RESULT_NEED_RESET),
then recovery proceeds to STEP 4 (Slot Reset).

If the platform is unable to recover the slot, the next step
is STEP 6 (Permanent Failure).

STEP 1 PowerPC 구현 주의

현재 PowerPC 구현은 이 routine에서 driver가 schedule하거나 semaphore를 사용하지 않는다고 가정합니다. Kernel thread 하나가 모든 device에 알리므로 device 하나가 sleep하거나 schedule하면 모두가 영향을 받습니다.

더 나은 구현은 모든 notification thread가 합류할 때까지 기다리는 복잡한 multi-thread logic이 필요하며, 문서는 이를 지나치게 복잡하고 구현 가치가 낮다고 봅니다.

현재 PowerPC에서는 이 시점의 I/O 여부를 크게 신경 쓰지 않습니다. Read는 `0xff`, write는 drop됩니다. Frozen adapter에 `EEH_MAX_FAILS`보다 많은 I/O를 시도하면 EEH가 driver의 infinite loop로 판단해 syslog에 error를 출력하며, device를 다시 동작시키려면 reboot해야 합니다.

.. note::

   The current powerpc implementation assumes that a device driver will
   *not* schedule or semaphore in this routine; the current powerpc
   implementation uses one kernel thread to notify all devices;
   thus, if one device sleeps/schedules, all devices are affected.
   Doing better requires complex multi-threaded logic in the error
   recovery implementation (e.g. waiting for all notification threads
   to "join" before proceeding with recovery.)  This seems excessively
   complex and not worth implementing.

   The current powerpc implementation doesn't much care if the device
   attempts I/O at this point, or not.  I/Os will fail, returning
   a value of 0xff on read, and writes will be dropped. If more than
   EEH_MAX_FAILS I/Os are attempted to a frozen adapter, EEH
   assumes that the device driver has gone into an infinite loop
   and prints an error to syslog.  A reboot is then required to
   get the device working again.

STEP 2: MMIO enabled

Platform은 device의 MMIO를 다시 enable하지만 보통 DMA는 enable하지 않고, 영향받은 모든 driver에 `mmio_enabled()`를 호출합니다.

이는 early recovery callback입니다. 제한된 I/O는 가능하지만 DMA는 불가능합니다. 정상 동작을 시작하는 지점이 아니라 device를 peek/poke하고 diagnostic 정보를 추출하거나 device-local reset을 trigger하는 데 사용합니다.

Segment의 모든 driver가 복구를 시도할 수 있다고 합의했고 hardware가 자동 link reset을 하지 않았을 때 호출합니다. Slot 또는 link reset 없이 I/O만 enable할 수 없다면 이 callback을 건너뛰고 STEP 3 또는 STEP 4로 갑니다.

AER platform에서는 STEP 1부터 faulting device가 접근 가능할 수 있지만 PowerPC EEH와 s390 호환성을 위해 STEP 2까지 접근을 미뤄야 합니다.

DPC platform은 STEP 3에서 sub-hierarchy link를 다시 enable하므로 device는 STEP 4까지 접근할 수 없습니다. Surprise Down 같은 error에서는 STEP 4에도 device가 접근 불가능할 수 있으며, read가 모두 1인지 `PCI_POSSIBLE_ERROR()`로 확인할 수 있습니다.

STEP 2: MMIO Enabled
--------------------
The platform re-enables MMIO to the device (but typically not the
DMA), and then calls the mmio_enabled() callback on all affected
device drivers.

This is the "early recovery" call. IOs are allowed again, but DMA is
not, with some restrictions. This is NOT a callback for the driver to
start operations again, only to peek/poke at the device, extract diagnostic
information, if any, and eventually do things like trigger a device local
reset or some such, but not restart operations. This callback is made if
all drivers on a segment agree that they can try to recover and if no automatic
link reset was performed by the HW. If the platform can't just re-enable IOs
without a slot reset or a link reset, it will not call this callback, and
instead will have gone directly to STEP 3 (Link Reset) or STEP 4 (Slot Reset).

.. note::

   On platforms supporting Advanced Error Reporting (PCIe r7.0 sec 6.2),
   the faulting device may already be accessible in STEP 1 (Notification).
   Drivers should nevertheless defer accesses to STEP 2 (MMIO Enabled)
   to be compatible with EEH on powerpc and with s390 (where devices are
   inaccessible until STEP 2).

   On platforms supporting Downstream Port Containment, the link to the
   sub-hierarchy with the faulting device is re-enabled in STEP 3 (Link
   Reset). Hence devices in the sub-hierarchy are inaccessible until
   STEP 4 (Slot Reset).

   For errors such as Surprise Down (PCIe r7.0 sec 6.2.7), the device
   may not even be accessible in STEP 4 (Slot Reset). Drivers can detect
   accessibility by checking whether reads from the device return all 1's
   (PCI_POSSIBLE_ERROR()).

STEP 2 제안과 결과 분기

아직 어떤 platform도 구현하지 않은 제안으로, 이 callback의 모든 I/O를 synchronous하게 수행하고 error는 일반 `pci_check_whatever()` API로 반환하며 여기서 생긴 error 때문에 새 `error_detected()`를 호출하지 않도록 합니다.

그러나 이런 error가 segment 전체 I/O를 다시 차단하면 같은 segment의 다른 device가 수행한 복구를 무효화할 수 있으므로 전체 segment가 link reset 또는 slot reset으로 넘어가야 할 수 있습니다.

Driver가 완전히 동작 가능하고 정상 operation을 시작할 준비가 됐다고 판단하면 `RECOVERED`를 반환합니다. 같은 segment의 다른 driver 실패가 slot reset을 일으킬 수 있어 실제 진행은 보장되지 않습니다.

현재 상태에서 복구할 수 없어 slot reset이 필요하면 `NEED_RESET`, reset 뒤에도 복구할 수 없는 완전 실패라면 `DISCONNECT`를 반환합니다.

모든 driver가 `RECOVERED`이면 STEP 3 Link Reset 또는 STEP 5 Resume Operations로 갑니다. 하나라도 `NEED_RESET`이면 STEP 4 Slot Reset으로 갑니다.

.. note::

   The following is proposed; no platform implements this yet:
   Proposal: All I/Os should be done _synchronously_ from within
   this callback, errors triggered by them will be returned via
   the normal pci_check_whatever() API, no new error_detected()
   callback will be issued due to an error happening here. However,
   such an error might cause IOs to be re-blocked for the whole
   segment, and thus invalidate the recovery that other devices
   on the same segment might have done, forcing the whole segment
   into one of the next states, that is, link reset or slot reset.

The driver should return one of the following result codes:
  - PCI_ERS_RESULT_RECOVERED
      Driver returns this if it thinks the device is fully
      functional and thinks it is ready to start
      normal driver operations again. There is no
      guarantee that the driver will actually be
      allowed to proceed, as another driver on the
      same segment might have failed and thus triggered a
      slot reset on platforms that support it.

  - PCI_ERS_RESULT_NEED_RESET
      Driver returns this if it thinks the device is not
      recoverable in its current state and it needs a slot
      reset to proceed.

  - PCI_ERS_RESULT_DISCONNECT
      Same as above. Total failure, no recovery even after
      reset driver dead. (To be defined more precisely)

The next step taken depends on the results returned by the drivers.
If all drivers returned PCI_ERS_RESULT_RECOVERED, then the platform
proceeds to either STEP 3 (Link Reset) or to STEP 5 (Resume Operations).

If any driver returned PCI_ERS_RESULT_NEED_RESET, then the platform
proceeds to STEP 4 (Slot Reset)

STEP 3: Link reset

Platform이 link를 reset합니다. 이는 PCIe 전용 단계이며 link reset으로 해결할 수 있는 fatal error를 감지했을 때 수행합니다.

STEP 3: Link Reset
------------------
The platform resets the link.  This is a PCIe specific step
and is done whenever a fatal error has been detected that can be
"solved" by resetting the link.

STEP 4: Slot reset 방식

`PCI_ERS_RESULT_NEED_RESET` 응답을 받으면 platform은 요청한 PCI device의 slot reset을 수행합니다. 실제 절차는 platform에 따라 다르며 완료 후 `slot_reset()` callback을 호출합니다.

PowerPC는 기본 soft reset과 선택적 fundamental reset의 두 수준을 구현합니다.

Soft reset은 adapter `#RST` line을 assert한 뒤 PCI BAR와 configuration header를 새 power-on 및 BIOS/system firmware 초기화 직후와 같은 상태로 복원합니다. Hot reset이라고도 합니다.

PCIe card에서만 지원하는 fundamental reset은 device state machine, hardware logic, port state, configuration register를 기본 상태로 초기화합니다.

대부분의 PCI device는 soft reset으로 충분하며, 선택적 fundamental reset은 soft reset으로 복구되지 않는 일부 PCIe device용입니다. PCI hotplug를 지원하면 slot 전원을 껐다 켜 reset할 수도 있습니다.

Platform은 PCI config space를 마지막 상태가 아니라 fresh power-on 상태로 복원해야 합니다. Driver가 표준 초기화 routine을 다시 사용하므로 비정상 config 상태는 device hang, kernel panic, silent data corruption을 일으킬 수 있습니다.

STEP 4: Slot Reset
------------------

In response to a return value of PCI_ERS_RESULT_NEED_RESET, the
platform will perform a slot reset on the requesting PCI device(s).
The actual steps taken by a platform to perform a slot reset
will be platform-dependent. Upon completion of slot reset, the
platform will call the device slot_reset() callback.

Powerpc platforms implement two levels of slot reset:
soft reset(default) and fundamental(optional) reset.

Powerpc soft reset consists of asserting the adapter #RST line and then
restoring the PCI BARs and PCI configuration header to a state
that is equivalent to what it would be after a fresh system
power-on followed by power-on BIOS/system firmware initialization.
Soft reset is also known as hot-reset.

Powerpc fundamental reset is supported by PCIe cards only
and results in device's state machines, hardware logic, port states and
configuration registers to initialize to their default conditions.

For most PCI devices, a soft reset will be sufficient for recovery.
Optional fundamental reset is provided to support a limited number
of PCIe devices for which a soft reset is not sufficient
for recovery.

If the platform supports PCI hotplug, then the reset might be
performed by toggling the slot electrical power off/on.

It is important for the platform to restore the PCI config space
to the "fresh poweron" state, rather than the "last state". After
a slot reset, the device driver will almost always use its standard
device initialization routines, and an unusual config space setup
may result in hung devices, kernel panics, or silent data corruption.

slot_reset() callback과 multi-function 조정

`slot_reset()`은 driver가 firmware를 다시 내려받는 등 hardware를 재초기화할 기회를 줍니다. Card는 fresh state이며 완전히 동작한다고 가정할 수 있습니다.

Slot은 unfrozen이고 PCI config space, MMIO, DMA에 완전히 접근할 수 있으며 Legacy·MSI·MSI-X interrupt도 사용할 수 있습니다.

이 시점에는 정상 I/O를 다시 시작하면 안 됩니다. 모든 driver가 callback 성공을 보고하면 platform이 `resume()`을 호출해 정상 I/O 재개를 허용합니다.

Reset 뒤에도 device를 동작시킬 수 없다면 critical failure를 반환할 수 있습니다. Soft reset을 이미 시도했다면 hard reset(power cycle) 뒤 `slot_reset()`을 다시 부를 수 있습니다. 그래도 실패하면 permanent failure이며 device는 dead로 간주합니다.

Multi-function card의 driver들은 one-shot 또는 global device initialization을 어느 instance가 수행할지 조정해야 합니다. Symbios `sym53cxx2` driver는 PCI function 0에서만 device를 초기화합니다.

if (PCI_FUNC(pdev->devfn) == 0)
        sym_reset_scsi_bus(np, 0);

This call gives drivers the chance to re-initialize the hardware
(re-download firmware, etc.).  At this point, the driver may assume
that the card is in a fresh state and is fully functional. The slot
is unfrozen and the driver has full access to PCI config space,
memory mapped I/O space and DMA. Interrupts (Legacy, MSI, or MSI-X)
will also be available.

Drivers should not restart normal I/O processing operations
at this point.  If all device drivers report success on this
callback, the platform will call resume() to complete the sequence,
and let the driver restart normal I/O processing.

A driver can still return a critical failure for this function if
it can't get the device operational after reset.  If the platform
previously tried a soft reset, it might now try a hard reset (power
cycle) and then call slot_reset() again.  If the device still can't
be recovered, there is nothing more that can be done;  the platform
will typically report a "permanent failure" in such a case.  The
device will be considered "dead" in this case.

Drivers for multi-function cards will need to coordinate among
themselves as to which driver instance will perform any "one-shot"
or global device initialization. For example, the Symbios sym53cxx2
driver performs device init only from PCI function 0::

        +       if (PCI_FUNC(pdev->devfn) == 0)
        +               sym_reset_scsi_bus(np, 0);

Fundamental reset 요청과 다음 단계

`slot_reset()`의 실패 결과는 `PCI_ERS_RESULT_DISCONNECT`입니다.

Fundamental reset이 필요한 PCIe card driver는 probe function에서 `struct pci_dev`의 `needs_freset` bit를 설정해야 합니다. QLogic `qla2xxx`는 특정 card type에 이를 설정합니다.

if (IS_QLA24XX(ha) || IS_QLA25XX(ha) || IS_QLA81XX(ha))
        pdev->needs_freset = 1;

Platform은 STEP 5 Resume Operations 또는 STEP 6 Permanent Failure로 갑니다.

현재 PowerPC 구현은 driver가 `PCI_ERS_RESULT_DISCONNECT`를 반환하면 power-cycle reset을 시도하지 않지만, 문서는 시도하는 편이 맞을 수 있다고 지적합니다.

Result codes:
        - PCI_ERS_RESULT_DISCONNECT
          Same as above.

Drivers for PCIe cards that require a fundamental reset must
set the needs_freset bit in the pci_dev structure in their probe function.
For example, the QLogic qla2xxx driver sets the needs_freset bit for certain
PCI card types::

        +        /* Set EEH reset type to fundamental if required by hba  */
        +        if (IS_QLA24XX(ha) || IS_QLA25XX(ha) || IS_QLA81XX(ha))
        +                pdev->needs_freset = 1;
        +

Platform proceeds either to STEP 5 (Resume Operations) or STEP 6 (Permanent
Failure).

.. note::

   The current powerpc implementation does not try a power-cycle
   reset if the driver returned PCI_ERS_RESULT_DISCONNECT.
   However, it probably should.

STEP 5: Resume operations

앞선 세 callback 중 하나에서 segment의 모든 driver가 `PCI_ERS_RESULT_RECOVERED`를 반환하면 platform은 영향받은 모든 driver에 `resume()`을 호출합니다.

이 callback은 모든 것이 다시 동작함을 알려 driver가 activity를 재개하게 하며 result code를 반환하지 않습니다.

이후 새 error가 발생하면 platform은 새로운 error recovery sequence를 처음부터 시작합니다.

STEP 5: Resume Operations
-------------------------
The platform will call the resume() callback on all affected device
drivers if all drivers on the segment have returned
PCI_ERS_RESULT_RECOVERED from one of the 3 previous callbacks.
The goal of this callback is to tell the driver to restart activity,
that everything is back and running. This callback does not return
a result code.

At this point, if a new error happens, the platform will restart
a new error recovery sequence.

STEP 6: Permanent failure

Platform이 device를 복구할 수 없는 permanent failure에서는 `pci_channel_io_perm_failure` state로 `error_detected()`를 호출합니다.

Driver는 최악을 가정해 pending I/O를 모두 취소하고 새 I/O를 거부하며 상위 layer에 `-EIO`를 반환해야 합니다. System shutdown 때처럼 memory를 정리하고 kernel operation에서 자신을 제거합니다.

Platform은 보통 system operator에게 permanent failure를 알립니다. Hotplug device라면 제거·교체할 수 있습니다.

모든 failure가 실제로 영구적인 것은 아닙니다. Overheating이나 card 접촉 불량일 수 있고, 잘못된 address DMA 또는 programming error로 생긴 bogus split transaction 같은 software bug가 많은 PCI error event를 일으킵니다.

실제 software error 원인 경험은 `Documentation/arch/powerpc/eeh-pci-error-recovery.rst`를 참조하십시오.

STEP 6: Permanent Failure
-------------------------
A "permanent failure" has occurred, and the platform cannot recover
the device.  The platform will call error_detected() with a
pci_channel_state_t value of pci_channel_io_perm_failure.

The device driver should, at this point, assume the worst. It should
cancel all pending I/O, refuse all new I/O, returning -EIO to
higher layers. The device driver should then clean up all of its
memory and remove itself from kernel operations, much as it would
during system shutdown.

The platform will typically notify the system operator of the
permanent failure in some way.  If the device is hotplug-capable,
the operator will probably want to remove and replace the device.
Note, however, not all failures are truly "permanent". Some are
caused by over-heating, some by a poorly seated card. Many
PCI error events are caused by software bugs, e.g. DMAs to
wild addresses or bogus split transactions due to programming
errors. See the discussion in Documentation/arch/powerpc/eeh-pci-error-recovery.rst
for additional detail on real-life experience of the causes of
software errors.

Platform policy

Callback 호출 방식은 platform policy입니다. Slot reset capability가 없는 platform은 복구하지 못하는 driver를 disconnect하고 같은 segment의 다른 card 복구를 시도할 수 있습니다.

다만 실제 환경에서는 segment당 driver가 하나뿐인 경우가 대부분입니다.

Conclusion; General Remarks
---------------------------
The way the callbacks are called is platform policy. A platform with
no slot reset capability may want to just "ignore" drivers that can't
recover (disconnect them) and try to let other cards on the same segment
recover. Keep in mind that in most real life cases, though, there will
be only one driver per segment.

Error 처리 중 interrupt 규칙

Device가 죽거나 isolate된 상태에서 interrupt가 들어오는 문제도 platform policy로 다룹니다. Recovery API는 두 가지만 요구합니다.

첫째, error detection부터 `slot_reset()` 호출 전까지 segment의 어느 device에서도 interrupt delivery가 계속된다는 보장은 없습니다. `slot_reset()` 시점에는 interrupt가 완전히 동작해야 합니다.

둘째, interrupt delivery가 중단된다는 보장도 없습니다. Error 감지 뒤 interrupt를 받거나 handler 안에서 error 때문에 interrupt source를 제대로 acknowledge하지 못한 driver는 `IRQ_NOTHANDLED`를 반환해야 합니다.

Platform은 보통 error 처리 동안 IRQ source를 mask합니다. Error-management 가능 slot으로 route되는 interrupt를 알고 해당 IRQ number를 임시 disable할 수 있어야 합니다.

Interrupt를 공유하는 다른 device에는 IRQ latency가 생기지만 다른 방법이 없습니다. High-end platform은 원래 많은 device가 interrupt를 공유하지 않아야 합니다.

Now, a note about interrupts. If you get an interrupt and your
device is dead or has been isolated, there is a problem :)
The current policy is to turn this into a platform policy.
That is, the recovery API only requires that:

 - There is no guarantee that interrupt delivery can proceed from any
   device on the segment starting from the error detection and until the
   slot_reset callback is called, at which point interrupts are expected
   to be fully operational.

 - There is no guarantee that interrupt delivery is stopped, that is,
   a driver that gets an interrupt after detecting an error, or that detects
   an error within the interrupt handler such that it prevents proper
   ack'ing of the interrupt (and thus removal of the source) should just
   return IRQ_NOTHANDLED. It's up to the platform to deal with that
   condition, typically by masking the IRQ source during the duration of
   the error handling. It is expected that the platform "knows" which
   interrupts are routed to error-management capable slots and can deal
   with temporarily disabling that IRQ number during error processing (this
   isn't terribly complex). That means some IRQ latency for other devices
   sharing the interrupt, but there is simply no other way. High end
   platforms aren't supposed to share interrupts between many devices
   anyway :)

PowerPC 상세와 driver 예

PowerPC platform 구현 상세는 `Documentation/arch/powerpc/eeh-pci-error-recovery.rst`에 있습니다.

작성 당시 error recovery patch가 있는 driver 목록은 아래와 같으며 모두 mainline에 들어간 것은 아닙니다. 구현 예로 사용할 수 있습니다.

.. note::

   Implementation details for the powerpc platform are discussed in
   the file Documentation/arch/powerpc/eeh-pci-error-recovery.rst

   As of this writing, there is a growing list of device drivers with
   patches implementing error recovery. Not all of these patches are in
   mainline yet. These may be used as "examples":

   - drivers/scsi/ipr
   - drivers/scsi/sym53c8xx_2
   - drivers/scsi/qla2xxx
   - drivers/scsi/lpfc
   - drivers/next/bnx2.c
   - drivers/next/e100.c
   - drivers/net/e1000
   - drivers/net/e1000e
   - drivers/net/ixgbe
   - drivers/net/cxgb3
   - drivers/net/s2io.c

Correctable error callback

Error severity가 `correctable`이면 `handle_error_source()`가 선택적 `cor_error_detected()` callback을 호출합니다. Driver가 원하면 추가 logging을 수행할 수 있습니다.

구현 예는 `drivers/cxl/pci.c`입니다. 이것으로 문서를 마칩니다.

   The cor_error_detected() callback is invoked in handle_error_source() when
   the error severity is "correctable". The callback is optional and allows
   additional logging to be done if desired. See example:

   - drivers/cxl/pci.c

The End
-------