SAS Layer

SCSI Core와 SAS LLDD 사이의 관리 계층

SAS Layer는 SAS LLDD를 관리하는 기반 계층이며 SCSI Core와 SAS LLDD 사이에 놓인다. SCSI Core가 SAM/SPC를, SAS LLDD와 sequencer가 phy·OOB·link 관리를 담당한다면 SAS Layer는 event, port, domain과 device 수명 주기를 담당한다.

구체적으로 LLDD가 만든 SAS Phy/Port/HA event 처리, SAS Port 생성·삭제, SAS Domain discovery·revalidation, domain device 관리, SCSI Host 등록·해제, SAS device의 SCSI Core 등록과 SATA device의 libata 등록, expander 관리 및 user space control export를 수행한다.

SAS LLDD는 PCI device driver이며 phy/OOB와 vendor-specific 작업을 처리하고 SAS Layer에 event를 발생시킨다. SAS Layer는 SAS 1.1 specification에 기술된 대부분의 SAS 작업을 수행한다.

`sas_ha_struct`는 SAS Layer에 LLDD를 설명한다. Hardware 초기화 뒤 `probe()`에서 `sas_register_ha()`를 호출하면 SCSI host와 sysfs SAS tree가 생성된다. 호출이 반환된 다음 phys를 enable해 OOB를 시작하며, 그때부터 driver가 `notify_*` event callback을 호출한다.

.. SPDX-License-Identifier: GPL-2.0

=========
SAS Layer
=========

The SAS Layer is a management infrastructure which manages
SAS LLDDs.  It sits between SCSI Core and SAS LLDDs.  The
layout is as follows: while SCSI Core is concerned with
SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
phy/OOB/link management, the SAS layer is concerned with:

      * SAS Phy/Port/HA event management (LLDD generates,
        SAS Layer processes),
      * SAS Port management (creation/destruction),
      * SAS Domain discovery and revalidation,
      * SAS Domain device management,
      * SCSI Host registration/unregistration,
      * Device registration with SCSI Core (SAS) or libata
        (SATA), and
      * Expander management and exporting expander control
        to user space.

A SAS LLDD is a PCI device driver.  It is concerned with
phy/OOB management, and vendor specific tasks and generates
events to the SAS layer.

The SAS Layer does most SAS tasks as outlined in the SAS 1.1
spec.

The sas_ha_struct describes the SAS LLDD to the SAS layer.
Most of it is used by the SAS Layer but a few fields need to
be initialized by the LLDDs.

After initializing your hardware, from the probe() function
you call sas_register_ha(). It will register your LLDD with
the SCSI subsystem, creating a SCSI host and it will
register your SAS driver with the sysfs SAS tree it creates.
It will then return.  Then you enable your phys to actually
start OOB (at which point your driver will start calling the
notify_* event callbacks).

Structure descriptions

struct sas_phy

`struct sas_phy`는 보통 driver 고유 phy structure에 정적으로 embed한다. HA structure는 이 고유 structure의 배열을 가지며, LLDD가 각 phy와 그 안의 `sas_phy`를 초기화한다. 일반적으로 phy는 LLDD가, port는 SAS Layer가 관리한다. 양쪽이 쓰는 field와 읽기만 하는 field를 나눠 불필요한 locking을 피한다.

`enabled`는 0 또는 1로, `id`는 `[0, MAX_PHYS)` 범위로 반드시 설정한다. `class`, `proto`, `type`, `role`, `oob_mode`, `linkrate`도 설정해야 한다. OOB가 끝난 뒤 `oob_mode`를 기록하고 SAS Layer에 알린다.

`sas_addr`는 보통 driver의 phy structure 안에 있는 SAS address 배열을 가리킨다. `attached_sas_addr`는 LLDD가 IDENTIFY 또는 FIS frame을 받은 뒤 SAS Layer에 알리기 전에 설정한다. Frame의 address를 복사하는 것이 바람직하지만, LLDD가 다른 address를 제공하거나 직접 연결된 SATA device용 address를 만들 수도 있고 discovery가 나중에 바꿀 수 있다.

`frame_rcvd`는 받은 IDENTIFY/FIS frame의 복사본을 가리킨다. Hardware frame 크기를 정확히 미리 알 수 없으므로 실제 배열은 고유 phy structure에 두고 pointer만 연결한다. Lock을 잡고 DMA 가능 memory에서 복사한 뒤 `frame_rcvd_size`를 설정하고 lock을 풀어 event를 호출한다.

`sas_prim`에는 받은 primitive를 기록한다. Lock을 잡고 값을 설정한 뒤 풀고 알린다. `port`는 해당 phy가 속한 `sas_port`를 가리키며 SAS Layer가 설정하고 LLDD는 읽기만 한다. `ha`는 LLDD가 설정할 수 있으나 SAS Layer도 설정한다.

`lldd_phy`는 callback이 `sas_phy`를 넘겼을 때 LLDD가 자신의 phy를 빠르게 찾도록 고유 phy를 가리키게 한다. Embed한 경우 `container_of`를 사용해도 된다.

======================

``struct sas_phy``
------------------

Normally this is statically embedded to your driver's
phy structure::

    struct my_phy {
            blah;
            struct sas_phy sas_phy;
            bleh;
    };

And then all the phys are an array of my_phy in your HA
struct (shown below).

Then as you go along and initialize your phys you also
initialize the sas_phy struct, along with your own
phy structure.

In general, the phys are managed by the LLDD and the ports
are managed by the SAS layer.  So the phys are initialized
and updated by the LLDD and the ports are initialized and
updated by the SAS layer.

There is a scheme where the LLDD can RW certain fields,
and the SAS layer can only read such ones, and vice versa.
The idea is to avoid unnecessary locking.

enabled
    - must be set (0/1)

id
    - must be set [0,MAX_PHYS)]

class, proto, type, role, oob_mode, linkrate
    - must be set

oob_mode
    - you set this when OOB has finished and then notify
      the SAS Layer.

sas_addr
    - this normally points to an array holding the sas
      address of the phy, possibly somewhere in your my_phy
      struct.

attached_sas_addr
    - set this when you (LLDD) receive an
      IDENTIFY frame or a FIS frame, _before_ notifying the SAS
      layer.  The idea is that sometimes the LLDD may want to fake
      or provide a different SAS address on that phy/port and this
      allows it to do this.  At best you should copy the sas
      address from the IDENTIFY frame or maybe generate a SAS
      address for SATA directly attached devices.  The Discover
      process may later change this.

frame_rcvd
    - this is where you copy the IDENTIFY/FIS frame
      when you get it; you lock, copy, set frame_rcvd_size and
      unlock the lock, and then call the event.  It is a pointer
      since there's no way to know your hw frame size _exactly_,
      so you define the actual array in your phy struct and let
      this pointer point to it.  You copy the frame from your
      DMAable memory to that area holding the lock.

sas_prim
    - this is where primitives go when they're
      received.  See sas.h. Grab the lock, set the primitive,
      release the lock, notify.

port
    - this points to the sas_port if the phy belongs
      to a port -- the LLDD only reads this. It points to the
      sas_port this phy is part of.  Set by the SAS Layer.

ha
    - may be set; the SAS layer sets it anyway.

lldd_phy
    - you should set this to point to your phy so you
      can find your way around faster when the SAS layer calls one
      of your callbacks and passes you a phy.  If the sas_phy is
      embedded you can also use container_of -- whatever you
      prefer.

struct sas_port

LLDD는 `struct sas_port`의 field를 설정하지 않고 읽기만 한다. `phy_mask`는 32-bit이며 현재 알려진 HA가 8개를 넘는 phys를 갖지 않으므로 충분하다. `lldd_port`는 LLDD가 내부 port representation을 유지하려 할 때 사용할 수 있는 pointer다.

``struct sas_port``
-------------------

The LLDD doesn't set any fields of this struct -- it only
reads them.  They should be self explanatory.

phy_mask is 32 bit, this should be enough for now, as I
haven't heard of a HA having more than 8 phys.

lldd_port
    - I haven't found use for that -- maybe other
      LLDD who wish to have internal port representation can make
      use of this.

struct sas_ha_struct와 callback 등록

`struct sas_ha_struct`는 보통 adapter를 설명하는 LLDD 고유 HA structure에 정적으로 embed한다. LLDD가 자체 port representation을 갖지 않으면 `sas_port sas_ports[MAX_PHYS]` 배열도 함께 둔다.

`pcidev`를 연결하고, `sas_addr`는 HA의 정적 SAS address 배열을 가리키게 한다. `sas_phy`는 phy structure pointer 배열, `sas_port`는 port structure pointer 배열이다. `num_phys`는 두 배열에 존재하는 phy 수이자 가능한 최대 port 수이며 별도의 `num_ports`는 쓰지 않는다.

LLDD는 `sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t)`와 `sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t)`로 class에 event를 알린다. SAS class는 port가 형성되거나 해체될 때 선택적으로 `lldd_port_formed`와 `lldd_port_deformed`를 호출한다.

SAS LLDD는 SAM의 Task Management Function 가운데 하나 이상을 구현해야 한다. 후보는 `lldd_abort_task`, `lldd_abort_task_set`, `lldd_clear_task_set`, `lldd_I_T_nexus_reset`, `lldd_lu_reset`, `lldd_query_task`이며 자세한 내용은 T10.org의 SAM을 따른다.

Port와 adapter 관리를 위해 `lldd_clear_nexus_port` 또는 `lldd_clear_nexus_ha` 중 하나 이상을 구현한다. Phy 관리는 `lldd_control_phy(struct sas_phy *, enum phy_func)`가 담당한다. `lldd_ha`는 고유 HA를 가리키게 하거나 embed 구조에서 `container_of`를 사용할 수 있다.

예제 `register_sas_ha()`는 `sas_addr`, `sas_phy`, `sas_port`, `num_phys`를 채운 뒤 port·device·task·TMF·nexus·phy callback을 연결하고 `sas_register_ha()`를 호출한다. 이 함수는 `probe()`의 마지막에, phys에서 OOB를 enable하기 전에 실행한다. SAS 1.1은 I_T Nexus Reset TMF를 정의하지 않으므로 예제의 `lldd_I_T_nexus_reset`은 `NULL`이다.

``struct sas_ha_struct``
------------------------

It normally is statically declared in your own LLDD
structure describing your adapter::

    struct my_sas_ha {
        blah;
        struct sas_ha_struct sas_ha;
        struct my_phy phys[MAX_PHYS];
        struct sas_port sas_ports[MAX_PHYS]; /* (1) */
        bleh;
    };

    (1) If your LLDD doesn't have its own port representation.

What needs to be initialized (sample function given below).

pcidev
^^^^^^

sas_addr
       - since the SAS layer doesn't want to mess with
         memory allocation, etc, this points to statically
         allocated array somewhere (say in your host adapter
         structure) and holds the SAS address of the host
         adapter as given by you or the manufacturer, etc.

sas_port
^^^^^^^^

sas_phy
      - an array of pointers to structures. (see
        note above on sas_addr).
        These must be set.  See more notes below.

num_phys
       - the number of phys present in the sas_phy array,
         and the number of ports present in the sas_port
         array.  There can be a maximum num_phys ports (one per
         port) so we drop the num_ports, and only use
         num_phys.

The event interface::

        /* LLDD calls these to notify the class of an event. */
        void sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t);
        void sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t);

The port notification::

        /* The class calls these to notify the LLDD of an event. */
        void (*lldd_port_formed)(struct sas_phy *);
        void (*lldd_port_deformed)(struct sas_phy *);

If the LLDD wants notification when a port has been formed
or deformed it sets those to a function satisfying the type.

A SAS LLDD should also implement at least one of the Task
Management Functions (TMFs) described in SAM::

        /* Task Management Functions. Must be called from process context. */
        int (*lldd_abort_task)(struct sas_task *);
        int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
        int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
        int (*lldd_I_T_nexus_reset)(struct domain_device *);
        int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
        int (*lldd_query_task)(struct sas_task *);

For more information please read SAM from T10.org.

Port and Adapter management::

        /* Port and Adapter management */
        int (*lldd_clear_nexus_port)(struct sas_port *);
        int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);

A SAS LLDD should implement at least one of those.

Phy management::

        /* Phy management */
        int (*lldd_control_phy)(struct sas_phy *, enum phy_func);

lldd_ha
    - set this to point to your HA struct. You can also
      use container_of if you embedded it as shown above.

A sample initialization and registration function
can look like this (called last thing from probe())
*but* before you enable the phys to do OOB::

    static int register_sas_ha(struct my_sas_ha *my_ha)
    {
            int i;
            static struct sas_phy   *sas_phys[MAX_PHYS];
            static struct sas_port  *sas_ports[MAX_PHYS];

            my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];

            for (i = 0; i < MAX_PHYS; i++) {
                    sas_phys[i] = &my_ha->phys[i].sas_phy;
                    sas_ports[i] = &my_ha->sas_ports[i];
            }

            my_ha->sas_ha.sas_phy  = sas_phys;
            my_ha->sas_ha.sas_port = sas_ports;
            my_ha->sas_ha.num_phys = MAX_PHYS;

            my_ha->sas_ha.lldd_port_formed = my_port_formed;

            my_ha->sas_ha.lldd_dev_found = my_dev_found;
            my_ha->sas_ha.lldd_dev_gone = my_dev_gone;

            my_ha->sas_ha.lldd_execute_task = my_execute_task;

            my_ha->sas_ha.lldd_abort_task     = my_abort_task;
            my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
            my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
            my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
            my_ha->sas_ha.lldd_lu_reset       = my_lu_reset;
            my_ha->sas_ha.lldd_query_task     = my_query_task;

            my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
            my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;

            my_ha->sas_ha.lldd_control_phy = my_control_phy;

            return sas_register_ha(&my_ha->sas_ha);
    }

(2) SAS 1.1 does not define I_T Nexus Reset TMF.

Event와 Execute Command SCSI RPC

Event는 SAS LLDD가 내부 또는 SAS domain의 변화를 SAS Layer에 알리는 유일한 방법이다. Phy event는 `PHYE_LOSS_OF_SIGNAL`, `PHYE_OOB_DONE`, `PHYE_OOB_ERROR`, `PHYE_SPINUP_HOLD`이고, phy를 통해 전달하는 port event는 `PORTE_BYTES_DMAED`, `PORTE_BROADCAST_RCVD`, `PORTE_LINK_RESET_ERR`, `PORTE_TIMER_EVENT`, `PORTE_HARD_RESET`이다. Host adapter event는 `HAE_RESET`이다.

LLDD는 C group에서 최소 하나를 생성할 수 있어야 하고, M으로 표시된 `PORTE_BYTES_DMAED`는 필수다. SAS Layer가 domain revalidation을 처리하게 하려면 E 표시의 `PORTE_BROADCAST_RCVD`도 생성해야 한다. 표시 없는 event는 선택 사항이다.

`HAE_RESET`은 HA 내부 오류와 reset, `PORTE_BYTES_DMAED`는 IDENTIFY/FIS 수신, `PORTE_BROADCAST_RCVD`는 primitive 수신을 뜻한다. `PORTE_LINK_RESET_ERR`는 timer 만료·signal 또는 DWS 손실, `PORTE_TIMER_EVENT`는 DWS reset timeout, `PORTE_HARD_RESET`은 Hard Reset primitive 수신을 뜻한다.

`PHYE_LOSS_OF_SIGNAL`은 device 제거, `PHYE_OOB_DONE`은 OOB 성공과 유효한 `oob_mode`, `PHYE_OOB_ERROR`는 OOB 실패와 가능한 연결 해제, `PHYE_SPINUP_HOLD`는 SATA가 있지만 COMWAKE를 보내지 않은 상태다. 각 disconnect 관련 event 전에 phy field를 적절히 설정·해제하거나 tasklet에서 helper `sas_phy_disconnected()`를 호출해야 한다.

`lldd_execute_task(struct sas_task *, gfp_t gfp_flags)`는 task를 LLDD에 queue하는 Execute Command SCSI RPC다. 호출되면 command가 즉시 transport로 나가야 하며 SAS LLDD의 어느 계층에도 자체 queue가 있어서는 안 된다. `-SAS_QUEUE_FULL` 또는 `-ENOMEM`은 아무 task도 queue되지 않았음을, 0은 task가 queue됐음을 뜻한다.

`sas_task`는 목적지 `dev`, `enum sas_proto` 하나인 `task_proto`, scatter-gather 배열과 원소 수인 `scatter`·`num_scatter`, 예상 총 byte인 `total_xfer_len`, `PCI_DMA_...` 방향인 `data_dir`, 완료 callback `task_done`을 가진다.

Events
======

Events are **the only way** a SAS LLDD notifies the SAS layer
of anything.  There is no other method or way a LLDD to tell
the SAS layer of anything happening internally or in the SAS
domain.

Phy events::

        PHYE_LOSS_OF_SIGNAL, (C)
        PHYE_OOB_DONE,
        PHYE_OOB_ERROR,      (C)
        PHYE_SPINUP_HOLD.

Port events, passed on a _phy_::

        PORTE_BYTES_DMAED,      (M)
        PORTE_BROADCAST_RCVD,   (E)
        PORTE_LINK_RESET_ERR,   (C)
        PORTE_TIMER_EVENT,      (C)
        PORTE_HARD_RESET.

Host Adapter event:
        HAE_RESET

A SAS LLDD should be able to generate

        - at least one event from group C (choice),
        - events marked M (mandatory) are mandatory (only one),
        - events marked E (expander) if it wants the SAS layer
          to handle domain revalidation (only one such).
        - Unmarked events are optional.

Meaning:

HAE_RESET
    - when your HA got internal error and was reset.

PORTE_BYTES_DMAED
    - on receiving an IDENTIFY/FIS frame

PORTE_BROADCAST_RCVD
    - on receiving a primitive

PORTE_LINK_RESET_ERR
    - timer expired, loss of signal, loss of DWS, etc. [1]_

PORTE_TIMER_EVENT
    - DWS reset timeout timer expired [1]_

PORTE_HARD_RESET
    - Hard Reset primitive received.

PHYE_LOSS_OF_SIGNAL
    - the device is gone [1]_

PHYE_OOB_DONE
    - OOB went fine and oob_mode is valid

PHYE_OOB_ERROR
    - Error while doing OOB, the device probably
      got disconnected. [1]_

PHYE_SPINUP_HOLD
    - SATA is present, COMWAKE not sent.

.. [1] should set/clear the appropriate fields in the phy,
       or alternatively call the inlined sas_phy_disconnected()
       which is just a helper, from their tasklet.

The Execute Command SCSI RPC::

        int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);

Used to queue a task to the SAS LLDD.  @task is the task to be executed.
@gfp_mask is the gfp_mask defining the context of the caller.

This function should implement the Execute Command SCSI RPC,

That is, when lldd_execute_task() is called, the command
go out on the transport *immediately*.  There is *no*
queuing of any sort and at any level in a SAS LLDD.

Returns:

   * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
   * 0, the task(s) were queued.

::

    struct sas_task {
            dev -- the device this task is destined to
            task_proto -- _one_ of enum sas_proto
            scatter -- pointer to scatter gather list array
            num_scatter -- number of elements in scatter
            total_xfer_len -- total number of bytes expected to be transferred
            data_dir -- PCI_DMA_...
            task_done -- callback when the task has finished execution
    };

SAS domain discovery와 sysfs tree

Sysfs tree는 현재 SAS domain의 물리적 배치와 discovery 당시의 일부 device parameter를 보여준다. `tree(1)` program은 이 구조를 보는 데 유용하며 원문은 `ftp://mama.indstate.edu/linux/tree/`를 안내한다. User space가 이를 바탕으로 graphical interface를 만들 수 있다.

Sysfs domain tree는 현재 연결 상태를 보여주지만 READY LED MEANING 같은 설정의 의미를 바꾼 뒤 그 내부 상태 이력을 보존하지 않는다. Command set driver와 user space가 내부 device state 변화를 책임진다.

Device를 domain에서 분리하면 sysfs tree에 즉시 반영되고 system에서 제거된다. `domain_device`는 SAS domain의 모든 device를 나타내며 SAS Layer가 전적으로 생성·관리·삭제한다. Task는 이 structure를 가리켜 LLDD에 목적지를 알려주며 LLDD는 내용을 읽기만 한다.

Discovery
=========

The sysfs tree has the following purposes:

    a) It shows you the physical layout of the SAS domain at
       the current time, i.e. how the domain looks in the
       physical world right now.
    b) Shows some device parameters _at_discovery_time_.

This is a link to the tree(1) program, very useful in
viewing the SAS domain:
ftp://mama.indstate.edu/linux/tree/

I expect user space applications to actually create a
graphical interface of this.

That is, the sysfs domain tree doesn't show or keep state if
you e.g., change the meaning of the READY LED MEANING
setting, but it does show you the current connection status
of the domain device.

Keeping internal device state changes is responsibility of
upper layers (Command set drivers) and user space.

When a device or devices are unplugged from the domain, this
is reflected in the sysfs tree immediately, and the device(s)
removed from the system.

The structure domain_device describes any device in the SAS
domain.  It is completely managed by the SAS layer.  A task
points to a domain device, this is how the SAS LLDD knows
where to send the task(s) to.  A SAS LLDD only reads the
contents of the domain_device structure, but it never creates
or destroys one.

User space expander management

각 sysfs expander directory의 `smp_portal`은 binary sysfs attribute이며 SMP port 자체가 아니라 SMP portal이다. User space application은 이 파일에 SMP request를 보내고 response를 받을 수 있다.

먼저 SAS specification 형식으로 SMP frame을 만들되 CRC field는 0으로 둔다. `open(2)`로 expander의 portal을 RW mode로 열고, `write(2)`로 frame을 쓴 뒤, `read(2)`로 기대한 response byte 수를 읽고 `close(2)`로 닫는다. 실제 수신 길이가 기대와 다르면 오류가 발생한 것이다.

전체 절차는 `expander_conf.c`의 `do_smp_func()`와 caller에 구현되어 있고 kernel 쪽은 `sas_expander.c`에 있다. `expander_conf.c` program은 SMP portal sysfs filename 하나를 받아 routing table을 포함한 expander 정보를 표시한다.

SMP portal은 expander를 완전히 제어할 수 있으므로 신중하게 사용해야 한다.

Expander management from User Space
===================================

In each expander directory in sysfs, there is a file called
"smp_portal".  It is a binary sysfs attribute file, which
implements an SMP portal (Note: this is *NOT* an SMP port),
to which user space applications can send SMP requests and
receive SMP responses.

Functionality is deceptively simple:

1. Build the SMP frame you want to send. The format and layout
   is described in the SAS spec.  Leave the CRC field equal 0.

open(2)

2. Open the expander's SMP portal sysfs file in RW mode.

write(2)

3. Write the frame you built in 1.

read(2)

4. Read the amount of data you expect to receive for the frame you built.
   If you receive different amount of data you expected to receive,
   then there was some kind of error.

close(2)

All this process is shown in detail in the function do_smp_func()
and its callers, in the file "expander_conf.c".

The kernel functionality is implemented in the file
"sas_expander.c".

The program "expander_conf.c" implements this. It takes one
argument, the sysfs file name of the SMP portal to the
expander, and gives expander information, including routing
tables.

The SMP portal gives you complete control of the expander,
so please be careful.