Chelsio S3 iSCSI Driver

T3 ASIC의 digest, DDP와 PDU 복구

Chelsio T3 ASIC 기반 S310·S320·S302·S304·Mezzanine adapter는 iSCSI acceleration과 Direct Data Placement를 지원한다. Hardware가 CRC처럼 byte를 직접 다루는 비용과 최종 host memory로의 DMA를 맡는다.

송신 때 S3 hardware는 iSCSI PDU header·data digest를 계산해 삽입한다. 수신 때 두 digest를 계산하고 검증한다. DDP는 Data-In의 Initiator Task Tag `ITT` 또는 Data-Out의 Target Task Tag `TTT`를 이용해 payload를 미리 게시한 최종 host buffer에 직접 둔다.

송신 경로는 완전한 PDU를 받아 digest 삽입, TCP segmentation, wire 전송과 필요 시 retransmission을 처리한다. 수신 경로는 TCP segment를 PDU로 재조립하고 header와 data를 분리해 digest를 검증한다. 가능한 payload는 DDP buffer로, 그렇지 않으면 header와 함께 host로 보낸다.

`cxgb3i`는 open-iscsi initiator transport와 연결되어 적용 가능한 traffic을 Chelsio hardware 가속 경로로 보낸다.

.. SPDX-License-Identifier: GPL-2.0

=================================
Chelsio S3 iSCSI Driver for Linux
=================================

Introduction
============

The Chelsio T3 ASIC based Adapters (S310, S320, S302, S304, Mezz cards, etc.
series of products) support iSCSI acceleration and iSCSI Direct Data Placement
(DDP) where the hardware handles the expensive byte touching operations, such
as CRC computation and verification, and direct DMA to the final host memory
destination:

        - iSCSI PDU digest generation and verification

          On transmitting, Chelsio S3 h/w computes and inserts the Header and
          Data digest into the PDUs.
          On receiving, Chelsio S3 h/w computes and verifies the Header and
          Data digest of the PDUs.

        - Direct Data Placement (DDP)

          S3 h/w can directly place the iSCSI Data-In or Data-Out PDU's
          payload into pre-posted final destination host-memory buffers based
          on the Initiator Task Tag (ITT) in Data-In or Target Task Tag (TTT)
          in Data-Out PDUs.

        - PDU Transmit and Recovery

          On transmitting, S3 h/w accepts the complete PDU (header + data)
          from the host driver, computes and inserts the digests, decomposes
          the PDU into multiple TCP segments if necessary, and transmit all
          the TCP segments onto the wire. It handles TCP retransmission if
          needed.

          On receiving, S3 h/w recovers the iSCSI PDU by reassembling TCP
          segments, separating the header and data, calculating and verifying
          the digests, then forwarding the header to the host. The payload data,
          if possible, will be directly placed into the pre-posted host DDP
          buffer. Otherwise, the payload data will be sent to the host too.

Module과 open-iscsi interface 설정

`modprobe cxgb3i`로 module을 load하면 open-iscsi에 `cxgb3i` transport class를 등록한다. Kernel configuration에서는 Device Drivers, SCSI device support, SCSI low-level drivers 아래 `Chelsio S3xx iSCSI support`를 module로 선택한다.

`/etc/iscsi/ifaces/` 아래 interface file에 `iface.transport_name = cxgb3i`, `iface.net_ifacename = <ethX>`, `iface.ipaddress = <iscsi ip address>`를 쓴다. IP를 지정하면 ethX와 같거나 같은 subnet이어야 하고 network에서 고유해야 한다.

The cxgb3i driver interfaces with open-iscsi initiator and provides the iSCSI
acceleration through Chelsio hardware wherever applicable.

Using the cxgb3i Driver
=======================

The following steps need to be taken to accelerates the open-iscsi initiator:

1. Load the cxgb3i driver: "modprobe cxgb3i"

   The cxgb3i module registers a new transport class "cxgb3i" with open-iscsi.

   * in the case of recompiling the kernel, the cxgb3i selection is located at::

        Device Drivers
                SCSI device support --->
                        [*] SCSI low-level drivers  --->
                                <M>   Chelsio S3xx iSCSI support

2. Create an interface file located under /etc/iscsi/ifaces/ for the new
   transport class "cxgb3i".

   The content of the file should be in the following format::

        iface.transport_name = cxgb3i
        iface.net_ifacename = <ethX>
        iface.ipaddress = <iscsi ip address>

   * if iface.ipaddress is specified, <iscsi ip address> needs to be either the
     same as the ethX's ip address or an address on the same subnet. Make

Receive segment 제한과 iscsiadm 선택

`/etc/iscsi/iscsid.conf`의 기본 `MaxRecvDataSegmentLength=131072`는 너무 크다. 15360 이하, 예를 들어 `node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192`로 바꾼다.

값이 너무 크면 normal session login이 실패하고 dmesg에 `cxgb3i: ERR! MaxRecvSegmentLength <X> too big. Need to be <= <Y>.` 형식의 오류가 기록된다.

대부분의 `iscsiadm` command에 `-I <iface file name>`을 넣어 step 2에서 만든 transport interface를 지정해야 open-iscsi traffic이 cxgb3i 가속 경로를 사용한다.

     sure the ip address is unique in the network.

3. edit /etc/iscsi/iscsid.conf
   The default setting for MaxRecvDataSegmentLength (131072) is too big;
   replace with a value no bigger than 15360 (for example 8192)::

        node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192

   * The login would fail for a normal session if MaxRecvDataSegmentLength is
     too big.  A error message in the format of
     "cxgb3i: ERR! MaxRecvSegmentLength <X> too big. Need to be <= <Y>."
     would be logged to dmesg.

4. To direct open-iscsi traffic to go through cxgb3i's accelerated path,
   "-I <iface file name>" option needs to be specified with most of the
   iscsiadm command. <iface file name> is the transport interface file created
   in step 2.