fwctl CXL driver

CXL command와 access policy

CXL specification은 CXL device 또는 switch mailbox에 보낼 command 집합과 vendor-specific command 공간을 정의합니다. `fwctl`은 kernel driver가 중재하는 허용된 mailbox command 집합을 userspace에서 device로 보내는 경로입니다.

CXL Feature 지원에는 CXL spec r3.1의 `Get Supported Features`(Opcode `0500h`), `Get Feature`(`0501h`), `Set Feature`(`0502h`) command를 사용합니다.

Kernel driver는 `Get Supported Features` return data에서 kernel이 금지하거나 독점 사용하는 feature를 제거할 수 있습니다. 수정할 수 없는 feature는 `Supported Feature Entry`의 `Set Feature Size`를 0으로 설정합니다. `Get Supported Features`와 `Get Feature`는 `FWCTL_RPC_CONFIGURATION` policy에 속합니다.

`Set Feature`가 immediate device change를 일으키며 effect mask가 immediate config, data, policy 또는 log change이면 access policy는 `FWCTL_RPC_DEBUG_WRITE_FULL`이어야 합니다. Cold reset 또는 conventional reset을 수반하는 config change라면 `FWCTL_RPC_DEBUG_WRITE` 이상이어야 합니다.

.. SPDX-License-Identifier: GPL-2.0

================
fwctl cxl driver
================

:Author: Dave Jiang

Overview
========

The CXL spec defines a set of commands that can be issued to the mailbox of a
CXL device or switch. It also left room for vendor specific commands to be
issued to the mailbox as well. fwctl provides a path to issue a set of allowed
mailbox commands from user space to the device moderated by the kernel driver.

The following 3 commands will be used to support CXL Features:
CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)

The "Get Supported Features" return data may be filtered by the kernel driver to
drop any features that are forbidden by the kernel or being exclusively used by
the kernel. The driver will set the "Set Feature Size" of the "Get Supported
Features Supported Feature Entry" to 0 to indicate that the Feature cannot be
modified. The "Get Supported Features" command and the "Get Features" falls
under the fwctl policy of FWCTL_RPC_CONFIGURATION.

For "Set Feature" command, the access policy currently is broken down into two
categories depending on the Set Feature effects reported by the device. If the
Set Feature will cause immediate change to the device, the fwctl access policy
must be FWCTL_RPC_DEBUG_WRITE_FULL. The effects for this level are
"immediate config change", "immediate data change", "immediate policy change",
or "immediate log change" for the set effects mask. If the effects are "config
change with cold reset" or "config change with conventional reset", then the
fwctl access policy must be FWCTL_RPC_DEBUG_WRITE or higher.

1. Driver info query

Application의 첫 단계는 `ioctl(FWCTL_CMD_INFO)` 호출입니다. 성공은 Features capability가 동작 중임을 뜻하며 all-zero 32-bit payload를 반환합니다.

`struct fwctl_info`를 채우고 `fwctl_info.out_device_type`을 `FWCTL_DEVICE_TYPE_CXL`로 설정해야 합니다. Return data는 all-zero reserved 32-bit field를 가진 `struct fwctl_info_cxl`이어야 합니다.

fwctl cxl User API
==================

.. kernel-doc:: include/uapi/fwctl/cxl.h

1. Driver info query
--------------------

First step for the app is to issue the ioctl(FWCTL_CMD_INFO). Successful
invocation of the ioctl implies the Features capability is operational and
returns an all zeros 32bit payload. A ``struct fwctl_info`` needs to be filled
out with the ``fwctl_info.out_device_type`` set to ``FWCTL_DEVICE_TYPE_CXL``.
The return data should be ``struct fwctl_info_cxl`` that contains a reserved
32bit field that should be all zeros.

2. Hardware command 전송

다음 단계는 userspace에서 `ioctl(FWCTL_RPC)`로 `Get Supported Features` command를 driver에 보내는 것입니다. `fwctl_rpc.in`은 `struct fwctl_rpc_cxl`을 가리키고, 그 `in_payload`는 CXL spec이 정의한 hardware input structure를 가리킵니다.

`fwctl_rpc.out`은 `struct fwctl_rpc_cxl_out`을 담는 buffer를 가리키며 hardware output data는 `fwctl_rpc_cxl_out.payload`에 inline됩니다. 이 command는 먼저 지원 feature 수를 얻고 다시 구체적인 feature detail을 얻기 위해 두 번 호출합니다.

Feature detail을 얻은 뒤 `Get Feature` 또는 `Set Feature` command를 구성해 보낼 수 있습니다. `Set Feature`의 feature info에는 발생할 effect가 담기므로 현재 system configuration이 해당 command를 허용하는지 userspace가 판단할 수 있습니다.

2. Send hardware commands
-------------------------

Next step is to send the 'Get Supported Features' command to the driver from
user space via ioctl(FWCTL_RPC). A ``struct fwctl_rpc_cxl`` is pointed to
by ``fwctl_rpc.in``. ``struct fwctl_rpc_cxl.in_payload`` points to
the hardware input structure that is defined by the CXL spec. ``fwctl_rpc.out``
points to the buffer that contains a ``struct fwctl_rpc_cxl_out`` that includes
the hardware output data inlined as ``fwctl_rpc_cxl_out.payload``. This command
is called twice. First time to retrieve the number of features supported.
A second time to retrieve the specific feature details as the output data.

After getting the specific feature details, a Get/Set Feature command can be
appropriately programmed and sent. For a "Set Feature" command, the retrieved
feature info contains an effects field that details the resulting
"Set Feature" command will trigger. That will inform the user whether
the system is configured to allowed the "Set Feature" command or not.

Get Feature C 예제

예제 `cxl_fwctl_rpc_get_test_feature()`는 input과 output buffer를 16-byte alignment로 allocate하고 0으로 초기화합니다. UUID와 requested size를 `cxl_mbox_get_feat_in`에 채웁니다.

`in->opcode`는 `CXL_MBOX_OPCODE_GET_FEATURE`, `in->op_size`는 input structure size로 설정합니다. `struct fwctl_rpc`에는 structure size, `FWCTL_RPC_CONFIGURATION` scope, input/output length와 userspace pointer를 기록합니다.

`send_command()` 성공 뒤 `out->payload`의 little-endian 32-bit value를 `le32toh()`로 변환해 expected data와 비교합니다. 불일치하면 `-ENXIO`를 반환하고 모든 allocation을 free합니다.

전체 userspace 예제는 CXL CLI test의 `https://github.com/pmem/ndctl/tree/main/test/fwctl.c`에서 확인할 수 있습니다. 원문의 C code는 symbol, type, label과 control flow를 바꾸지 않고 보존합니다.

Code example of a Get Feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: c

        static int cxl_fwctl_rpc_get_test_feature(int fd, struct test_feature *feat_ctx,
                                                  const uint32_t expected_data)
        {
                struct cxl_mbox_get_feat_in *feat_in;
                struct fwctl_rpc_cxl_out *out;
                struct fwctl_rpc rpc = {0};
                struct fwctl_rpc_cxl *in;
                size_t out_size, in_size;
                uint32_t val;
                void *data;
                int rc;

                in_size = sizeof(*in) + sizeof(*feat_in);
                rc = posix_memalign((void **)&in, 16, in_size);
                if (rc)
                        return -ENOMEM;
                memset(in, 0, in_size);
                feat_in = &in->get_feat_in;

                uuid_copy(feat_in->uuid, feat_ctx->uuid);
                feat_in->count = feat_ctx->get_size;

                out_size = sizeof(*out) + feat_ctx->get_size;
                rc = posix_memalign((void **)&out, 16, out_size);
                if (rc)
                        goto free_in;
                memset(out, 0, out_size);

                in->opcode = CXL_MBOX_OPCODE_GET_FEATURE;
                in->op_size = sizeof(*feat_in);

                rpc.size = sizeof(rpc);
                rpc.scope = FWCTL_RPC_CONFIGURATION;
                rpc.in_len = in_size;
                rpc.out_len = out_size;
                rpc.in = (uint64_t)(uint64_t *)in;
                rpc.out = (uint64_t)(uint64_t *)out;

                rc = send_command(fd, &rpc, out);
                if (rc)
                        goto free_all;

                data = out->payload;
                val = le32toh(*(__le32 *)data);
                if (memcmp(&val, &expected_data, sizeof(val)) != 0) {
                        rc = -ENXIO;
                        goto free_all;
                }

        free_all:
                free(out);
        free_in:
                free(in);
                return rc;
        }

Take a look at CXL CLI test directory
<https://github.com/pmem/ndctl/tree/main/test/fwctl.c> for a detailed user code
for examples on how to exercise this path.

Kernel API

Kernel API reference는 `drivers/cxl/core/features.c`에서 exported kernel-doc을, `include/cxl/features.h`에서 CXL feature declaration을 가져옵니다.

fwctl cxl Kernel API
====================

.. kernel-doc:: drivers/cxl/core/features.c
   :export:
.. kernel-doc:: include/cxl/features.h