요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
==================================
vfio-ccw: the basic infrastructure
==================================
Introduction
------------
Here we describe the vfio support for I/O subchannel devices for
Linux/s390. Motivation for vfio-ccw is to passthrough subchannels to a
virtual machine, while vfio is the means.
Different than other hardware architectures, s390 has defined a unified
I/O access method, which is so called Channel I/O. It has its own access
patterns:
- Channel programs run asynchronously on a separate (co)processor.
- The channel subsystem will access any memory designated by the caller
in the channel program directly, i.e. there is no iommu involved.
Thus when we introduce vfio support for these devices, we realize it
with a mediated device (mdev) implementation. The vfio mdev will be
added to an iommu group, so as to make itself able to be managed by the
vfio framework. And we add read/write callbacks for special vfio I/O
regions to pass the channel programs from the mdev to its parent device
(the real I/O subchannel device) to do further address translation and
to perform I/O instructions.
This document does not intend to explain the s390 I/O architecture in
every detail. More information/reference could be found here:
- A good start to know Channel I/O in general:
https://en.wikipedia.org/wiki/Channel_I/O
- s390 architecture:
s390 Principles of Operation manual (IBM Form. No. SA22-7832)
- The existing QEMU code which implements a simple emulated channel
subsystem could also be a good reference. It makes it easier to follow
the flow.
qemu/hw/s390x/css.c
For vfio mediated device framework:
- Documentation/driver-api/vfio-mediated-device.rst
Motivation of vfio-ccw
----------------------
Typically, a guest virtualized via QEMU/KVM on s390 only sees
paravirtualized virtio devices via the "Virtio Over Channel I/O
(virtio-ccw)" transport. This makes virtio devices discoverable via
standard operating system algorithms for handling channel devices.
However this is not enough. On s390 for the majority of devices, which
use the standard Channel I/O based mechanism, we also need to provide
the functionality of passing through them to a QEMU virtual machine.
This includes devices that don't have a virtio counterpart (e.g. tape
drives) or that have specific characteristics which guests want to
exploit.
For passing a device to a guest, we want to use the same interface as
everybody else, namely vfio. We implement this vfio support for channel
devices via the vfio mediated device framework and the subchannel device
driver "vfio_ccw".
Access patterns of CCW devices
------------------------------
s390 architecture has implemented a so called channel subsystem, that
provides a unified view of the devices physically attached to the
systems. Though the s390 hardware platform knows about a huge variety of
different peripheral attachments like disk devices (aka. DASDs), tapes,
communication controllers, etc. They can all be accessed by a well
defined access method and they are presenting I/O completion a unified
way: I/O interruptions.
All I/O requires the use of channel command words (CCWs). A CCW is an
instruction to a specialized I/O channel processor. A channel program is
a sequence of CCWs which are executed by the I/O channel subsystem. To
issue a channel program to the channel subsystem, it is required to
build an operation request block (ORB), which can be used to point out
the format of the CCW and other control information to the system. The
operating system signals the I/O channel subsystem to begin executing
the channel program with a SSCH (start sub-channel) instruction. The
central processor is then free to proceed with non-I/O instructions
until interrupted. The I/O completion result is received by the
interrupt handler in the form of interrupt response block (IRB).
Back to vfio-ccw, in short:
- ORBs and channel programs are built in guest kernel (with guest
physical addresses).
- ORBs and channel programs are passed to the host kernel.
- Host kernel translates the guest physical addresses to real addresses
and starts the I/O with issuing a privileged Channel I/O instruction
(e.g SSCH).
- channel programs run asynchronously on a separate processor.
- I/O completion will be signaled to the host with I/O interruptions.
And it will be copied as IRB to user space to pass it back to the
guest.
Physical vfio ccw device and its child mdev
-------------------------------------------
As mentioned above, we realize vfio-ccw with a mdev implementation.
Channel I/O does not have IOMMU hardware support, so the physical
vfio-ccw device does not have an IOMMU level translation or isolation.
Subchannel I/O instructions are all privileged instructions. When
handling the I/O instruction interception, vfio-ccw has the software
policing and translation how the channel program is programmed before
it gets sent to hardware.
Within this implementation, we have two drivers for two types of
devices:
- The vfio_ccw driver for the physical subchannel device.
This is an I/O subchannel driver for the real subchannel device. It
realizes a group of callbacks and registers to the mdev framework as a
parent (physical) device. As a consequence, mdev provides vfio_ccw a
generic interface (sysfs) to create mdev devices. A vfio mdev could be
created by vfio_ccw then and added to the mediated bus. It is the vfio
device that added to an IOMMU group and a vfio group.
vfio_ccw also provides an I/O region to accept channel program
request from user space and store I/O interrupt result for user
space to retrieve. To notify user space an I/O completion, it offers
an interface to setup an eventfd fd for asynchronous signaling.
- The vfio_mdev driver for the mediated vfio ccw device.
This is provided by the mdev framework. It is a vfio device driver for
the mdev that created by vfio_ccw.
It realizes a group of vfio device driver callbacks, adds itself to a
vfio group, and registers itself to the mdev framework as a mdev
driver.
It uses a vfio iommu backend that uses the existing map and unmap
ioctls, but rather than programming them into an IOMMU for a device,
it simply stores the translations for use by later requests. This
means that a device programmed in a VM with guest physical addresses
can have the vfio kernel convert that address to process virtual
address, pin the page and program the hardware with the host physical
address in one step.
For a mdev, the vfio iommu backend will not pin the pages during the
VFIO_IOMMU_MAP_DMA ioctl. Mdev framework will only maintain a database
of the iova<->vaddr mappings in this operation. And they export a
vfio_pin_pages and a vfio_unpin_pages interfaces from the vfio iommu
backend for the physical devices to pin and unpin pages by demand.
Below is a high Level block diagram::
+-------------+
| |
| +---------+ | mdev_register_driver() +--------------+
| | Mdev | +<-----------------------+ |
| | bus | | | vfio_mdev.ko |
| | driver | +----------------------->+ |<-> VFIO user
| +---------+ | probe()/remove() +--------------+ APIs
| |
| MDEV CORE |
| MODULE |
| mdev.ko |
| +---------+ | mdev_register_parent() +--------------+
| |Physical | +<-----------------------+ |
| | device | | | vfio_ccw.ko |<-> subchannel
| |interface| +----------------------->+ | device
| +---------+ | callback +--------------+
+-------------+
The process of how these work together.
1. vfio_ccw.ko drives the physical I/O subchannel, and registers the
physical device (with callbacks) to mdev framework.
When vfio_ccw probing the subchannel device, it registers device
pointer and callbacks to the mdev framework. Mdev related file nodes
under the device node in sysfs would be created for the subchannel
device, namely 'mdev_create', 'mdev_destroy' and
'mdev_supported_types'.
2. Create a mediated vfio ccw device.
Use the 'mdev_create' sysfs file, we need to manually create one (and
only one for our case) mediated device.
3. vfio_mdev.ko drives the mediated ccw device.
vfio_mdev is also the vfio device driver. It will probe the mdev and
add it to an iommu_group and a vfio_group. Then we could pass through
the mdev to a guest.
VFIO-CCW Regions
----------------
The vfio-ccw driver exposes MMIO regions to accept requests from and return
results to userspace.
vfio-ccw I/O region
-------------------
An I/O region is used to accept channel program request from user
space and store I/O interrupt result for user space to retrieve. The
definition of the region is::
struct ccw_io_region {
#define ORB_AREA_SIZE 12
__u8 orb_area[ORB_AREA_SIZE];
#define SCSW_AREA_SIZE 12
__u8 scsw_area[SCSW_AREA_SIZE];
#define IRB_AREA_SIZE 96
__u8 irb_area[IRB_AREA_SIZE];
__u32 ret_code;
} __packed;
This region is always available.
While starting an I/O request, orb_area should be filled with the
guest ORB, and scsw_area should be filled with the SCSW of the Virtual
Subchannel.
irb_area stores the I/O result.
ret_code stores a return code for each access of the region. The following
values may occur:
``0``
The operation was successful.
``-EOPNOTSUPP``
The ORB specified transport mode or the
SCSW specified a function other than the start function.
``-EIO``
A request was issued while the device was not in a state ready to accept
requests, or an internal error occurred.
``-EBUSY``
The subchannel was status pending or busy, or a request is already active.
``-EAGAIN``
A request was being processed, and the caller should retry.
``-EACCES``
The channel path(s) used for the I/O were found to be not operational.
``-ENODEV``
The device was found to be not operational.
``-EINVAL``
The orb specified a chain longer than 255 ccws, or an internal error
occurred.
vfio-ccw cmd region
-------------------
The vfio-ccw cmd region is used to accept asynchronous instructions
from userspace::
#define VFIO_CCW_ASYNC_CMD_HSCH (1 << 0)
#define VFIO_CCW_ASYNC_CMD_CSCH (1 << 1)
struct ccw_cmd_region {
__u32 command;
__u32 ret_code;
} __packed;
This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_ASYNC_CMD.
Currently, CLEAR SUBCHANNEL and HALT SUBCHANNEL use this region.
command specifies the command to be issued; ret_code stores a return code
for each access of the region. The following values may occur:
``0``
The operation was successful.
``-ENODEV``
The device was found to be not operational.
``-EINVAL``
A command other than halt or clear was specified.
``-EIO``
A request was issued while the device was not in a state ready to accept
requests.
``-EAGAIN``
A request was being processed, and the caller should retry.
``-EBUSY``
The subchannel was status pending or busy while processing a halt request.
vfio-ccw schib region
---------------------
The vfio-ccw schib region is used to return Subchannel-Information
Block (SCHIB) data to userspace::
struct ccw_schib_region {
#define SCHIB_AREA_SIZE 52
__u8 schib_area[SCHIB_AREA_SIZE];
} __packed;
This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_SCHIB.
Reading this region triggers a STORE SUBCHANNEL to be issued to the
associated hardware.
vfio-ccw crw region
---------------------
The vfio-ccw crw region is used to return Channel Report Word (CRW)
data to userspace::
struct ccw_crw_region {
__u32 crw;
__u32 pad;
} __packed;
This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_CRW.
Reading this region returns a CRW if one that is relevant for this
subchannel (e.g. one reporting changes in channel path state) is
pending, or all zeroes if not. If multiple CRWs are pending (including
possibly chained CRWs), reading this region again will return the next
one, until no more CRWs are pending and zeroes are returned. This is
similar to how STORE CHANNEL REPORT WORD works.
vfio-ccw operation details
--------------------------
vfio-ccw follows what vfio-pci did on the s390 platform and uses
vfio-iommu-type1 as the vfio iommu backend.
* CCW translation APIs
A group of APIs (start with `cp_`) to do CCW translation. The CCWs
passed in by a user space program are organized with their guest
physical memory addresses. These APIs will copy the CCWs into kernel
space, and assemble a runnable kernel channel program by updating the
guest physical addresses with their corresponding host physical addresses.
Note that we have to use IDALs even for direct-access CCWs, as the
referenced memory can be located anywhere, including above 2G.
* vfio_ccw device driver
This driver utilizes the CCW translation APIs and introduces
vfio_ccw, which is the driver for the I/O subchannel devices you want
to pass through.
vfio_ccw implements the following vfio ioctls::
VFIO_DEVICE_GET_INFO
VFIO_DEVICE_GET_IRQ_INFO
VFIO_DEVICE_GET_REGION_INFO
VFIO_DEVICE_RESET
VFIO_DEVICE_SET_IRQS
This provides an I/O region, so that the user space program can pass a
channel program to the kernel, to do further CCW translation before
issuing them to a real device.
This also provides the SET_IRQ ioctl to setup an event notifier to
notify the user space program the I/O completion in an asynchronous
way.
The use of vfio-ccw is not limited to QEMU, while QEMU is definitely a
good example to get understand how these patches work. Here is a little
bit more detail how an I/O request triggered by the QEMU guest will be
handled (without error handling).
Explanation:
- Q1-Q7: QEMU side process.
- K1-K5: Kernel side process.
Q1.
Get I/O region info during initialization.
Q2.
Setup event notifier and handler to handle I/O completion.
... ...
Q3.
Intercept a ssch instruction.
Q4.
Write the guest channel program and ORB to the I/O region.
K1.
Copy from guest to kernel.
K2.
Translate the guest channel program to a host kernel space
channel program, which becomes runnable for a real device.
K3.
With the necessary information contained in the orb passed in
by QEMU, issue the ccwchain to the device.
K4.
Return the ssch CC code.
Q5.
Return the CC code to the guest.
... ...
K5.
Interrupt handler gets the I/O result and write the result to
the I/O region.
K6.
Signal QEMU to retrieve the result.
Q6.
Get the signal and event handler reads out the result from the I/O
region.
Q7.
Update the irb for the guest.
Limitations
-----------
The current vfio-ccw implementation focuses on supporting basic commands
needed to implement block device functionality (read/write) of DASD/ECKD
device only. Some commands may need special handling in the future, for
example, anything related to path grouping.
DASD is a kind of storage device. While ECKD is a data recording format.
More information for DASD and ECKD could be found here:
https://en.wikipedia.org/wiki/Direct-access_storage_device
https://en.wikipedia.org/wiki/Count_key_data
Together with the corresponding work in QEMU, we can bring the passed
through DASD/ECKD device online in a guest now and use it as a block
device.
The current code allows the guest to start channel programs via
START SUBCHANNEL, and to issue HALT SUBCHANNEL, CLEAR SUBCHANNEL,
and STORE SUBCHANNEL.
Currently all channel programs are prefetched, regardless of the
p-bit setting in the ORB. As a result, self modifying channel
programs are not supported. For this reason, IPL has to be handled as
a special case by a userspace/guest program; this has been implemented
in QEMU's s390-ccw bios as of QEMU 4.1.
vfio-ccw supports classic (command mode) channel I/O only. Transport
mode (HPF) is not supported.
QDIO subchannels are currently not supported. Classic devices other than
DASD/ECKD might work, but have not been tested.
Reference
---------
1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)
2. ESA/390 Common I/O Device Commands manual (IBM Form. No. SA22-7204)
3. https://en.wikipedia.org/wiki/Channel_I/O
4. Documentation/arch/s390/cds.rst
5. Documentation/driver-api/vfio.rst
6. Documentation/driver-api/vfio-mediated-device.rst
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
vfio-ccw 기본 구조 소개
1-42이 문서는 Linux/s390 I/O subchannel device를 위한 VFIO 지원을 설명합니다. `vfio-ccw`의 목적은 VFIO를 수단으로 subchannel을 virtual machine에 passthrough하는 것입니다.
다른 hardware architecture와 달리 s390은 Channel I/O라는 통합 I/O 접근 방식을 정의합니다. channel program은 별도 (co)processor에서 비동기로 실행되며, channel subsystem은 channel program에서 호출자가 지정한 memory에 IOMMU 없이 직접 접근합니다.
이 특성 때문에 device 지원은 mediated device(mdev)로 구현합니다. VFIO mdev를 IOMMU group에 추가해 VFIO framework가 관리하게 하고, 특수 VFIO I/O region에 read/write callback을 제공해 channel program을 mdev에서 parent인 실제 I/O subchannel device로 전달합니다. parent 쪽에서 주소를 변환하고 I/O instruction을 실행합니다.
이 문서는 s390 I/O architecture 전체를 설명하지 않습니다. 일반 Channel I/O는 다음 링크, s390 architecture는 "s390 Principles of Operation"(SA22-7832), 단순 channel subsystem flow는 `qemu/hw/s390x/css.c`, VFIO mdev framework는 `Documentation/driver-api/vfio-mediated-device.rst`를 참조하십시오.
vfio-ccw의 필요성
43-62s390의 QEMU/KVM guest는 보통 `Virtio Over Channel I/O(virtio-ccw)` transport를 통해 paravirtualized virtio device만 봅니다. 이 방식은 표준 channel device 처리 algorithm으로 virtio device를 탐색하게 합니다.
그러나 standard Channel I/O mechanism을 쓰는 대다수 s390 device도 QEMU virtual machine으로 passthrough해야 합니다. tape drive처럼 virtio 대응물이 없거나 guest가 활용하려는 고유 특성이 있는 device가 여기에 포함됩니다.
다른 device와 같은 VFIO interface를 사용하기 위해 channel device 지원을 VFIO mediated device framework와 subchannel device driver `vfio_ccw`로 구현합니다.
CCW device 접근 방식
63-98s390 channel subsystem은 DASD, tape, communication controller 등 다양한 물리 peripheral을 통합된 관점으로 제공합니다. 모두 잘 정의된 방식으로 접근하며 I/O completion은 I/O interruption이라는 동일한 방식으로 보고합니다.
모든 I/O는 channel command word(CCW)를 사용합니다. CCW는 전용 I/O channel processor의 instruction이고 channel program은 channel subsystem이 실행하는 CCW sequence입니다. channel program을 발행하려면 CCW format과 기타 control 정보를 가리키는 operation request block(ORB)을 만듭니다.
operating system은 `SSCH`(start sub-channel) instruction으로 channel subsystem에 program 실행 시작을 알립니다. central processor는 interruption을 받을 때까지 non-I/O instruction을 계속 실행하며, interrupt handler는 I/O completion result를 interrupt response block(IRB) 형태로 받습니다.
- guest kernel이 guest physical address로 ORB와 channel program을 만듭니다.
- ORB와 channel program을 host kernel에 전달합니다.
- host가 guest physical address를 real address로 변환하고 `SSCH` 같은 privileged Channel I/O instruction을 발행합니다.
- channel program이 별도 processor에서 비동기로 실행됩니다.
- I/O interruption이 host에 completion을 알리고 IRB를 userspace로 복사해 guest에 되돌립니다.
physical vfio_ccw device와 child mdev
99-145Channel I/O에는 IOMMU hardware 지원이 없으므로 physical vfio-ccw device에는 IOMMU 수준의 translation이나 isolation이 없습니다. 모든 subchannel I/O instruction은 privileged instruction이며, interception 처리 중 `vfio-ccw` software가 hardware로 보내기 전에 channel program을 검사하고 주소를 변환합니다.
| driver | 역할 |
|---|---|
| `vfio_ccw` | 실제 I/O subchannel을 구동하는 physical subchannel driver입니다. callback을 구현하고 mdev framework에 parent device로 등록합니다. sysfs의 generic mdev interface로 mdev를 하나 만들고 mediated bus, IOMMU group, VFIO group에 추가합니다. channel program 요청과 interruption result를 주고받는 I/O region 및 completion용 eventfd 설정도 제공합니다. |
| `vfio_mdev` | mdev framework가 제공하는 mediated vfio ccw device driver입니다. VFIO device callback을 구현하고 mdev를 IOMMU/VFIO group에 추가하며 mdev driver로 등록합니다. |
`vfio_mdev`의 VFIO IOMMU backend는 기존 map/unmap ioctl을 사용하지만 hardware IOMMU를 programming하지 않고 이후 요청에 쓸 translation을 저장합니다. guest physical address를 process virtual address로 바꾸고 page를 pin한 뒤 host physical address로 hardware를 programming할 수 있습니다.
mdev의 `VFIO_IOMMU_MAP_DMA`에서는 page를 pin하지 않고 `iova<->vaddr` mapping database만 유지합니다. physical device가 필요할 때 pin/unpin하도록 backend가 `vfio_pin_pages`와 `vfio_unpin_pages` interface를 export합니다.
MDEV core와 driver 결합 절차
146-183원문 high-level block diagram의 driver 등록, callback, userspace 연결 관계입니다.
- 1. `vfio_ccw.ko`가 physical I/O subchannel을 구동하고 device pointer와 callback을 mdev framework에 등록합니다. sysfs device node 아래에 `mdev_create`, `mdev_destroy`, `mdev_supported_types`가 생성됩니다.
- 2. `mdev_create` sysfs file로 mediated vfio ccw device를 수동 생성합니다. 이 구현에서는 하나만 만듭니다.
- 3. `vfio_mdev.ko`가 mediated ccw device를 probe해 `iommu_group`과 `vfio_group`에 추가하며, 이후 mdev를 guest로 passthrough할 수 있습니다.
VFIO-CCW region 개요
184-189`vfio-ccw` driver는 userspace 요청을 받고 결과를 반환하는 MMIO region을 노출합니다.
vfio-ccw I/O region
190-245I/O region은 userspace의 channel program 요청을 받고 userspace가 가져갈 I/O interruption result를 저장합니다.
struct ccw_io_region {
#define ORB_AREA_SIZE 12
__u8 orb_area[ORB_AREA_SIZE];
#define SCSW_AREA_SIZE 12
__u8 scsw_area[SCSW_AREA_SIZE];
#define IRB_AREA_SIZE 96
__u8 irb_area[IRB_AREA_SIZE];
__u32 ret_code;
} __packed;이 region은 항상 사용 가능합니다. I/O를 시작할 때 `orb_area`에는 guest ORB를, `scsw_area`에는 Virtual Subchannel의 SCSW를 채웁니다. `irb_area`는 I/O result를 저장하고 `ret_code`는 region 접근 결과를 저장합니다.
| ret_code | 의미 |
|---|---|
| `0` | 작업이 성공했습니다. |
| `-EOPNOTSUPP` | ORB가 transport mode를 지정했거나 SCSW가 start 이외의 function을 지정했습니다. |
| `-EIO` | device가 요청을 받을 준비가 되지 않은 상태에서 요청했거나 internal error가 발생했습니다. |
| `-EBUSY` | subchannel이 status pending 또는 busy이거나 이미 active request가 있습니다. |
| `-EAGAIN` | 요청을 처리 중이므로 호출자가 다시 시도해야 합니다. |
| `-EACCES` | I/O에 사용한 channel path가 operational 상태가 아닙니다. |
| `-ENODEV` | device가 operational 상태가 아닙니다. |
| `-EINVAL` | ORB가 255개보다 긴 CCW chain을 지정했거나 internal error가 발생했습니다. |
vfio-ccw cmd region
246-284cmd region은 userspace의 asynchronous instruction을 받습니다.
#define VFIO_CCW_ASYNC_CMD_HSCH (1 << 0)
#define VFIO_CCW_ASYNC_CMD_CSCH (1 << 1)
struct ccw_cmd_region {
__u32 command;
__u32 ret_code;
} __packed;이 region은 `VFIO_REGION_SUBTYPE_CCW_ASYNC_CMD` type으로 노출되며 현재 `CLEAR SUBCHANNEL`과 `HALT SUBCHANNEL`이 사용합니다. `command`는 발행할 command이고 `ret_code`는 접근 결과입니다.
| ret_code | 의미 |
|---|---|
| `0` | 작업이 성공했습니다. |
| `-ENODEV` | device가 operational 상태가 아닙니다. |
| `-EINVAL` | halt 또는 clear 이외의 command를 지정했습니다. |
| `-EIO` | device가 요청을 받을 준비가 되지 않은 상태에서 요청했습니다. |
| `-EAGAIN` | 요청을 처리 중이므로 다시 시도해야 합니다. |
| `-EBUSY` | halt 요청 처리 중 subchannel이 status pending 또는 busy였습니다. |
vfio-ccw schib region
285-300schib region은 Subchannel-Information Block(SCHIB) data를 userspace에 반환합니다.
struct ccw_schib_region {
#define SCHIB_AREA_SIZE 52
__u8 schib_area[SCHIB_AREA_SIZE];
} __packed;이 region은 `VFIO_REGION_SUBTYPE_CCW_SCHIB` type으로 노출됩니다. region을 읽으면 연결된 hardware에 `STORE SUBCHANNEL`을 발행합니다.
vfio-ccw crw region
301-320crw region은 Channel Report Word(CRW) data를 userspace에 반환합니다.
struct ccw_crw_region {
__u32 crw;
__u32 pad;
} __packed;이 region은 `VFIO_REGION_SUBTYPE_CCW_CRW` type으로 노출됩니다. 읽을 때 channel path state 변경처럼 이 subchannel과 관련된 CRW가 pending이면 하나를 반환하고, 없으면 모두 0을 반환합니다.
chained CRW를 포함해 여러 CRW가 pending이면 region을 반복해서 읽을 때 다음 CRW를 차례로 반환하다가 더 이상 없으면 0을 반환합니다. 이는 `STORE CHANNEL REPORT WORD` 동작과 비슷합니다.
CCW translation API와 VFIO ioctl
321-354`vfio-ccw`는 s390의 `vfio-pci` 방식을 따라 `vfio-iommu-type1`을 VFIO IOMMU backend로 사용합니다.
`cp_`로 시작하는 CCW translation API는 guest physical memory address로 구성된 userspace CCW를 kernel space로 복사하고, 각 guest physical address를 대응 host physical address로 바꿔 실제 device에서 실행 가능한 kernel channel program을 조립합니다. 참조 memory가 2G 위를 포함해 어디에나 있을 수 있으므로 direct-access CCW에도 IDAL을 사용해야 합니다.
`vfio_ccw` physical I/O subchannel driver는 translation API를 사용하고 다음 VFIO ioctl을 구현합니다.
VFIO_DEVICE_GET_INFO
VFIO_DEVICE_GET_IRQ_INFO
VFIO_DEVICE_GET_REGION_INFO
VFIO_DEVICE_RESET
VFIO_DEVICE_SET_IRQSI/O region을 통해 userspace channel program을 kernel로 전달하고 실제 device에 발행하기 전에 CCW를 변환합니다. `SET_IRQ` ioctl은 I/O completion을 userspace에 비동기로 알릴 event notifier를 설정합니다.
QEMU guest I/O 요청 처리 흐름
355-404`vfio-ccw` 사용은 QEMU에 한정되지 않지만 QEMU는 동작을 이해하기 좋은 예입니다. 오류 처리를 제외한 QEMU guest 요청의 흐름에서 Q1-Q7은 QEMU 단계입니다. 원문 설명은 kernel 단계를 K1-K5로 적었지만, 뒤의 실제 순서에는 QEMU에 signal을 보내는 K6도 포함됩니다.
QEMU의 SSCH interception부터 host channel program 실행과 guest IRB 갱신까지의 왕복입니다.
현재 제한 사항
405-437현재 구현은 DASD/ECKD device의 block read/write에 필요한 기본 command 지원에 초점을 둡니다. path grouping 관련 command처럼 향후 특수 처리가 필요한 command가 있을 수 있습니다. DASD는 storage device 종류이고 ECKD는 data recording format입니다.
- https://en.wikipedia.org/wiki/Direct-access_storage_device
https://en.wikipedia.org/wiki/Direct-access_storage_device - https://en.wikipedia.org/wiki/Count_key_data
https://en.wikipedia.org/wiki/Count_key_data
대응 QEMU 작업과 함께 passthrough한 DASD/ECKD device를 guest에서 online으로 만들고 block device로 사용할 수 있습니다. guest는 `START SUBCHANNEL`로 channel program을 시작하고 `HALT SUBCHANNEL`, `CLEAR SUBCHANNEL`, `STORE SUBCHANNEL`을 발행할 수 있습니다.
현재 모든 channel program은 ORB의 p-bit 설정과 무관하게 prefetch됩니다. 따라서 self-modifying channel program을 지원하지 않습니다. IPL은 userspace/guest program이 특수 처리해야 하며 QEMU 4.1부터 `s390-ccw bios`에 구현돼 있습니다.
- classic command mode Channel I/O만 지원하며 transport mode(HPF)는 지원하지 않습니다.
- QDIO subchannel은 지원하지 않습니다.
- DASD/ECKD 이외 classic device도 동작할 수 있지만 시험되지 않았습니다.
참고 자료
438-445- ESA/s390 Principles of Operation manual, IBM SA22-7832
- ESA/390 Common I/O Device Commands manual, IBM SA22-7204
- Channel I/O 참고 링크
- `Documentation/arch/s390/cds.rst`
- `Documentation/driver-api/vfio.rst`
- `Documentation/driver-api/vfio-mediated-device.rst`
요약과 해설
vfio-ccw.rst:1-445Channel I/O는 hardware IOMMU 없이 channel program이 지정 memory를 직접 접근하므로 `vfio-ccw`가 privileged instruction interception, guest-to-host CCW 주소 변환과 page pinning을 software로 수행합니다. physical `vfio_ccw` parent와 mediated `vfio_mdev` child를 결합해 VFIO group에 노출합니다.
I/O, async command, SCHIB, CRW region이 ORB/SCSW 요청과 IRB·channel 상태·report 결과를 교환하며 eventfd가 completion을 비동기로 알립니다. 현재 DASD/ECKD의 classic command mode에 집중하고 transport mode, QDIO, self-modifying channel program은 지원하지 않습니다.