요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
===========================================
Shared Virtual Addressing (SVA) with ENQCMD
===========================================
Background
==========
Shared Virtual Addressing (SVA) allows the processor and device to use the
same virtual addresses avoiding the need for software to translate virtual
addresses to physical addresses. SVA is what PCIe calls Shared Virtual
Memory (SVM).
In addition to the convenience of using application virtual addresses
by the device, it also doesn't require pinning pages for DMA.
PCIe Address Translation Services (ATS) along with Page Request Interface
(PRI) allow devices to function much the same way as the CPU handling
application page-faults. For more information please refer to the PCIe
specification Chapter 10: ATS Specification.
Use of SVA requires IOMMU support in the platform. IOMMU is also
required to support the PCIe features ATS and PRI. ATS allows devices
to cache translations for virtual addresses. The IOMMU driver uses the
mmu_notifier() support to keep the device TLB cache and the CPU cache in
sync. When an ATS lookup fails for a virtual address, the device should
use the PRI in order to request the virtual address to be paged into the
CPU page tables. The device must use ATS again in order to fetch the
translation before use.
Shared Hardware Workqueues
==========================
Unlike Single Root I/O Virtualization (SR-IOV), Scalable IOV (SIOV) permits
the use of Shared Work Queues (SWQ) by both applications and Virtual
Machines (VM's). This allows better hardware utilization vs. hard
partitioning resources that could result in under utilization. In order to
allow the hardware to distinguish the context for which work is being
executed in the hardware by SWQ interface, SIOV uses Process Address Space
ID (PASID), which is a 20-bit number defined by the PCIe SIG.
PASID value is encoded in all transactions from the device. This allows the
IOMMU to track I/O on a per-PASID granularity in addition to using the PCIe
Resource Identifier (RID) which is the Bus/Device/Function.
ENQCMD
======
ENQCMD is a new instruction on Intel platforms that atomically submits a
work descriptor to a device. The descriptor includes the operation to be
performed, virtual addresses of all parameters, virtual address of a completion
record, and the PASID (process address space ID) of the current process.
ENQCMD works with non-posted semantics and carries a status back if the
command was accepted by hardware. This allows the submitter to know if the
submission needs to be retried or other device specific mechanisms to
implement fairness or ensure forward progress should be provided.
ENQCMD is the glue that ensures applications can directly submit commands
to the hardware and also permits hardware to be aware of application context
to perform I/O operations via use of PASID.
Process Address Space Tagging
=============================
A new thread-scoped MSR (IA32_PASID) provides the connection between
user processes and the rest of the hardware. When an application first
accesses an SVA-capable device, this MSR is initialized with a newly
allocated PASID. The driver for the device calls an IOMMU-specific API
that sets up the routing for DMA and page-requests.
For example, the Intel Data Streaming Accelerator (DSA) uses
iommu_sva_bind_device(), which will do the following:
- Allocate the PASID, and program the process page-table (%cr3 register) in the
PASID context entries.
- Register for mmu_notifier() to track any page-table invalidations to keep
the device TLB in sync. For example, when a page-table entry is invalidated,
the IOMMU propagates the invalidation to the device TLB. This will force any
future access by the device to this virtual address to participate in
ATS. If the IOMMU responds with proper response that a page is not
present, the device would request the page to be paged in via the PCIe PRI
protocol before performing I/O.
This MSR is managed with the XSAVE feature set as "supervisor state" to
ensure the MSR is updated during context switch.
PASID Management
================
The kernel must allocate a PASID on behalf of each process which will use
ENQCMD and program it into the new MSR to communicate the process identity to
platform hardware. ENQCMD uses the PASID stored in this MSR to tag requests
from this process. When a user submits a work descriptor to a device using the
ENQCMD instruction, the PASID field in the descriptor is auto-filled with the
value from MSR_IA32_PASID. Requests for DMA from the device are also tagged
with the same PASID. The platform IOMMU uses the PASID in the transaction to
perform address translation. The IOMMU APIs setup the corresponding PASID
entry in IOMMU with the process address used by the CPU (e.g. %cr3 register in
x86).
The MSR must be configured on each logical CPU before any application
thread can interact with a device. Threads that belong to the same
process share the same page tables, thus the same MSR value.
PASID Life Cycle Management
===========================
PASID is initialized as IOMMU_PASID_INVALID (-1) when a process is created.
Only processes that access SVA-capable devices need to have a PASID
allocated. This allocation happens when a process opens/binds an SVA-capable
device but finds no PASID for this process. Subsequent binds of the same, or
other devices will share the same PASID.
Although the PASID is allocated to the process by opening a device,
it is not active in any of the threads of that process. It's loaded to the
IA32_PASID MSR lazily when a thread tries to submit a work descriptor
to a device using the ENQCMD.
That first access will trigger a #GP fault because the IA32_PASID MSR
has not been initialized with the PASID value assigned to the process
when the device was opened. The Linux #GP handler notes that a PASID has
been allocated for the process, and so initializes the IA32_PASID MSR
and returns so that the ENQCMD instruction is re-executed.
On fork(2) or exec(2) the PASID is removed from the process as it no
longer has the same address space that it had when the device was opened.
On clone(2) the new task shares the same address space, so will be
able to use the PASID allocated to the process. The IA32_PASID is not
preemptively initialized as the PASID value might not be allocated yet or
the kernel does not know whether this thread is going to access the device
and the cleared IA32_PASID MSR reduces context switch overhead by xstate
init optimization. Since #GP faults have to be handled on any threads that
were created before the PASID was assigned to the mm of the process, newly
created threads might as well be treated in a consistent way.
Due to complexity of freeing the PASID and clearing all IA32_PASID MSRs in
all threads in unbind, free the PASID lazily only on mm exit.
If a process does a close(2) of the device file descriptor and munmap(2)
of the device MMIO portal, then the driver will unbind the device. The
PASID is still marked VALID in the PASID_MSR for any threads in the
process that accessed the device. But this is harmless as without the
MMIO portal they cannot submit new work to the device.
Relationships
=============
* Each process has many threads, but only one PASID.
* Devices have a limited number (~10's to 1000's) of hardware workqueues.
The device driver manages allocating hardware workqueues.
* A single mmap() maps a single hardware workqueue as a "portal" and
each portal maps down to a single workqueue.
* For each device with which a process interacts, there must be
one or more mmap()'d portals.
* Many threads within a process can share a single portal to access
a single device.
* Multiple processes can separately mmap() the same portal, in
which case they still share one device hardware workqueue.
* The single process-wide PASID is used by all threads to interact
with all devices. There is not, for instance, a PASID for each
thread or each thread<->device pair.
FAQ
===
* What is SVA/SVM?
Shared Virtual Addressing (SVA) permits I/O hardware and the processor to
work in the same address space, i.e., to share it. Some call it Shared
Virtual Memory (SVM), but Linux community wanted to avoid confusing it with
POSIX Shared Memory and Secure Virtual Machines which were terms already in
circulation.
* What is a PASID?
A Process Address Space ID (PASID) is a PCIe-defined Transaction Layer Packet
(TLP) prefix. A PASID is a 20-bit number allocated and managed by the OS.
PASID is included in all transactions between the platform and the device.
* How are shared workqueues different?
Traditionally, in order for userspace applications to interact with hardware,
there is a separate hardware instance required per process. For example,
consider doorbells as a mechanism of informing hardware about work to process.
Each doorbell is required to be spaced 4k (or page-size) apart for process
isolation. This requires hardware to provision that space and reserve it in
MMIO. This doesn't scale as the number of threads becomes quite large. The
hardware also manages the queue depth for Shared Work Queues (SWQ), and
consumers don't need to track queue depth. If there is no space to accept
a command, the device will return an error indicating retry.
A user should check Deferrable Memory Write (DMWr) capability on the device
and only submits ENQCMD when the device supports it. In the new DMWr PCIe
terminology, devices need to support DMWr completer capability. In addition,
it requires all switch ports to support DMWr routing and must be enabled by
the PCIe subsystem, much like how PCIe atomic operations are managed for
instance.
SWQ allows hardware to provision just a single address in the device. When
used with ENQCMD to submit work, the device can distinguish the process
submitting the work since it will include the PASID assigned to that
process. This helps the device scale to a large number of processes.
* Is this the same as a user space device driver?
Communicating with the device via the shared workqueue is much simpler
than a full blown user space driver. The kernel driver does all the
initialization of the hardware. User space only needs to worry about
submitting work and processing completions.
* Is this the same as SR-IOV?
Single Root I/O Virtualization (SR-IOV) focuses on providing independent
hardware interfaces for virtualizing hardware. Hence, it's required to be
an almost fully functional interface to software supporting the traditional
BARs, space for interrupts via MSI-X, its own register layout.
Virtual Functions (VFs) are assisted by the Physical Function (PF)
driver.
Scalable I/O Virtualization builds on the PASID concept to create device
instances for virtualization. SIOV requires host software to assist in
creating virtual devices; each virtual device is represented by a PASID
along with the bus/device/function of the device. This allows device
hardware to optimize device resource creation and can grow dynamically on
demand. SR-IOV creation and management is very static in nature. Consult
references below for more details.
* Why not just create a virtual function for each app?
Creating PCIe SR-IOV type Virtual Functions (VF) is expensive. VFs require
duplicated hardware for PCI config space and interrupts such as MSI-X.
Resources such as interrupts have to be hard partitioned between VFs at
creation time, and cannot scale dynamically on demand. The VFs are not
completely independent from the Physical Function (PF). Most VFs require
some communication and assistance from the PF driver. SIOV, in contrast,
creates a software-defined device where all the configuration and control
aspects are mediated via the slow path. The work submission and completion
happen without any mediation.
* Does this support virtualization?
ENQCMD can be used from within a guest VM. In these cases, the VMM helps
with setting up a translation table to translate from Guest PASID to Host
PASID. Please consult the ENQCMD instruction set reference for more
details.
* Does memory need to be pinned?
When devices support SVA along with platform hardware such as IOMMU
supporting such devices, there is no need to pin memory for DMA purposes.
Devices that support SVA also support other PCIe features that remove the
pinning requirement for memory.
Device TLB support - Device requests the IOMMU to lookup an address before
use via Address Translation Service (ATS) requests. If the mapping exists
but there is no page allocated by the OS, IOMMU hardware returns that no
mapping exists.
Device requests the virtual address to be mapped via Page Request
Interface (PRI). Once the OS has successfully completed the mapping, it
returns the response back to the device. The device requests again for
a translation and continues.
IOMMU works with the OS in managing consistency of page-tables with the
device. When removing pages, it interacts with the device to remove any
device TLB entry that might have been cached before removing the mappings from
the OS.
References
==========
VT-D:
https://01.org/blogs/ashokraj/2018/recent-enhancements-intel-virtualization-technology-directed-i/o-intel-vt-d
SIOV:
https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux
ENQCMD in ISE:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
DSA spec:
https://software.intel.com/sites/default/files/341204-intel-data-streaming-accelerator-spec.pdf
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
SVA와 ATS·PRI 배경
1-30이 문서는 `SPDX-License-Identifier: GPL-2.0`으로 배포됩니다. Shared Virtual Addressing(SVA)은 processor와 device가 같은 virtual address를 사용하게 해 software가 virtual address를 physical address로 변환할 필요를 없앱니다. PCIe 용어로는 Shared Virtual Memory(SVM)입니다.
device가 application virtual address를 편리하게 사용할 뿐 아니라 DMA를 위해 page를 pin할 필요도 없습니다. PCIe Address Translation Services(ATS)와 Page Request Interface(PRI)는 device가 application page fault를 다루는 CPU와 비슷하게 동작하게 합니다. 자세한 내용은 PCIe specification Chapter 10 ATS Specification을 참고하십시오.
SVA에는 platform IOMMU 지원이 필요하며 IOMMU도 PCIe ATS와 PRI를 지원해야 합니다. ATS는 device가 virtual-address translation을 cache하게 합니다. IOMMU driver는 `mmu_notifier()`로 device TLB cache와 CPU cache를 동기화합니다.
virtual address의 ATS lookup이 실패하면 device는 PRI로 해당 address를 CPU page table에 page-in하도록 요청해야 합니다. 사용 전에 ATS를 다시 실행해 translation을 가져옵니다.
shared workqueue와 ENQCMD
31-63SR-IOV와 달리 Scalable IOV(SIOV)는 application과 VM이 Shared Work Queue(SWQ)를 함께 사용하게 합니다. resource를 hard partition해 utilization이 낮아지는 일을 피하고 hardware 활용도를 높입니다.
SWQ interface가 실행하는 work의 context를 hardware가 구분하도록 SIOV는 PCIe SIG가 정의한 20-bit Process Address Space ID(PASID)를 사용합니다.
device의 모든 transaction에 PASID가 encode됩니다. IOMMU는 Bus/Device/Function인 PCIe Resource Identifier(RID)뿐 아니라 PASID별 granularity로 I/O를 추적할 수 있습니다.
`ENQCMD`는 Intel platform에서 work descriptor를 device에 atomic하게 submit하는 instruction입니다. descriptor에는 수행할 operation, 모든 parameter의 virtual address, completion record의 virtual address, current process의 PASID가 포함됩니다.
`ENQCMD`는 non-posted semantic으로 동작하며 hardware가 command를 accept했는지 status를 반환합니다. submitter는 retry가 필요한지, fairness 또는 forward progress를 위해 device-specific mechanism이 필요한지 판단할 수 있습니다.
`ENQCMD`는 application이 hardware에 직접 command를 submit하게 하고 PASID로 application context를 hardware에 알려 I/O operation을 수행하게 하는 연결 고리입니다.
process address-space tagging
64-88새 thread-scoped MSR `IA32_PASID`가 user process와 나머지 hardware를 연결합니다. application이 SVA-capable device에 처음 access하면 새로 allocate한 PASID로 이 MSR을 초기화합니다. device driver는 IOMMU-specific API를 호출해 DMA와 page-request routing을 설정합니다.
예를 들어 Intel Data Streaming Accelerator(DSA)는 `iommu_sva_bind_device()`를 호출해 다음 작업을 수행합니다.
- PASID를 allocate하고 process page table인 `%cr3` register를 PASID context entry에 programming합니다.
- `mmu_notifier()`에 등록해 page-table invalidation을 추적하고 device TLB를 동기화합니다. PTE가 invalidate되면 IOMMU가 device TLB에도 전파합니다. 이후 access는 ATS를 거치며 page가 없다는 response를 받으면 device가 PCIe PRI로 page-in을 요청한 뒤 I/O를 수행합니다.
이 MSR은 context switch 때 update되도록 XSAVE feature set에서 `supervisor state`로 관리됩니다.
PASID 설정과 transaction tagging
89-106kernel은 `ENQCMD`를 사용할 process마다 PASID를 allocate하고 새 MSR에 programming해 process identity를 platform hardware에 전달해야 합니다.
`ENQCMD`는 MSR의 PASID로 process request를 tag합니다. user가 work descriptor를 submit하면 descriptor의 PASID field가 `MSR_IA32_PASID` 값으로 자동 채워지고 device의 DMA request도 같은 PASID로 tag됩니다.
platform IOMMU는 transaction의 PASID로 address translation을 수행합니다. IOMMU API는 CPU가 사용하는 process address, x86에서는 `%cr3` register를 이용해 해당 PASID entry를 설정합니다.
application thread가 device와 상호작용하기 전에 각 logical CPU에서 MSR을 설정해야 합니다. 같은 process의 thread는 page table을 공유하므로 MSR 값도 같습니다.
PASID lifecycle
107-148process 생성 시 PASID는 `IOMMU_PASID_INVALID (-1)`로 초기화됩니다.
SVA-capable device에 access하는 process만 PASID가 필요합니다. process가 device를 open/bind했지만 PASID가 없을 때 allocate하며 같은 device나 다른 device에 대한 후속 bind도 같은 PASID를 공유합니다.
device open으로 PASID를 allocate해도 process thread에서는 아직 active하지 않습니다. thread가 처음 `ENQCMD`로 work descriptor를 submit할 때 `IA32_PASID` MSR에 lazy load합니다.
첫 access에서는 device open 시 process에 할당한 PASID로 `IA32_PASID`가 초기화되지 않아 `#GP`가 발생합니다. Linux `#GP` handler는 process에 PASID가 할당된 사실을 확인하고 MSR을 초기화한 뒤 돌아가 `ENQCMD`를 다시 실행합니다.
`fork(2)` 또는 `exec(2)`에서는 device open 당시와 address space가 달라지므로 process에서 PASID를 제거합니다. `clone(2)`의 새 task는 같은 address space를 공유하므로 process PASID를 사용할 수 있습니다.
`clone(2)`에서는 PASID가 아직 없을 수 있고 thread가 device를 사용할지 kernel이 모르므로 `IA32_PASID`를 미리 초기화하지 않습니다. clear된 MSR은 xstate init optimization으로 context-switch overhead를 줄입니다. PASID가 mm에 할당되기 전에 생성된 thread도 어차피 `#GP` 처리가 필요하므로 새 thread도 같은 방식으로 다룹니다.
unbind 때 모든 thread의 `IA32_PASID`를 지우고 PASID를 free하는 일은 복잡하므로 mm exit 시에만 lazy free합니다.
process가 device file descriptor를 `close(2)`하고 device MMIO portal을 `munmap(2)`하면 driver가 device를 unbind합니다. device에 access했던 thread의 `PASID_MSR`에는 PASID가 VALID로 남지만 MMIO portal 없이 새 work를 submit할 수 없어 무해합니다.
process·PASID·portal·workqueue 관계
149-166- process에는 thread가 여러 개지만 PASID는 하나뿐입니다.
- device에는 제한된 수, 대략 수십에서 수천 개의 hardware workqueue가 있고 driver가 이를 allocate합니다.
- `mmap()` 하나는 hardware workqueue 하나를 portal로 mapping하며 각 portal은 단일 workqueue로 이어집니다.
- process가 상호작용하는 device마다 `mmap()`된 portal이 하나 이상 필요합니다.
- process 안의 여러 thread가 단일 portal을 공유해 한 device에 access할 수 있습니다.
- 여러 process가 같은 portal을 각각 `mmap()`해도 하나의 device hardware workqueue를 공유합니다.
- process-wide PASID 하나를 모든 thread가 모든 device와 상호작용하는 데 사용합니다. thread별 또는 thread-device pair별 PASID는 없습니다.
FAQ: SVA, PASID, SWQ와 SIOV
167-230SVA/SVM은 I/O hardware와 processor가 같은 address space를 공유하게 합니다. Shared Virtual Memory라는 이름도 쓰지만 Linux community는 POSIX Shared Memory와 Secure Virtual Machine과의 혼동을 피하려고 SVA를 선택했습니다.
PASID는 PCIe가 정의한 Transaction Layer Packet(TLP) prefix이며 OS가 allocate·manage하는 20-bit number입니다. platform과 device 사이 모든 transaction에 포함됩니다.
전통적인 userspace-hardware 상호작용은 process마다 별도 hardware instance가 필요합니다. 예를 들어 doorbell은 process isolation을 위해 4k, 즉 page-size 간격이 필요해 MMIO space를 크게 예약해야 하므로 thread 수가 늘면 확장하기 어렵습니다.
SWQ는 hardware가 queue depth를 관리하므로 consumer가 추적할 필요가 없고 command 공간이 없으면 retry error를 반환합니다. user는 device의 Deferrable Memory Write(DMWr) completer capability를 확인해 지원할 때만 `ENQCMD`를 submit해야 합니다. 모든 switch port도 DMWr routing을 지원하고 PCIe subsystem이 이를 활성화해야 합니다.
SWQ는 device address 하나만 마련하면 됩니다. `ENQCMD`에 process PASID가 들어가므로 device가 submitter를 구분해 많은 process로 확장할 수 있습니다.
shared workqueue 통신은 full userspace device driver보다 단순합니다. kernel driver가 hardware를 초기화하고 userspace는 work submit과 completion 처리만 담당합니다.
SR-IOV는 traditional BAR, MSI-X interrupt 공간, 자체 register layout을 갖춘 거의 완전한 독립 hardware interface를 제공합니다. VF는 PF driver의 도움을 받습니다.
SIOV는 PASID를 기반으로 virtualization용 device instance를 만듭니다. host software가 virtual device 생성을 돕고 각 instance는 PASID와 device의 bus/device/function으로 표현됩니다. demand에 따라 동적으로 늘릴 수 있는 반면 SR-IOV 생성과 관리는 정적입니다.
FAQ: VF 비용, virtualization과 memory pinning
231-272application마다 SR-IOV VF를 만드는 것은 비쌉니다. PCI config space와 MSI-X 같은 interrupt hardware를 복제하고 생성 시 resource를 hard partition해야 해 demand에 따라 동적으로 확장할 수 없습니다. VF는 PF와 완전히 독립적이지 않아 PF driver와 통신·지원이 필요한 경우가 많습니다.
SIOV는 configuration과 control을 slow path가 중재하는 software-defined device를 만들고 work submission과 completion은 중재 없이 진행합니다.
`ENQCMD`는 guest VM 안에서도 사용할 수 있습니다. VMM이 Guest PASID를 Host PASID로 바꾸는 translation table 설정을 돕습니다. 자세한 내용은 ENQCMD instruction-set reference를 참고하십시오.
device와 platform IOMMU가 SVA를 지원하면 DMA를 위해 memory를 pin할 필요가 없습니다. device는 ATS request로 IOMMU에 address lookup을 요청합니다. mapping은 있지만 OS가 page를 allocate하지 않았다면 IOMMU hardware는 mapping이 없다고 반환합니다.
device는 PRI로 virtual address mapping을 요청합니다. OS가 mapping을 완료해 response를 돌려주면 device가 translation을 다시 요청하고 계속합니다.
IOMMU는 OS와 함께 page table과 device의 consistency를 관리합니다. page를 제거할 때 OS mapping을 없애기 전에 device와 상호작용해 cache된 device TLB entry를 제거합니다.
참고 자료
273-286| 주제 | URL |
|---|---|
| VT-D | https://01.org/blogs/ashokraj/2018/recent-enhancements-intel-virtualization-technology-directed-i/o-intel-vt-d |
| SIOV | https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux |
| ENQCMD in ISE | https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf |
| DSA spec | https://software.intel.com/sites/default/files/341204-intel-data-streaming-accelerator-spec.pdf |
요약과 해설
sva.rst:1-286SVA는 CPU와 device가 같은 virtual address를 사용하게 하며 PASID가 process address space를 transaction에 tag합니다. ATS가 translation을 cache하고 PRI가 page-in을 요청하며 IOMMU와 `mmu_notifier()`가 CPU/device TLB를 동기화합니다.
`ENQCMD`는 shared workqueue에 work를 atomic submit하고 accept status를 돌려줍니다. `IA32_PASID`는 첫 submit의 `#GP` path에서 lazy load되며 process의 모든 thread와 device가 PASID 하나를 공유합니다.