요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
PCI pass-thru devices
=========================
In a Hyper-V guest VM, PCI pass-thru devices (also called
virtual PCI devices, or vPCI devices) are physical PCI devices
that are mapped directly into the VM's physical address space.
Guest device drivers can interact directly with the hardware
without intermediation by the host hypervisor. This approach
provides higher bandwidth access to the device with lower
latency, compared with devices that are virtualized by the
hypervisor. The device should appear to the guest just as it
would when running on bare metal, so no changes are required
to the Linux device drivers for the device.
Hyper-V terminology for vPCI devices is "Discrete Device
Assignment" (DDA). Public documentation for Hyper-V DDA is
available here: `DDA`_
.. _DDA: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment
DDA is typically used for storage controllers, such as NVMe,
and for GPUs. A similar mechanism for NICs is called SR-IOV
and produces the same benefits by allowing a guest device
driver to interact directly with the hardware. See Hyper-V
public documentation here: `SR-IOV`_
.. _SR-IOV: https://learn.microsoft.com/en-us/windows-hardware/drivers/network/overview-of-single-root-i-o-virtualization--sr-iov-
This discussion of vPCI devices includes DDA and SR-IOV
devices.
Device Presentation
-------------------
Hyper-V provides full PCI functionality for a vPCI device when
it is operating, so the Linux device driver for the device can
be used unchanged, provided it uses the correct Linux kernel
APIs for accessing PCI config space and for other integration
with Linux. But the initial detection of the PCI device and
its integration with the Linux PCI subsystem must use Hyper-V
specific mechanisms. Consequently, vPCI devices on Hyper-V
have a dual identity. They are initially presented to Linux
guests as VMBus devices via the standard VMBus "offer"
mechanism, so they have a VMBus identity and appear under
/sys/bus/vmbus/devices. The VMBus vPCI driver in Linux at
drivers/pci/controller/pci-hyperv.c handles a newly introduced
vPCI device by fabricating a PCI bus topology and creating all
the normal PCI device data structures in Linux that would
exist if the PCI device were discovered via ACPI on a bare-
metal system. Once those data structures are set up, the
device also has a normal PCI identity in Linux, and the normal
Linux device driver for the vPCI device can function as if it
were running in Linux on bare-metal. Because vPCI devices are
presented dynamically through the VMBus offer mechanism, they
do not appear in the Linux guest's ACPI tables. vPCI devices
may be added to a VM or removed from a VM at any time during
the life of the VM, and not just during initial boot.
With this approach, the vPCI device is a VMBus device and a
PCI device at the same time. In response to the VMBus offer
message, the hv_pci_probe() function runs and establishes a
VMBus connection to the vPCI VSP on the Hyper-V host. That
connection has a single VMBus channel. The channel is used to
exchange messages with the vPCI VSP for the purpose of setting
up and configuring the vPCI device in Linux. Once the device
is fully configured in Linux as a PCI device, the VMBus
channel is used only if Linux changes the vCPU to be interrupted
in the guest, or if the vPCI device is removed from
the VM while the VM is running. The ongoing operation of the
device happens directly between the Linux device driver for
the device and the hardware, with VMBus and the VMBus channel
playing no role.
PCI Device Setup
----------------
PCI device setup follows a sequence that Hyper-V originally
created for Windows guests, and that can be ill-suited for
Linux guests due to differences in the overall structure of
the Linux PCI subsystem compared with Windows. Nonetheless,
with a bit of hackery in the Hyper-V virtual PCI driver for
Linux, the virtual PCI device is setup in Linux so that
generic Linux PCI subsystem code and the Linux driver for the
device "just work".
Each vPCI device is set up in Linux to be in its own PCI
domain with a host bridge. The PCI domainID is derived from
bytes 4 and 5 of the instance GUID assigned to the VMBus vPCI
device. The Hyper-V host does not guarantee that these bytes
are unique, so hv_pci_probe() has an algorithm to resolve
collisions. The collision resolution is intended to be stable
across reboots of the same VM so that the PCI domainIDs don't
change, as the domainID appears in the user space
configuration of some devices.
hv_pci_probe() allocates a guest MMIO range to be used as PCI
config space for the device. This MMIO range is communicated
to the Hyper-V host over the VMBus channel as part of telling
the host that the device is ready to enter d0. See
hv_pci_enter_d0(). When the guest subsequently accesses this
MMIO range, the Hyper-V host intercepts the accesses and maps
them to the physical device PCI config space.
hv_pci_probe() also gets BAR information for the device from
the Hyper-V host, and uses this information to allocate MMIO
space for the BARs. That MMIO space is then setup to be
associated with the host bridge so that it works when generic
PCI subsystem code in Linux processes the BARs.
Finally, hv_pci_probe() creates the root PCI bus. At this
point the Hyper-V virtual PCI driver hackery is done, and the
normal Linux PCI machinery for scanning the root bus works to
detect the device, to perform driver matching, and to
initialize the driver and device.
PCI Device Removal
------------------
A Hyper-V host may initiate removal of a vPCI device from a
guest VM at any time during the life of the VM. The removal
is instigated by an admin action taken on the Hyper-V host and
is not under the control of the guest OS.
A guest VM is notified of the removal by an unsolicited
"Eject" message sent from the host to the guest over the VMBus
channel associated with the vPCI device. Upon receipt of such
a message, the Hyper-V virtual PCI driver in Linux
asynchronously invokes Linux kernel PCI subsystem calls to
shutdown and remove the device. When those calls are
complete, an "Ejection Complete" message is sent back to
Hyper-V over the VMBus channel indicating that the device has
been removed. At this point, Hyper-V sends a VMBus rescind
message to the Linux guest, which the VMBus driver in Linux
processes by removing the VMBus identity for the device. Once
that processing is complete, all vestiges of the device having
been present are gone from the Linux kernel. The rescind
message also indicates to the guest that Hyper-V has stopped
providing support for the vPCI device in the guest. If the
guest were to attempt to access that device's MMIO space, it
would be an invalid reference. Hypercalls affecting the device
return errors, and any further messages sent in the VMBus
channel are ignored.
After sending the Eject message, Hyper-V allows the guest VM
60 seconds to cleanly shutdown the device and respond with
Ejection Complete before sending the VMBus rescind
message. If for any reason the Eject steps don't complete
within the allowed 60 seconds, the Hyper-V host forcibly
performs the rescind steps, which will likely result in
cascading errors in the guest because the device is now no
longer present from the guest standpoint and accessing the
device MMIO space will fail.
Because ejection is asynchronous and can happen at any point
during the guest VM lifecycle, proper synchronization in the
Hyper-V virtual PCI driver is very tricky. Ejection has been
observed even before a newly offered vPCI device has been
fully setup. The Hyper-V virtual PCI driver has been updated
several times over the years to fix race conditions when
ejections happen at inopportune times. Care must be taken when
modifying this code to prevent re-introducing such problems.
See comments in the code.
Interrupt Assignment
--------------------
The Hyper-V virtual PCI driver supports vPCI devices using
MSI, multi-MSI, or MSI-X. Assigning the guest vCPU that will
receive the interrupt for a particular MSI or MSI-X message is
complex because of the way the Linux setup of IRQs maps onto
the Hyper-V interfaces. For the single-MSI and MSI-X cases,
Linux calls hv_compse_msi_msg() twice, with the first call
containing a dummy vCPU and the second call containing the
real vCPU. Furthermore, hv_irq_unmask() is finally called
(on x86) or the GICD registers are set (on arm64) to specify
the real vCPU again. Each of these three calls interact
with Hyper-V, which must decide which physical CPU should
receive the interrupt before it is forwarded to the guest VM.
Unfortunately, the Hyper-V decision-making process is a bit
limited, and can result in concentrating the physical
interrupts on a single CPU, causing a performance bottleneck.
See details about how this is resolved in the extensive
comment above the function hv_compose_msi_req_get_cpu().
The Hyper-V virtual PCI driver implements the
irq_chip.irq_compose_msi_msg function as hv_compose_msi_msg().
Unfortunately, on Hyper-V the implementation requires sending
a VMBus message to the Hyper-V host and awaiting an interrupt
indicating receipt of a reply message. Since
irq_chip.irq_compose_msi_msg can be called with IRQ locks
held, it doesn't work to do the normal sleep until awakened by
the interrupt. Instead hv_compose_msi_msg() must send the
VMBus message, and then poll for the completion message. As
further complexity, the vPCI device could be ejected/rescinded
while the polling is in progress, so this scenario must be
detected as well. See comments in the code regarding this
very tricky area.
Most of the code in the Hyper-V virtual PCI driver (pci-
hyperv.c) applies to Hyper-V and Linux guests running on x86
and on arm64 architectures. But there are differences in how
interrupt assignments are managed. On x86, the Hyper-V
virtual PCI driver in the guest must make a hypercall to tell
Hyper-V which guest vCPU should be interrupted by each
MSI/MSI-X interrupt, and the x86 interrupt vector number that
the x86_vector IRQ domain has picked for the interrupt. This
hypercall is made by hv_arch_irq_unmask(). On arm64, the
Hyper-V virtual PCI driver manages the allocation of an SPI
for each MSI/MSI-X interrupt. The Hyper-V virtual PCI driver
stores the allocated SPI in the architectural GICD registers,
which Hyper-V emulates, so no hypercall is necessary as with
x86. Hyper-V does not support using LPIs for vPCI devices in
arm64 guest VMs because it does not emulate a GICv3 ITS.
The Hyper-V virtual PCI driver in Linux supports vPCI devices
whose drivers create managed or unmanaged Linux IRQs. If the
smp_affinity for an unmanaged IRQ is updated via the /proc/irq
interface, the Hyper-V virtual PCI driver is called to tell
the Hyper-V host to change the interrupt targeting and
everything works properly. However, on x86 if the x86_vector
IRQ domain needs to reassign an interrupt vector due to
running out of vectors on a CPU, there's no path to inform the
Hyper-V host of the change, and things break. Fortunately,
guest VMs operate in a constrained device environment where
using all the vectors on a CPU doesn't happen. Since such a
problem is only a theoretical concern rather than a practical
concern, it has been left unaddressed.
DMA
---
By default, Hyper-V pins all guest VM memory in the host
when the VM is created, and programs the physical IOMMU to
allow the VM to have DMA access to all its memory. Hence
it is safe to assign PCI devices to the VM, and allow the
guest operating system to program the DMA transfers. The
physical IOMMU prevents a malicious guest from initiating
DMA to memory belonging to the host or to other VMs on the
host. From the Linux guest standpoint, such DMA transfers
are in "direct" mode since Hyper-V does not provide a virtual
IOMMU in the guest.
Hyper-V assumes that physical PCI devices always perform
cache-coherent DMA. When running on x86, this behavior is
required by the architecture. When running on arm64, the
architecture allows for both cache-coherent and
non-cache-coherent devices, with the behavior of each device
specified in the ACPI DSDT. But when a PCI device is assigned
to a guest VM, that device does not appear in the DSDT, so the
Hyper-V VMBus driver propagates cache-coherency information
from the VMBus node in the ACPI DSDT to all VMBus devices,
including vPCI devices (since they have a dual identity as a VMBus
device and as a PCI device). See vmbus_dma_configure().
Current Hyper-V versions always indicate that the VMBus is
cache coherent, so vPCI devices on arm64 always get marked as
cache coherent and the CPU does not perform any sync
operations as part of dma_map/unmap_*() calls.
vPCI protocol versions
----------------------
As previously described, during vPCI device setup and teardown
messages are passed over a VMBus channel between the Hyper-V
host and the Hyper-v vPCI driver in the Linux guest. Some
messages have been revised in newer versions of Hyper-V, so
the guest and host must agree on the vPCI protocol version to
be used. The version is negotiated when communication over
the VMBus channel is first established. See
hv_pci_protocol_negotiation(). Newer versions of the protocol
extend support to VMs with more than 64 vCPUs, and provide
additional information about the vPCI device, such as the
guest virtual NUMA node to which it is most closely affined in
the underlying hardware.
Guest NUMA node affinity
------------------------
When the vPCI protocol version provides it, the guest NUMA
node affinity of the vPCI device is stored as part of the Linux
device information for subsequent use by the Linux driver. See
hv_pci_assign_numa_node(). If the negotiated protocol version
does not support the host providing NUMA affinity information,
the Linux guest defaults the device NUMA node to 0. But even
when the negotiated protocol version includes NUMA affinity
information, the ability of the host to provide such
information depends on certain host configuration options. If
the guest receives NUMA node value "0", it could mean NUMA
node 0, or it could mean "no information is available".
Unfortunately it is not possible to distinguish the two cases
from the guest side.
PCI config space access in a CoCo VM
------------------------------------
Linux PCI device drivers access PCI config space using a
standard set of functions provided by the Linux PCI subsystem.
In Hyper-V guests these standard functions map to functions
hv_pcifront_read_config() and hv_pcifront_write_config()
in the Hyper-V virtual PCI driver. In normal VMs,
these hv_pcifront_*() functions directly access the PCI config
space, and the accesses trap to Hyper-V to be handled.
But in CoCo VMs, memory encryption prevents Hyper-V
from reading the guest instruction stream to emulate the
access, so the hv_pcifront_*() functions must invoke
hypercalls with explicit arguments describing the access to be
made.
Config Block back-channel
-------------------------
The Hyper-V host and Hyper-V virtual PCI driver in Linux
together implement a non-standard back-channel communication
path between the host and guest. The back-channel path uses
messages sent over the VMBus channel associated with the vPCI
device. The functions hyperv_read_cfg_blk() and
hyperv_write_cfg_blk() are the primary interfaces provided to
other parts of the Linux kernel. As of this writing, these
interfaces are used only by the Mellanox mlx5 driver to pass
diagnostic data to a Hyper-V host running in the Azure public
cloud. The functions hyperv_read_cfg_blk() and
hyperv_write_cfg_blk() are implemented in a separate module
(pci-hyperv-intf.c, under CONFIG_PCI_HYPERV_INTERFACE) that
effectively stubs them out when running in non-Hyper-V
environments.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
PCI pass-through, DDA와 SR-IOV
1-31Hyper-V guest VM에서 PCI pass-through device는 virtual PCI device 또는 vPCI device라고도 하며, VM의 physical address space에 직접 mapping된 physical PCI device입니다.
Guest device driver는 host hypervisor의 중개 없이 hardware와 직접 상호 작용할 수 있습니다. Hypervisor가 virtualize하는 device보다 latency가 낮고 bandwidth가 높은 접근을 제공합니다.
Device는 bare-metal에서 실행할 때와 똑같이 guest에 보여야 하므로 해당 device의 Linux driver를 변경할 필요가 없습니다.
Hyper-V 용어로 vPCI device는 Discrete Device Assignment(DDA)입니다. 원문은 Microsoft의 Hyper-V DDA 공개 문서 링크를 제공합니다.
DDA는 보통 NVMe 같은 storage controller와 GPU에 사용합니다. NIC에는 유사한 SR-IOV mechanism을 사용하며 guest driver가 hardware와 직접 통신하게 해 같은 이점을 제공합니다. 이 문서에서 vPCI device라는 말은 DDA와 SR-IOV device를 모두 포함합니다.
대상 장치와 공통 효과를 정리합니다.
.. SPDX-License-Identifier: GPL-2.0
PCI pass-thru devices
=========================
In a Hyper-V guest VM, PCI pass-thru devices (also called
virtual PCI devices, or vPCI devices) are physical PCI devices
that are mapped directly into the VM's physical address space.
Guest device drivers can interact directly with the hardware
without intermediation by the host hypervisor. This approach
provides higher bandwidth access to the device with lower
latency, compared with devices that are virtualized by the
hypervisor. The device should appear to the guest just as it
would when running on bare metal, so no changes are required
to the Linux device drivers for the device.
Hyper-V terminology for vPCI devices is "Discrete Device
Assignment" (DDA). Public documentation for Hyper-V DDA is
available here: `DDA`_
.. _DDA: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment
DDA is typically used for storage controllers, such as NVMe,
and for GPUs. A similar mechanism for NICs is called SR-IOV
and produces the same benefits by allowing a guest device
driver to interact directly with the hardware. See Hyper-V
public documentation here: `SR-IOV`_
.. _SR-IOV: https://learn.microsoft.com/en-us/windows-hardware/drivers/network/overview-of-single-root-i-o-virtualization--sr-iov-
This discussion of vPCI devices includes DDA and SR-IOV
devices.Device presentation과 이중 신원
32-73Hyper-V는 동작 중인 vPCI device에 완전한 PCI 기능을 제공합니다. Driver가 PCI config space 접근과 Linux 통합에 올바른 kernel API를 사용한다면 기존 Linux device driver를 수정 없이 사용할 수 있습니다.
하지만 최초 PCI device 탐지와 Linux PCI subsystem 통합에는 Hyper-V 전용 mechanism이 필요합니다. 따라서 Hyper-V의 vPCI device는 VMBus device와 PCI device라는 이중 신원을 가집니다.
처음에는 표준 VMBus `offer` mechanism으로 Linux guest에 제시되므로 VMBus identity가 생기고 `/sys/bus/vmbus/devices` 아래에 나타납니다.
`drivers/pci/controller/pci-hyperv.c`의 Linux VMBus vPCI driver는 새 vPCI device를 받으면 PCI bus topology를 인위적으로 만들고, bare-metal에서 ACPI로 device를 발견했을 때 존재했을 정상 PCI device data structure를 모두 생성합니다.
이 data structure 설정이 끝나면 device는 정상적인 Linux PCI identity도 얻습니다. 이후 일반 Linux driver는 bare-metal에서처럼 동작합니다.
vPCI device는 VMBus offer를 통해 동적으로 제시되므로 Linux guest의 ACPI table에는 나타나지 않습니다. Initial boot 때뿐 아니라 VM이 살아 있는 동안 언제든 추가하거나 제거할 수 있습니다.
장치 도입 단계와 정상 실행 단계의 표현을 구분합니다.
VMBus offer에 응답해 `hv_pci_probe()`가 실행되고 Hyper-V host의 vPCI VSP와 VMBus connection을 만듭니다. 이 connection은 VMBus channel 하나를 가지며 Linux 안에서 vPCI device를 설정하고 구성하는 message 교환에 사용합니다.
PCI device 구성이 끝난 뒤 VMBus channel은 Linux가 guest에서 interrupt 받을 vCPU를 변경하거나 실행 중인 VM에서 vPCI device가 제거될 때만 사용됩니다. 일상적인 device 동작은 Linux device driver와 hardware 사이에서 직접 이루어지며 VMBus와 channel은 관여하지 않습니다.
VMBus offer에서 일반 PCI device로 전환되는 흐름입니다.
Device Presentation
-------------------
Hyper-V provides full PCI functionality for a vPCI device when
it is operating, so the Linux device driver for the device can
be used unchanged, provided it uses the correct Linux kernel
APIs for accessing PCI config space and for other integration
with Linux. But the initial detection of the PCI device and
its integration with the Linux PCI subsystem must use Hyper-V
specific mechanisms. Consequently, vPCI devices on Hyper-V
have a dual identity. They are initially presented to Linux
guests as VMBus devices via the standard VMBus "offer"
mechanism, so they have a VMBus identity and appear under
/sys/bus/vmbus/devices. The VMBus vPCI driver in Linux at
drivers/pci/controller/pci-hyperv.c handles a newly introduced
vPCI device by fabricating a PCI bus topology and creating all
the normal PCI device data structures in Linux that would
exist if the PCI device were discovered via ACPI on a bare-
metal system. Once those data structures are set up, the
device also has a normal PCI identity in Linux, and the normal
Linux device driver for the vPCI device can function as if it
were running in Linux on bare-metal. Because vPCI devices are
presented dynamically through the VMBus offer mechanism, they
do not appear in the Linux guest's ACPI tables. vPCI devices
may be added to a VM or removed from a VM at any time during
the life of the VM, and not just during initial boot.
With this approach, the vPCI device is a VMBus device and a
PCI device at the same time. In response to the VMBus offer
message, the hv_pci_probe() function runs and establishes a
VMBus connection to the vPCI VSP on the Hyper-V host. That
connection has a single VMBus channel. The channel is used to
exchange messages with the vPCI VSP for the purpose of setting
up and configuring the vPCI device in Linux. Once the device
is fully configured in Linux as a PCI device, the VMBus
channel is used only if Linux changes the vCPU to be interrupted
in the guest, or if the vPCI device is removed from
the VM while the VM is running. The ongoing operation of the
device happens directly between the Linux device driver for
the device and the hardware, with VMBus and the VMBus channel
playing no role.
PCI device 설정
74-114PCI device setup은 Hyper-V가 원래 Windows guest용으로 만든 sequence를 따릅니다. Linux PCI subsystem의 전체 구조가 Windows와 달라 Linux guest에는 잘 맞지 않을 수 있지만 Hyper-V virtual PCI driver의 몇 가지 우회 구현으로 generic Linux PCI code와 device driver가 그대로 동작하게 합니다.
Linux에서는 vPCI device마다 host bridge가 있는 별도 PCI domain을 만듭니다. PCI domainID는 VMBus vPCI device의 instance GUID 중 4번과 5번 byte에서 유도합니다.
Hyper-V host는 두 byte의 고유성을 보장하지 않으므로 `hv_pci_probe()`에는 충돌 해결 algorithm이 있습니다. 일부 device의 user-space configuration에 domainID가 기록되므로 같은 VM을 reboot해도 domainID가 바뀌지 않도록 안정적인 해를 고르는 것이 목적입니다.
`hv_pci_probe()`는 device의 PCI config space로 사용할 guest MMIO range를 할당합니다. Device가 d0 state에 들어갈 준비가 됐다고 host에 알리는 과정에서 이 range를 VMBus channel로 전달합니다. 관련 함수는 `hv_pci_enter_d0()`입니다.
Guest가 나중에 이 MMIO range에 접근하면 Hyper-V host가 access를 intercept해 physical device의 PCI config space로 mapping합니다.
`hv_pci_probe()`는 host에서 device BAR 정보도 받아 BAR용 MMIO space를 할당합니다. Generic Linux PCI subsystem이 BAR를 처리할 때 올바르게 동작하도록 이 MMIO space를 host bridge와 연결합니다.
마지막으로 `hv_pci_probe()`가 root PCI bus를 생성합니다. 이 시점부터 Hyper-V virtual PCI driver의 우회 작업은 끝나고 정상 Linux PCI machinery가 root bus를 scan해 device를 발견하고 driver matching과 driver·device 초기화를 수행합니다.
Hyper-V 전용 준비가 generic PCI scan으로 이어집니다.
Config access와 BAR access를 위한 설정을 구분합니다.
PCI Device Setup
----------------
PCI device setup follows a sequence that Hyper-V originally
created for Windows guests, and that can be ill-suited for
Linux guests due to differences in the overall structure of
the Linux PCI subsystem compared with Windows. Nonetheless,
with a bit of hackery in the Hyper-V virtual PCI driver for
Linux, the virtual PCI device is setup in Linux so that
generic Linux PCI subsystem code and the Linux driver for the
device "just work".
Each vPCI device is set up in Linux to be in its own PCI
domain with a host bridge. The PCI domainID is derived from
bytes 4 and 5 of the instance GUID assigned to the VMBus vPCI
device. The Hyper-V host does not guarantee that these bytes
are unique, so hv_pci_probe() has an algorithm to resolve
collisions. The collision resolution is intended to be stable
across reboots of the same VM so that the PCI domainIDs don't
change, as the domainID appears in the user space
configuration of some devices.
hv_pci_probe() allocates a guest MMIO range to be used as PCI
config space for the device. This MMIO range is communicated
to the Hyper-V host over the VMBus channel as part of telling
the host that the device is ready to enter d0. See
hv_pci_enter_d0(). When the guest subsequently accesses this
MMIO range, the Hyper-V host intercepts the accesses and maps
them to the physical device PCI config space.
hv_pci_probe() also gets BAR information for the device from
the Hyper-V host, and uses this information to allocate MMIO
space for the BARs. That MMIO space is then setup to be
associated with the host bridge so that it works when generic
PCI subsystem code in Linux processes the BARs.
Finally, hv_pci_probe() creates the root PCI bus. At this
point the Hyper-V virtual PCI driver hackery is done, and the
normal Linux PCI machinery for scanning the root bus works to
detect the device, to perform driver matching, and to
initialize the driver and device.
PCI device 제거
115-160Hyper-V host는 VM이 살아 있는 동안 언제든 vPCI device 제거를 시작할 수 있습니다. Host 관리자의 동작으로 시작되며 guest OS가 제어하지 않습니다.
Host는 해당 vPCI device의 VMBus channel로 unsolicited `Eject` message를 보내 제거 사실을 guest에 알립니다. Linux Hyper-V virtual PCI driver는 이를 받으면 Linux kernel PCI subsystem 호출을 비동기로 실행해 device를 shutdown하고 제거합니다.
PCI 제거 호출이 끝나면 guest는 같은 VMBus channel로 `Ejection Complete` message를 보내 device가 제거됐다고 알립니다. Hyper-V는 이어서 VMBus rescind message를 보냅니다.
Linux VMBus driver가 rescind를 처리하면 device의 VMBus identity도 제거되고 kernel에서 device가 있었던 흔적이 모두 사라집니다. Rescind는 Hyper-V가 guest의 해당 vPCI device 지원을 중단했다는 뜻이기도 합니다.
그 뒤 해당 device MMIO space에 접근하면 invalid reference가 됩니다. Device에 영향을 주는 hypercall은 error를 반환하고 VMBus channel로 보내는 추가 message도 무시됩니다.
PCI identity를 먼저 제거하고 VMBus identity를 나중에 정리합니다.
Hyper-V는 Eject를 보낸 뒤 guest가 clean shutdown을 마치고 Ejection Complete로 응답할 때까지 60초를 기다립니다. 60초 안에 끝나지 않으면 host가 강제로 rescind 단계로 넘어갑니다.
강제 rescind 뒤 guest 관점에서는 device가 이미 사라졌으므로 MMIO access가 실패하고 연쇄 error가 발생할 가능성이 큽니다.
Ejection은 비동기이며 VM 수명 주기의 어느 순간에도 일어날 수 있어 Hyper-V virtual PCI driver의 올바른 synchronization이 매우 어렵습니다. 새로 offer된 vPCI device가 완전히 설정되기 전에 ejection이 관찰된 적도 있습니다.
불리한 시점의 ejection race를 고치기 위해 driver는 여러 해에 걸쳐 여러 번 갱신됐습니다. Code를 수정할 때 race condition을 다시 만들지 않도록 code comment를 주의 깊게 확인해야 합니다.
정상 완료와 timeout 후 강제 rescind를 비교합니다.
PCI Device Removal
------------------
A Hyper-V host may initiate removal of a vPCI device from a
guest VM at any time during the life of the VM. The removal
is instigated by an admin action taken on the Hyper-V host and
is not under the control of the guest OS.
A guest VM is notified of the removal by an unsolicited
"Eject" message sent from the host to the guest over the VMBus
channel associated with the vPCI device. Upon receipt of such
a message, the Hyper-V virtual PCI driver in Linux
asynchronously invokes Linux kernel PCI subsystem calls to
shutdown and remove the device. When those calls are
complete, an "Ejection Complete" message is sent back to
Hyper-V over the VMBus channel indicating that the device has
been removed. At this point, Hyper-V sends a VMBus rescind
message to the Linux guest, which the VMBus driver in Linux
processes by removing the VMBus identity for the device. Once
that processing is complete, all vestiges of the device having
been present are gone from the Linux kernel. The rescind
message also indicates to the guest that Hyper-V has stopped
providing support for the vPCI device in the guest. If the
guest were to attempt to access that device's MMIO space, it
would be an invalid reference. Hypercalls affecting the device
return errors, and any further messages sent in the VMBus
channel are ignored.
After sending the Eject message, Hyper-V allows the guest VM
60 seconds to cleanly shutdown the device and respond with
Ejection Complete before sending the VMBus rescind
message. If for any reason the Eject steps don't complete
within the allowed 60 seconds, the Hyper-V host forcibly
performs the rescind steps, which will likely result in
cascading errors in the guest because the device is now no
longer present from the guest standpoint and accessing the
device MMIO space will fail.
Because ejection is asynchronous and can happen at any point
during the guest VM lifecycle, proper synchronization in the
Hyper-V virtual PCI driver is very tricky. Ejection has been
observed even before a newly offered vPCI device has been
fully setup. The Hyper-V virtual PCI driver has been updated
several times over the years to fix race conditions when
ejections happen at inopportune times. Care must be taken when
modifying this code to prevent re-introducing such problems.
See comments in the code.Interrupt 할당
161-225Hyper-V virtual PCI driver는 MSI, multi-MSI, MSI-X를 사용하는 vPCI device를 지원합니다. 특정 MSI 또는 MSI-X message를 받을 guest vCPU를 할당하는 과정은 Linux IRQ 설정과 Hyper-V interface가 mapping되는 방식 때문에 복잡합니다.
Single-MSI와 MSI-X에서는 Linux가 `hv_compose_msi_msg()`를 두 번 호출합니다. 첫 호출에는 dummy vCPU가, 두 번째 호출에는 실제 vCPU가 들어갑니다. 이어서 x86에서는 `hv_irq_unmask()`를 호출하고 arm64에서는 GICD register를 설정해 실제 vCPU를 다시 지정합니다.
세 호출은 모두 Hyper-V와 상호 작용합니다. Hyper-V는 interrupt를 guest VM에 전달하기 전에 어느 physical CPU가 받을지 결정해야 하지만 이 의사 결정은 제약이 있어 physical interrupt가 CPU 하나에 몰리고 성능 병목이 생길 수 있습니다. 해결 방법은 `hv_compose_msi_req_get_cpu()` 위의 긴 comment를 참조합니다.
같은 실제 vCPU 정보가 Hyper-V에 여러 단계로 전달됩니다.
Linux Hyper-V virtual PCI driver는 `irq_chip.irq_compose_msi_msg`를 `hv_compose_msi_msg()`로 구현합니다. Hyper-V에서는 이 구현이 host에 VMBus message를 보내고 reply 수신 interrupt를 기다려야 합니다.
하지만 IRQ lock을 잡은 상태에서 `irq_chip.irq_compose_msi_msg`가 호출될 수 있어 일반적인 sleep-and-wakeup 방식은 사용할 수 없습니다. 따라서 `hv_compose_msi_msg()`는 VMBus message를 보낸 뒤 completion message를 polling해야 합니다.
Polling 중 vPCI device가 eject 또는 rescind될 수도 있으므로 이 상황도 감지해야 합니다. 원문은 이 까다로운 영역의 code comment를 확인하라고 강조합니다.
`pci-hyperv.c`의 대부분은 x86과 arm64 Hyper-V Linux guest에서 공통이지만 interrupt assignment는 architecture별로 다릅니다.
x86에서는 guest Hyper-V virtual PCI driver가 hypercall로 각 MSI/MSI-X interrupt가 향할 guest vCPU와 `x86_vector` IRQ domain이 선택한 x86 interrupt vector 번호를 Hyper-V에 알려야 합니다. 이 hypercall은 `hv_arch_irq_unmask()`가 수행합니다.
Arm64에서는 driver가 MSI/MSI-X interrupt마다 SPI를 할당하고 Hyper-V가 emulate하는 architecture GICD register에 기록합니다. x86 같은 hypercall은 필요하지 않습니다. Hyper-V는 GICv3 ITS를 emulate하지 않으므로 arm64 guest의 vPCI device에 LPI를 사용할 수 없습니다.
Guest vCPU와 interrupt 번호를 Hyper-V에 전달하는 방법입니다.
Linux driver는 managed 또는 unmanaged Linux IRQ를 만드는 vPCI device를 모두 지원합니다. Unmanaged IRQ의 `smp_affinity`를 `/proc/irq`에서 바꾸면 Hyper-V virtual PCI driver가 host에 target 변경을 알려 정상 동작합니다.
다만 x86에서 CPU의 vector가 부족해 `x86_vector` IRQ domain이 interrupt vector를 재할당해야 할 때는 변경을 Hyper-V host에 알릴 경로가 없어 동작이 깨집니다. Guest VM의 제한된 device 환경에서는 CPU vector를 모두 쓰는 일이 사실상 없어 이 문제는 실무보다 이론적 우려로 남겨 두었습니다.
IRQ lock context에서도 host reply를 받아야 하는 제약입니다.
Interrupt Assignment
--------------------
The Hyper-V virtual PCI driver supports vPCI devices using
MSI, multi-MSI, or MSI-X. Assigning the guest vCPU that will
receive the interrupt for a particular MSI or MSI-X message is
complex because of the way the Linux setup of IRQs maps onto
the Hyper-V interfaces. For the single-MSI and MSI-X cases,
Linux calls hv_compse_msi_msg() twice, with the first call
containing a dummy vCPU and the second call containing the
real vCPU. Furthermore, hv_irq_unmask() is finally called
(on x86) or the GICD registers are set (on arm64) to specify
the real vCPU again. Each of these three calls interact
with Hyper-V, which must decide which physical CPU should
receive the interrupt before it is forwarded to the guest VM.
Unfortunately, the Hyper-V decision-making process is a bit
limited, and can result in concentrating the physical
interrupts on a single CPU, causing a performance bottleneck.
See details about how this is resolved in the extensive
comment above the function hv_compose_msi_req_get_cpu().
The Hyper-V virtual PCI driver implements the
irq_chip.irq_compose_msi_msg function as hv_compose_msi_msg().
Unfortunately, on Hyper-V the implementation requires sending
a VMBus message to the Hyper-V host and awaiting an interrupt
indicating receipt of a reply message. Since
irq_chip.irq_compose_msi_msg can be called with IRQ locks
held, it doesn't work to do the normal sleep until awakened by
the interrupt. Instead hv_compose_msi_msg() must send the
VMBus message, and then poll for the completion message. As
further complexity, the vPCI device could be ejected/rescinded
while the polling is in progress, so this scenario must be
detected as well. See comments in the code regarding this
very tricky area.
Most of the code in the Hyper-V virtual PCI driver (pci-
hyperv.c) applies to Hyper-V and Linux guests running on x86
and on arm64 architectures. But there are differences in how
interrupt assignments are managed. On x86, the Hyper-V
virtual PCI driver in the guest must make a hypercall to tell
Hyper-V which guest vCPU should be interrupted by each
MSI/MSI-X interrupt, and the x86 interrupt vector number that
the x86_vector IRQ domain has picked for the interrupt. This
hypercall is made by hv_arch_irq_unmask(). On arm64, the
Hyper-V virtual PCI driver manages the allocation of an SPI
for each MSI/MSI-X interrupt. The Hyper-V virtual PCI driver
stores the allocated SPI in the architectural GICD registers,
which Hyper-V emulates, so no hypercall is necessary as with
x86. Hyper-V does not support using LPIs for vPCI devices in
arm64 guest VMs because it does not emulate a GICv3 ITS.
The Hyper-V virtual PCI driver in Linux supports vPCI devices
whose drivers create managed or unmanaged Linux IRQs. If the
smp_affinity for an unmanaged IRQ is updated via the /proc/irq
interface, the Hyper-V virtual PCI driver is called to tell
the Hyper-V host to change the interrupt targeting and
everything works properly. However, on x86 if the x86_vector
IRQ domain needs to reassign an interrupt vector due to
running out of vectors on a CPU, there's no path to inform the
Hyper-V host of the change, and things break. Fortunately,
guest VMs operate in a constrained device environment where
using all the vectors on a CPU doesn't happen. Since such a
problem is only a theoretical concern rather than a practical
concern, it has been left unaddressed.
DMA와 cache coherency
226-254기본적으로 Hyper-V는 VM을 만들 때 guest memory 전체를 host에 pin하고 physical IOMMU를 설정해 VM이 자기 memory 전체에 DMA access할 수 있게 합니다.
따라서 PCI device를 VM에 할당하고 guest OS가 DMA transfer를 program해도 안전합니다. Physical IOMMU는 악의적인 guest가 host 또는 같은 host의 다른 VM memory로 DMA를 시작하지 못하게 막습니다.
Linux guest 관점에서 Hyper-V가 guest 안에 virtual IOMMU를 제공하지 않으므로 이 DMA transfer는 `direct` mode입니다.
Guest에는 direct DMA로 보이지만 host physical IOMMU가 경계를 강제합니다.
Hyper-V는 physical PCI device가 항상 cache-coherent DMA를 수행한다고 가정합니다. x86에서는 architecture가 이를 요구합니다.
Arm64는 cache-coherent device와 non-cache-coherent device를 모두 허용하며 보통 각 device의 동작을 ACPI DSDT에 기록합니다. 하지만 VM에 할당된 PCI device는 DSDT에 나타나지 않습니다.
그래서 Hyper-V VMBus driver는 ACPI DSDT의 VMBus node가 가진 cache-coherency 정보를 vPCI device를 포함한 모든 VMBus device에 전파합니다. vPCI device가 VMBus와 PCI 이중 신원을 가지기 때문에 이 전파가 적용됩니다. 관련 함수는 `vmbus_dma_configure()`입니다.
현재 Hyper-V version은 항상 VMBus가 cache coherent라고 표시합니다. 따라서 arm64의 vPCI device도 cache coherent로 표시되고 CPU는 `dma_map/unmap_*()` 호출 과정에서 sync operation을 수행하지 않습니다.
Architecture와 firmware 정보 경로를 비교합니다.
DMA
---
By default, Hyper-V pins all guest VM memory in the host
when the VM is created, and programs the physical IOMMU to
allow the VM to have DMA access to all its memory. Hence
it is safe to assign PCI devices to the VM, and allow the
guest operating system to program the DMA transfers. The
physical IOMMU prevents a malicious guest from initiating
DMA to memory belonging to the host or to other VMs on the
host. From the Linux guest standpoint, such DMA transfers
are in "direct" mode since Hyper-V does not provide a virtual
IOMMU in the guest.
Hyper-V assumes that physical PCI devices always perform
cache-coherent DMA. When running on x86, this behavior is
required by the architecture. When running on arm64, the
architecture allows for both cache-coherent and
non-cache-coherent devices, with the behavior of each device
specified in the ACPI DSDT. But when a PCI device is assigned
to a guest VM, that device does not appear in the DSDT, so the
Hyper-V VMBus driver propagates cache-coherency information
from the VMBus node in the ACPI DSDT to all VMBus devices,
including vPCI devices (since they have a dual identity as a VMBus
device and as a PCI device). See vmbus_dma_configure().
Current Hyper-V versions always indicate that the VMBus is
cache coherent, so vPCI devices on arm64 always get marked as
cache coherent and the CPU does not perform any sync
operations as part of dma_map/unmap_*() calls.
vPCI protocol version
255-269앞서 설명한 것처럼 vPCI device 설정과 teardown 동안 Hyper-V host와 Linux guest의 vPCI driver가 VMBus channel로 message를 교환합니다.
새 Hyper-V version에서 일부 message 형식이 바뀌었으므로 guest와 host는 사용할 vPCI protocol version에 합의해야 합니다. VMBus channel 통신을 처음 만들 때 `hv_pci_protocol_negotiation()`으로 version을 협상합니다.
새 protocol version은 vCPU가 64개를 넘는 VM까지 지원 범위를 넓히며, underlying hardware에서 vPCI device와 가장 가까운 guest virtual NUMA node 같은 추가 device 정보도 제공합니다.
협상 version이 제공하는 대표 기능입니다.
vPCI protocol versions
----------------------
As previously described, during vPCI device setup and teardown
messages are passed over a VMBus channel between the Hyper-V
host and the Hyper-v vPCI driver in the Linux guest. Some
messages have been revised in newer versions of Hyper-V, so
the guest and host must agree on the vPCI protocol version to
be used. The version is negotiated when communication over
the VMBus channel is first established. See
hv_pci_protocol_negotiation(). Newer versions of the protocol
extend support to VMs with more than 64 vCPUs, and provide
additional information about the vPCI device, such as the
guest virtual NUMA node to which it is most closely affined in
the underlying hardware.
Guest NUMA node affinity
270-285협상한 vPCI protocol이 guest NUMA affinity 정보를 제공하면 `hv_pci_assign_numa_node()`가 이를 Linux device 정보에 저장하고 Linux driver가 나중에 사용합니다.
협상 protocol이 host의 NUMA affinity 제공을 지원하지 않으면 Linux guest는 device NUMA node를 0으로 정합니다.
Protocol이 NUMA 정보를 지원하더라도 host가 실제 정보를 제공할 수 있는지는 특정 host configuration option에 달려 있습니다. Guest가 NUMA node 값 `0`을 받았을 때 실제 node 0인지 정보가 없다는 뜻인지 guest 쪽에서는 구분할 수 없습니다.
Protocol과 host configuration 때문에 같은 값이 두 의미를 가질 수 있습니다.
Guest NUMA node affinity
------------------------
When the vPCI protocol version provides it, the guest NUMA
node affinity of the vPCI device is stored as part of the Linux
device information for subsequent use by the Linux driver. See
hv_pci_assign_numa_node(). If the negotiated protocol version
does not support the host providing NUMA affinity information,
the Linux guest defaults the device NUMA node to 0. But even
when the negotiated protocol version includes NUMA affinity
information, the ability of the host to provide such
information depends on certain host configuration options. If
the guest receives NUMA node value "0", it could mean NUMA
node 0, or it could mean "no information is available".
Unfortunately it is not possible to distinguish the two cases
from the guest side.
CoCo VM의 PCI config space 접근
286-300Linux PCI device driver는 Linux PCI subsystem이 제공하는 표준 함수로 PCI config space에 접근합니다. Hyper-V guest에서는 이 함수들이 Hyper-V virtual PCI driver의 `hv_pcifront_read_config()`와 `hv_pcifront_write_config()`에 mapping됩니다.
일반 VM에서는 `hv_pcifront_*()` 함수가 PCI config space에 직접 접근하고 access가 trap되어 Hyper-V가 처리합니다.
CoCo VM에서는 memory encryption 때문에 Hyper-V가 guest instruction stream을 읽어 access를 emulate할 수 없습니다. 따라서 `hv_pcifront_*()` 함수가 수행할 access를 명시적인 argument로 설명해 hypercall을 호출해야 합니다.
일반 VM의 implicit trap과 CoCo VM의 explicit hypercall을 비교합니다.
PCI config space access in a CoCo VM
------------------------------------
Linux PCI device drivers access PCI config space using a
standard set of functions provided by the Linux PCI subsystem.
In Hyper-V guests these standard functions map to functions
hv_pcifront_read_config() and hv_pcifront_write_config()
in the Hyper-V virtual PCI driver. In normal VMs,
these hv_pcifront_*() functions directly access the PCI config
space, and the accesses trap to Hyper-V to be handled.
But in CoCo VMs, memory encryption prevents Hyper-V
from reading the guest instruction stream to emulate the
access, so the hv_pcifront_*() functions must invoke
hypercalls with explicit arguments describing the access to be
made.
Config Block back-channel
301-316Hyper-V host와 Linux Hyper-V virtual PCI driver는 host와 guest 사이에 비표준 back-channel communication path를 함께 구현합니다. 이 경로는 vPCI device에 연결된 VMBus channel로 message를 보냅니다.
다른 Linux kernel 영역에 제공하는 주 interface는 `hyperv_read_cfg_blk()`와 `hyperv_write_cfg_blk()`입니다.
문서 작성 시점에는 Azure public cloud의 Hyper-V host에 diagnostic data를 전달하기 위해 Mellanox `mlx5` driver만 이 interface를 사용합니다.
두 함수는 `CONFIG_PCI_HYPERV_INTERFACE` 아래 별도 module인 `pci-hyperv-intf.c`에 구현됩니다. Non-Hyper-V 환경에서는 사실상 아무 동작도 하지 않는 stub이 됩니다.
현재 mlx5 진단 data가 이동하는 비표준 경로입니다.
Build option과 runtime 환경에 따른 동작입니다.
Config Block back-channel
-------------------------
The Hyper-V host and Hyper-V virtual PCI driver in Linux
together implement a non-standard back-channel communication
path between the host and guest. The back-channel path uses
messages sent over the VMBus channel associated with the vPCI
device. The functions hyperv_read_cfg_blk() and
hyperv_write_cfg_blk() are the primary interfaces provided to
other parts of the Linux kernel. As of this writing, these
interfaces are used only by the Mellanox mlx5 driver to pass
diagnostic data to a Hyper-V host running in the Azure public
cloud. The functions hyperv_read_cfg_blk() and
hyperv_write_cfg_blk() are implemented in a separate module
(pci-hyperv-intf.c, under CONFIG_PCI_HYPERV_INTERFACE) that
effectively stubs them out when running in non-Hyper-V
environments.
요약·해설
vpci.rst:1-316Hyper-V vPCI는 DDA 또는 SR-IOV physical PCI device를 VM address space에 직접 연결하면서, device 도입과 제거에는 VMBus를 사용하고 정상 I/O에는 일반 PCI driver가 hardware와 직접 통신하게 합니다. 이 이중 신원 덕분에 dynamic offer를 Linux PCI topology로 바꿀 수 있습니다.
설정 때는 instance GUID 기반 PCI domain, config·BAR MMIO, root bus를 만든 뒤 generic PCI scan으로 넘깁니다. 제거는 Eject, PCI remove, Ejection Complete, VMBus rescind 순서이며 60초 timeout이나 설정 중 ejection 때문에 synchronization이 특히 중요합니다.
MSI routing은 VMBus reply를 기다려야 하지만 IRQ lock context에서는 sleep할 수 없어 polling하며 eject도 동시에 감지합니다. x86은 hypercall로 vCPU와 vector를 알리고 arm64는 emulated GICD에 SPI를 기록하며 LPI는 지원하지 않습니다.
DMA는 guest에 virtual IOMMU가 없는 direct mode지만 host physical IOMMU가 VM 경계를 보호합니다. CoCo VM의 PCI config access는 encrypted instruction을 host가 decode할 수 없어 implicit trap 대신 explicit hypercall argument를 사용합니다.