FPGA Device Feature List (DFL) Framework Overview

DFL framework의 목적

Device Feature List(DFL) FPGA framework와 그 규약을 따르는 driver는 하위 hardware 세부를 감추고 사용자 공간에 통합 interface를 제공한다. Application은 device memory에 DFL을 구현한 platform에서 이 interface로 FPGA accelerator를 구성하고, 열거하고, 열고, 접근할 수 있다.

DFL framework는 accelerator 접근뿐 아니라 FPGA reconfiguration 같은 system-level management 기능도 제공한다. 따라서 하나의 고정 accelerator API가 아니라 hardware feature 발견, management unit, programmable region, 사용자 공간 access를 연결하는 공통 기반으로 이해해야 한다.

문서 저자는 Enno Luebbers, Xiao Guangrong, Wu Hao, Xu Yilun이며 각 이름과 email은 원문에 보존되어 있다.

=================================================
FPGA Device Feature List (DFL) Framework Overview
=================================================

Authors:

- Enno Luebbers <[email protected]>
- Xiao Guangrong <[email protected]>
- Wu Hao <[email protected]>
- Xu Yilun <[email protected]>

The Device Feature List (DFL) FPGA framework (and drivers according to
this framework) hides the very details of low layer hardware and provides
unified interfaces to userspace. Applications could use these interfaces to
configure, enumerate, open and access FPGA accelerators on platforms which
implement the DFL in the device memory. Besides this, the DFL framework
enables system level management functions such as FPGA reconfiguration.

Device Feature List 구조

DFL은 device MMIO space 안의 feature header를 linked list로 연결해 새 기능을 확장 가능하게 추가하는 형식이다. Software는 미리 정의된 구조를 순회하여 FPGA Interface Unit(FIU), Accelerated Function Unit(AFU), Private Feature를 열거한다.

FIU는 FPGA와의 interface를 제공하는 독립 functional unit이며 대표적으로 FPGA Management Engine(FME)과 Port가 있다. AFU는 FPGA programmable region을 나타내며 항상 Port 같은 FIU의 child로 연결된다. Private Feature는 FIU 또는 AFU 내부의 sub-feature로, 서로 다른 ID를 가진 function block일 수 있다.

같은 FIU나 AFU에 속한 Private Feature는 `Next_DFH` pointer를 통해 하나의 list로 연결되어야 한다. FIU와 AFU의 functional register 집합은 Header Register Set, Private Feature의 register 집합은 Feature Register Set이라 부른다. 예로 FME Header Register Set과 FME Partial Reconfiguration Feature Register Set이 있다.

원문의 ASCII 도식은 첫 FIU header에서 `Next_DFH`로 여러 Private Feature를 순회하고, 별도의 `Next_AFU`가 AFU header를 가리키는 관계를 보여 준다.

Device Feature List (DFL) Overview
==================================
Device Feature List (DFL) defines a linked list of feature headers within the
device MMIO space to provide an extensible way of adding features. Software can
walk through these predefined data structures to enumerate FPGA features:
FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
as illustrated below::

    Header            Header            Header            Header
 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
 |    ID    |      +----------+      +----------+      +----------+
 +----------+      |    ID    |      |    ID    |      |    ID    |
 | Next_AFU |--+   +----------+      +----------+      +----------+
 +----------+  |   | Feature  |      | Feature  |      | Feature  |
 |  Header  |  |   | Register |      | Register |      | Register |
 | Register |  |   |   Set    |      |   Set    |      |   Set    |
 |   Set    |  |   +----------+      +----------+      +----------+
 +----------+  |      Header
               +-->+----------+
                   |   Type   |
                   |   AFU    |
                   +----------+
                   | Next_DFH |--> NULL
                   +----------+
                   |   GUID   |
                   +----------+
                   |  Header  |
                   | Register |
                   |   Set    |
                   +----------+

FPGA Interface Unit (FIU) represents a standalone functional unit for the
interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
descriptions on FME and Port in later sections).

Accelerated Function Unit (AFU) represents an FPGA programmable region and
always connects to a FIU (e.g. a Port) as its child as illustrated above.

Private Features represent sub features of the FIU and AFU. They could be
various function blocks with different IDs, but all private features which
belong to the same FIU or AFU, must be linked to one list via the Next Device
Feature Header (Next_DFH) pointer.

Each FIU, AFU and Private Feature could implement its own functional registers.
The functional register set for FIU and AFU, is named as Header Register Set,
e.g. FME Header Register Set, and the one for Private Feature, is named as
Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.

This Device Feature List provides a way of linking features together, it's
convenient for software to locate each feature by walking through this list,
and can be implemented in register regions of any FPGA device.

Device Feature Header Version 0

DFHv0는 원래 Device Feature Header 형식이며 모든 multi-byte quantity는 little-endian이다. Offset `0x00`의 64-bit word에는 `Type`, `DFH VER`, reserved field, `EOL`, `Next`, `REV`, `ID`가 배치된다.

`Type`은 FME·AFU·Private Feature를 구분하고 `DFH VER`는 header version이다. `EOL`은 현재 DFH가 DFL 끝임을 나타낸다. `Next`는 현재 DFH 시작점에서 다음 DFH까지의 byte offset이며 DFH 시작 주소는 8-byte boundary에 정렬되어야 한다. EOL이면 `Next`는 마지막 feature의 MMIO 크기다.

`REV`는 이 header와 연관된 feature revision이고 `ID`는 Type이 Private Feature일 때의 feature ID다. Offset `0x08`의 `GUID_L`과 `0x10`의 `GUID_H`는 Type이 FME 또는 AFU일 때만 존재하며 128-bit GUID의 하위·상위 64 bit를 담는다.

Device Feature Header - Version 0
=================================
Version 0 (DFHv0) is the original version of the Device Feature Header.
All multi-byte quantities in DFHv0 are little-endian.
The format of DFHv0 is shown below::

    +-----------------------------------------------------------------------+
    |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00
    +-----------------------------------------------------------------------+
    |63                                 GUID_L                             0| 0x08
    +-----------------------------------------------------------------------+
    |63                                 GUID_H                             0| 0x10
    +-----------------------------------------------------------------------+

- Offset 0x00

  * Type - The type of DFH (e.g. FME, AFU, or private feature).
  * DFH VER - The version of the DFH.
  * Rsvd - Currently unused.
  * EOL - Set if the DFH is the end of the Device Feature List (DFL).
  * Next - The offset in bytes of the next DFH in the DFL from the DFH start,
    and the start of a DFH must be aligned to an 8 byte boundary.
    If EOL is set, Next is the size of MMIO of the last feature in the list.
  * REV - The revision of the feature associated with this header.
  * ID - The feature ID if Type is private feature.

- Offset 0x08

  * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier
    (present only if Type is FME or AFU).

- Offset 0x10

  * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier
    (present only if Type is FME or AFU).

Device Feature Header Version 1

DFHv1은 feature가 parameter와 capability를 software에 설명하는 표준 mechanism을 추가하고, 모든 type에서 GUID 사용을 표준화하며, DFH 위치와 feature register space 위치를 분리한다. 모든 multi-byte quantity는 DFHv0와 마찬가지로 little-endian이다.

Offset `0x00`, `0x08`, `0x10`의 기본 field는 DFHv0와 같지만 GUID는 모든 DFHv1 type에 적용된다. Offset `0x18`은 `Reg Address/Offset`과 `Rel` bit를 가진다. `Rel`이 설정되면 값은 feature register의 16-bit aligned absolute address 상위 63 bit이고, 그렇지 않으면 DFH 시작점 기준 offset이다.

Offset `0x20`에는 register set byte 크기인 `Reg Size`, parameter block list 존재 여부인 `Params`, feature group ID인 `Group`, group 안의 feature instance ID인 `Instance`가 있다.

Parameter가 있으면 offset `0x28`부터 block을 읽는다. `Next`는 다음 parameter block까지의 8-byte word offset이며 `EOP`가 설정되면 마지막 parameter 크기다. `Param Version`과 `Param ID`가 data format을 식별하고, offset `0x30` 이후 `Parameter Data`의 크기와 형식은 이 version과 ID로 정의된다.

Device Feature Header - Version 1
=================================
Version 1 (DFHv1) of the Device Feature Header adds the following functionality:

* Provides a standardized mechanism for features to describe
  parameters/capabilities to software.
* Standardize the use of a GUID for all DFHv1 types.
* Decouples the DFH location from the register space of the feature itself.

All multi-byte quantities in DFHv1 are little-endian.
The format of Version 1 of the Device Feature Header (DFH) is shown below::

    +-----------------------------------------------------------------------+
    |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00
    +-----------------------------------------------------------------------+
    |63                                 GUID_L                             0| 0x08
    +-----------------------------------------------------------------------+
    |63                                 GUID_H                             0| 0x10
    +-----------------------------------------------------------------------+
    |63                   Reg Address/Offset                      1|  Rel  0| 0x18
    +-----------------------------------------------------------------------+
    |63        Reg Size       32|Params 31|30 Group    16|15 Instance      0| 0x20
    +-----------------------------------------------------------------------+
    |63 Next    35|34RSV33|EOP32|31 Param Version 16|15 Param ID           0| 0x28
    +-----------------------------------------------------------------------+
    |63                 Parameter Data                                     0| 0x30
    +-----------------------------------------------------------------------+

                                  ...

    +-----------------------------------------------------------------------+
    |63 Next    35|34RSV33|EOP32|31 Param Version 16|15 Param ID           0|
    +-----------------------------------------------------------------------+
    |63                 Parameter Data                                     0|
    +-----------------------------------------------------------------------+

- Offset 0x00

  * Type - The type of DFH (e.g. FME, AFU, or private feature).
  * DFH VER - The version of the DFH.
  * Rsvd - Currently unused.
  * EOL - Set if the DFH is the end of the Device Feature List (DFL).
  * Next - The offset in bytes of the next DFH in the DFL from the DFH start,
    and the start of a DFH must be aligned to an 8 byte boundary.
    If EOL is set, Next is the size of MMIO of the last feature in the list.
  * REV - The revision of the feature associated with this header.
  * ID - The feature ID if Type is private feature.

- Offset 0x08

  * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier.

- Offset 0x10

  * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier.

- Offset 0x18

  * Reg Address/Offset - If Rel bit is set, then the value is the high 63 bits
    of a 16-bit aligned absolute address of the feature's registers. Otherwise
    the value is the offset from the start of the DFH of the feature's registers.

- Offset 0x20

  * Reg Size - Size of feature's register set in bytes.
  * Params - Set if DFH has a list of parameter blocks.
  * Group - Id of group if feature is part of a group.
  * Instance - Id of feature instance within a group.

- Offset 0x28 if feature has parameters

  * Next - Offset to the next parameter block in 8 byte words. If EOP set,
    size in 8 byte words of last parameter.
  * Param Version - Version of Param ID.
  * Param ID - ID of parameter.

- Offset 0x30

  * Parameter Data - Parameter data whose size and format is defined by
    version and ID of the parameter.

FIU FME 관리 기능

FPGA Management Engine(FME)은 reconfiguration과 기타 infrastructure 기능을 수행하며 FPGA device마다 하나만 존재한다. 사용자 공간 application은 `open()`으로 FME에 exclusive access를 얻고 `close()`로 해제한다.

Ioctl은 API version 조회, extension 확인, bitstream programming, Port의 PF assign·release, FME global error interrupt 개수 조회와 trigger 설정을 제공한다. Symbol은 각각 `DFL_FPGA_GET_API_VERSION`, `DFL_FPGA_CHECK_EXTENSION`, `DFL_FPGA_FME_PORT_PR`, `DFL_FPGA_FME_PORT_ASSIGN`, `DFL_FPGA_FME_PORT_RELEASE`, `DFL_FPGA_FME_ERR_GET_IRQ_NUM`, `DFL_FPGA_FME_ERR_SET_IRQ`다.

Sysfs 경로 `/sys/class/fpga_region/regionX/dfl-fme.n/`에서는 `bitstream_id`, `bitstream_metadata`, `ports_num`, `errors/`를 제공한다. `dfl_fme_power`와 `dfl_fme_thermal` hwmon interface는 전력·온도·threshold 정보를 노출하며, performance counter는 perf PMU API로 제공한다.

FIU - FME (FPGA Management Engine)
==================================
The FPGA Management Engine performs reconfiguration and other infrastructure
functions. Each FPGA device only has one FME.

User-space applications can acquire exclusive access to the FME using open(),
and release it using close().

The following functions are exposed through ioctls:

- Get driver API version (DFL_FPGA_GET_API_VERSION)
- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
- Program bitstream (DFL_FPGA_FME_PORT_PR)
- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
- Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM)
- Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ)

More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):

 Read bitstream ID (bitstream_id)
     bitstream_id indicates version of the static FPGA region.

 Read bitstream metadata (bitstream_metadata)
     bitstream_metadata includes detailed information of static FPGA region,
     e.g. synthesis date and seed.

 Read number of ports (ports_num)
     one FPGA device may have more than one port, this sysfs interface indicates
     how many ports the FPGA device has.

 Global error reporting management (errors/)
     error reporting sysfs interfaces allow user to read errors detected by the
     hardware, and clear the logged errors.

 Power management (dfl_fme_power hwmon)
     power management hwmon sysfs interfaces allow user to read power management
     information (power consumption, thresholds, threshold status, limits, etc.)
     and configure power thresholds for different throttling levels.

 Thermal management (dfl_fme_thermal hwmon)
     thermal management hwmon sysfs interfaces allow user to read thermal
     management information (current temperature, thresholds, threshold status,
     etc.).

 Performance reporting
     performance counters are exposed through perf PMU APIs. Standard perf tool
     can be used to monitor all available perf events. Please see performance
     counter section below for more detailed information.

FIU Port

Port는 static FPGA fabric과 AFU를 포함한 partially reconfigurable region 사이의 interface다. Software에서 accelerator로 향하는 communication을 제어하며 reset과 debug 같은 기능을 노출한다.

하나의 FPGA device에는 Port가 여러 개 있을 수 있지만 Port 하나에는 항상 AFU 하나만 연결된다. 이 일대일 관계는 SRIOV 격리와 sysfs enumeration에서도 유지되는 핵심 단위다.

FIU - PORT
==========
A port represents the interface between the static FPGA fabric and a partially
reconfigurable region containing an AFU. It controls the communication from SW
to the accelerator and exposes features such as reset and debug. Each FPGA
device may have more than one port, but always one AFU per port.

AFU 사용자 공간 interface

AFU는 Port FIU에 연결되고 accelerator-specific control register에 사용할 고정 길이 MMIO region을 노출한다. 사용자 공간 application은 Port device node를 `open()`해 해당 AFU의 exclusive access를 얻고 `close()`로 해제한다. Accelerator MMIO region은 `mmap()`할 수도 있다.

Port ioctl은 API version과 extension 확인, Port 정보·MMIO region 정보 조회, DMA buffer map·unmap, AFU reset, Port error와 UINT interrupt 수 조회 및 trigger 설정을 제공한다.

`DFL_FPGA_PORT_RESET`은 FPGA Port와 AFU를 reset한다. DMA나 Partial Reconfiguration 중에도 언제든 실행할 수 있지만 system-level 문제를 일으켜서는 안 된다. DMA 또는 PR operation failure 같은 functional failure는 가능하되 복구 가능해야 한다.

Sysfs `/sys/class/fpga_region/<regionX>/<dfl-port.m>/`의 `afu_id`는 어떤 PR bitstream이 AFU에 programming되었는지 나타내고, `errors/`는 Port/AFU error를 읽고 clear하게 한다.

AFU
===
An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
used for accelerator-specific control registers.

User-space applications can acquire exclusive access to an AFU attached to a
port by using open() on the port device node and release it using close().

The following functions are exposed through ioctls:

- Get driver API version (DFL_FPGA_GET_API_VERSION)
- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
- Get port info (DFL_FPGA_PORT_GET_INFO)
- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
- Reset AFU (DFL_FPGA_PORT_RESET)
- Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM)
- Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ)
- Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM)
- Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ)

DFL_FPGA_PORT_RESET:
  reset the FPGA Port and its AFU. Userspace can do Port
  reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
  never cause any system level issue, only functional failure (e.g. DMA or PR
  operation failure) and be recoverable from the failure.

User-space applications can also mmap() accelerator MMIO regions.

More functions are exposed through sysfs:
(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):

 Read Accelerator GUID (afu_id)
     afu_id indicates which PR bitstream is programmed to this AFU.

 Error reporting (errors/)
     error reporting sysfs interfaces allow user to read port/afu errors
     detected by the hardware, and clear the logged errors.

Kernel DFL framework와 driver 계층

Kernel DFL framework는 container device인 FPGA base region을 만들고, 주어진 DFL에서 feature device와 Private Feature를 발견하며, FME·Port·AFU용 resource를 가진 platform device를 container 아래에 생성한다. Private Feature operation을 추상화해 feature device driver에 common operation도 제공한다.

FPGA DFL Device는 PCIe device나 platform device 등 서로 다른 hardware일 수 있다. System이 device를 만들면 해당 device driver module이 먼저 load되어 memory에서 DFL 위치를 찾고, resource와 함께 `drivers/fpga/dfl.c`의 common enumeration API에 넘긴다.

FME platform driver는 FME device 생성 뒤 자동으로 load된다. Static region version·metadata를 sysfs로 노출하고, PR sub-feature 초기화 때 FPGA manager·bridge·region을 생성한다. `DFL_FPGA_FME_PORT_PR` ioctl을 받으면 FPGA Region common interface를 호출해 대상 Port에 PR bitstream을 적용한다.

AFU driver는 AFU platform device가 만들어지면 probe되며 Port reset, AFU MMIO export, DMA buffer mapping을 사용자 공간에 제공한다. Feature platform device 생성 뒤에는 match되는 platform driver가 자동 load되어 각 기능을 처리한다.

DFL Framework Overview
======================

::

         +----------+    +--------+ +--------+ +--------+
         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
         |  Module  |    | Module | | Module | | Module |
         +----------+    +--------+ +--------+ +--------+
                 +-----------------------+
                 | FPGA Container Device |    Device Feature List
                 |  (FPGA Base Region)   |         Framework
                 +-----------------------+
  ------------------------------------------------------------------
               +----------------------------+
               |   FPGA DFL Device Module   |
               | (e.g. PCIE/Platform Device)|
               +----------------------------+
                 +------------------------+
                 |  FPGA Hardware Device  |
                 +------------------------+

DFL framework in kernel provides common interfaces to create container device
(FPGA base region), discover feature devices and their private features from the
given Device Feature Lists and create platform devices for feature devices
(e.g. FME, Port and AFU) with related resources under the container device. It
also abstracts operations for the private features and exposes common ops to
feature device drivers.

The FPGA DFL Device could be different hardware, e.g. PCIe device, platform
device and etc. Its driver module is always loaded first once the device is
created by the system. This driver plays an infrastructural role in the
driver architecture. It locates the DFLs in the device memory, handles them
and related resources to common interfaces from DFL framework for enumeration.
(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).

The FPGA Management Engine (FME) driver is a platform driver which is loaded
automatically after FME platform device creation from the DFL device module. It
provides the key features for FPGA management, including:

        a) Expose static FPGA region information, e.g. version and metadata.
           Users can read related information via sysfs interfaces exposed
           by FME driver.

        b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
           bridges and FPGA regions during PR sub feature initialization. Once
           it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
           common interface function from FPGA Region to complete the partial
           reconfiguration of the PR bitstream to the given port.

Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
probed once the AFU platform device is created. The main function of this module
is to provide an interface for userspace applications to access the individual
accelerators, including basic reset control on port, AFU MMIO region export, dma
buffer mapping service functions.

After feature platform devices creation, matched platform drivers will be loaded
automatically to handle different functionalities. Please refer to next sections
for detailed information on functional units which have been already implemented
under this DFL framework.

Partial Reconfiguration 호환성

Accelerator는 PR bitstream file을 사용한 partial reconfiguration으로 다시 구성할 수 있다. 이 file은 정확히 같은 static FPGA region과 대상 reconfigurable region인 Port를 위해 생성된 것이어야 한다.

호환되지 않는 bitstream을 사용하면 operation이 실패하고 system instability까지 일으킬 수 있다. 사용자 공간은 보통 reconfiguration ioctl을 호출하기 전에 PR bitstream header의 compatibility ID와 대상 FPGA region이 노출하는 `compat_id`를 비교해 검증한다.

Partial Reconfiguration
=======================
As mentioned above, accelerators can be reconfigured through partial
reconfiguration of a PR bitstream file. The PR bitstream file must have been
generated for the exact static FPGA region and targeted reconfigurable region
(port) of the FPGA, otherwise, the reconfiguration operation will fail and
possibly cause system instability. This compatibility can be checked by
comparing the compatibility ID noted in the header of PR bitstream file against
the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.

PCIe SRIOV virtualization

이 절은 DFL 기반 PCIe FPGA device의 SRIOV 지원을 사용해 VM application에서 accelerator에 접근하는 방법을 설명한다. FME는 항상 physical function(PF)을 통해 접근한다. Port와 AFU도 기본적으로 PF에 속하지만 PCIe SRIOV로 virtual function(VF)에 노출할 수 있다.

격리를 위해 VF 하나에는 Port 하나와 AFU 하나만 들어간다. 개별 VF를 VM에 assign하면 VM 안의 application이 같은 VF driver를 통해 AFU를 사용할 수 있다. FME는 VF에 assign할 수 없으므로 PR과 기타 management 기능은 PF에서만 가능하다.

FPGA PCIe driver는 PF나 VF를 발견하면 항상 먼저 load되어 DFL common interface로 양쪽 device를 enumerate하고 SRIOV를 지원한다. FME driver는 Port를 PF에서 release하거나 다시 PF에 assign하는 ioctl을 제공한다. PF에서 release한 뒤에만 SRIOV sysfs로 해당 Port를 VF에 안전하게 노출할 수 있다.

절차는 PF가 기본 소유한 Port를 FME device의 `DFL_FPGA_FME_PORT_RELEASE`로 release하고, N개 Port가 release되면 `echo N > $PCI_DEVICE_PATH/sriov_numvfs`로 VF를 enable한 다음 VM에 pass-through하는 순서다.

FPGA virtualization - PCIe SRIOV
================================
This section describes the virtualization support on DFL based FPGA device to
enable accessing an accelerator from applications running in a virtual machine
(VM). This section only describes the PCIe based FPGA device with SRIOV support.

Features supported by the particular FPGA device are exposed through Device
Feature Lists, as illustrated below:

::

    +-------------------------------+  +-------------+
    |              PF               |  |     VF      |
    +-------------------------------+  +-------------+
        ^            ^         ^              ^
        |            |         |              |
  +-----|------------|---------|--------------|-------+
  |     |            |         |              |       |
  |  +-----+     +-------+ +-------+      +-------+   |
  |  | FME |     | Port0 | | Port1 |      | Port2 |   |
  |  +-----+     +-------+ +-------+      +-------+   |
  |                  ^         ^              ^       |
  |                  |         |              |       |
  |              +-------+ +------+       +-------+   |
  |              |  AFU  | |  AFU |       |  AFU  |   |
  |              +-------+ +------+       +-------+   |
  |                                                   |
  |            DFL based FPGA PCIe Device             |
  +---------------------------------------------------+

FME is always accessed through the physical function (PF).

Ports (and related AFUs) are accessed via PF by default, but could be exposed
through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
created via PCIe SRIOV interface, to virtual machines.

The driver organization in virtualization case is illustrated below:
::

    +-------++------++------+             |
    | FME   || FME  || FME  |             |
    | FPGA  || FPGA || FPGA |             |
    |Manager||Bridge||Region|             |
    +-------++------++------+             |
    +-----------------------+  +--------+ |             +--------+
    |          FME          |  |  AFU   | |             |  AFU   |
    |         Module        |  | Module | |             | Module |
    +-----------------------+  +--------+ |             +--------+
          +-----------------------+       |       +-----------------------+
          | FPGA Container Device |       |       | FPGA Container Device |
          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
          +-----------------------+       |       +-----------------------+
            +------------------+          |         +------------------+
            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
            +------------------+   Host   | Machine +------------------+
   -------------------------------------- | ------------------------------
             +---------------+            |          +---------------+
             | PCI PF Device |            |          | PCI VF Device |
             +---------------+            |          +---------------+

FPGA PCIe device driver is always loaded first once an FPGA PCIe PF or VF device
is detected. It:

* Finishes enumeration on both FPGA PCIe PF and VF device using common
  interfaces from DFL framework.
* Supports SRIOV.

The FME device driver plays a management role in this driver architecture, it
provides ioctls to release Port from PF and assign Port to PF. After release
a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
sysfs interface.

To enable accessing an accelerator from applications running in a VM, the
respective AFU's port needs to be assigned to a VF using the following steps:

#. The PF owns all AFU ports by default. Any port that needs to be
   reassigned to a VF must first be released through the
   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.

#. Once N ports are released from PF, then user can use command below
   to enable SRIOV and VFs. Each VF owns only one Port with AFU.

   ::

      echo N > $PCI_DEVICE_PATH/sriov_numvfs

#. Pass through the VFs to VMs

#. The AFU under VF is accessible from applications in VM (using the
   same driver inside the VF).

Note that an FME can't be assigned to a VF, thus PR and other management
functions are only available via the PF.

Sysfs device enumeration

Application은 `/sys/class/fpga_region` 아래 hierarchy에서 FPGA device를 enumerate한다. 예제 host에는 DFL FPGA device 두 개가 있고 각 device는 FME 하나와 Port/AFU 두 개를 가진다. FPGA region은 `region0`, `region1`, `region2`처럼 생성된다.

각 `regionX` folder를 검색해 `dfl-port.n` 또는 `dfl-fme.m` feature device가 있으면 그 region이 FPGA device를 나타내는 base region이다. Child는 `region0/dfl-fme.0`, `region0/dfl-port.0`, `region0/dfl-port.1` 같은 형태다.

일반 형식은 `/sys/class/fpga_region/<regionX>/<dfl-fme.n>/`과 `/sys/class/fpga_region/<regionX>/<dfl-port.m>/`이다. `n`은 모든 FME를 연속 번호화하고 `m`은 모든 Port를 연속 번호화한다. `ioctl()` 또는 `mmap()` device node는 각 directory의 `dev` entry를 통해 참조한다.

Device enumeration
==================
This section introduces how applications enumerate the fpga device from
the sysfs hierarchy under /sys/class/fpga_region.

In the example below, two DFL based FPGA devices are installed in the host. Each
fpga device has one FME and two ports (AFUs).

FPGA regions are created under /sys/class/fpga_region/::

        /sys/class/fpga_region/region0
        /sys/class/fpga_region/region1
        /sys/class/fpga_region/region2
        ...

Application needs to search each regionX folder, if feature device is found,
(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
fpga region which represents the FPGA device.

Each base region has one FME and two ports (AFUs) as child devices::

        /sys/class/fpga_region/region0/dfl-fme.0
        /sys/class/fpga_region/region0/dfl-port.0
        /sys/class/fpga_region/region0/dfl-port.1
        ...

        /sys/class/fpga_region/region3/dfl-fme.1
        /sys/class/fpga_region/region3/dfl-port.2
        /sys/class/fpga_region/region3/dfl-port.3
        ...

In general, the FME/AFU sysfs interfaces are named as follows::

        /sys/class/fpga_region/<regionX>/<dfl-fme.n>/
        /sys/class/fpga_region/<regionX>/<dfl-port.m>/

with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
ports.

The device nodes used for ioctl() or mmap() can be referenced through::

        /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
        /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev

FME Performance Counters

Performance reporting은 FME에 구현된 Private Feature다. Hardware는 `basic`, `cache`, `fabric`, `vtd`, `vtd_sip` 같은 독립적인 system-wide counter set을 제공할 수 있다. 사용자는 표준 perf tool로 cache hit/miss, transaction 수, AFU interface clock 등 지원 event를 감시한다.

Hardware별 counter set은 다르므로 `perf list`로 대상 device의 event를 확인해야 한다. Driver는 perf PMU와 `/sys/bus/event_source/devices/dfl_fme*` 아래 sysfs interface를 만들어 event와 option을 설명한다.

`format` directory는 `struct perf_event_attr.config`의 세 bitfield를 정의한다. `evtype`은 event category, `event`는 category 안의 event ID, `portid`는 전체 FPGA와 특정 Port 중 감시 대상을 정한다. `events` directory는 perf에서 바로 쓸 configuration template를 제공한다.

예를 들어 `fab_mmio_read`는 `event=0x06,evtype=0x02,portid=0xff`이며 전체 fabric을 감시한다. `fab_port_mmio_read`는 특정 Port를 위해 `portid=?`를 명시해야 한다. Symbolic event, field expression, raw `config` 세 방식으로 같은 event를 지정할 수 있다.

Fabric overall event인 `fab_*`와 per-Port event인 `fab_port_*`는 hardware counter set 하나를 공유하므로 동시에 감시할 수 없다. `cpumask`에는 system-wide counter를 access할 CPU ID 하나만 들어가며 multi-CPU counting은 허용되지 않는다. 현재 driver는 sampling을 지원하지 않아 `perf record`도 지원하지 않는다.

Performance Counters
====================
Performance reporting is one private feature implemented in FME. It could
supports several independent, system-wide, device counter sets in hardware to
monitor and count for performance events, including "basic", "cache", "fabric",
"vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor
FPGA cache hit/miss rate, transaction number, interface clock counter of AFU
and other FPGA performance events.

Different FPGA devices may have different counter sets, depending on hardware
implementation. E.g., some discrete FPGA cards don't have any cache. User could
use "perf list" to check which perf events are supported by target hardware.

In order to allow user to use standard perf API to access these performance
counters, driver creates a perf PMU, and related sysfs interfaces in
/sys/bus/event_source/devices/dfl_fme* to describe available perf events and
configuration options.

The "format" directory describes the format of the config field of struct
perf_event_attr. There are 3 bitfields for config: "evtype" defines which type
the perf event belongs to; "event" is the identity of the event within its
category; "portid" is introduced to decide counters set to monitor on FPGA
overall data or a specific port.

The "events" directory describes the configuration templates for all available
events which can be used with perf tool directly. For example, fab_mmio_read
has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this
event belongs to fabric type (0x02), the local event id is 0x06 and it is for
overall monitoring (portid=0xff).

Example usage of perf::

  $# perf list |grep dfl_fme

  dfl_fme0/fab_mmio_read/                              [Kernel PMU event]
  <...>
  dfl_fme0/fab_port_mmio_read,portid=?/                [Kernel PMU event]
  <...>

  $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command>
  or
  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command>
  or
  $# perf stat -a -e dfl_fme0/config=0xff2006/ <command>

Another example, fab_port_mmio_read monitors mmio read of a specific port. So
its configuration template is "event=0x06,evtype=0x01,portid=?". The portid
should be explicitly set.

Its usage of perf::

  $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command>
  or
  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command>
  or
  $# perf stat -a -e dfl_fme0/config=0x2006/ <command>

Please note for fabric counters, overall perf events (fab_*) and port perf
events (fab_port_*) actually share one set of counters in hardware, so it can't
monitor both at the same time. If this set of counters is configured to monitor
overall data, then per port perf data is not supported. See below example::

  $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\
                                                    portid=0/ sleep 1

  Performance counter stats for 'system wide':

                 3      dfl_fme0/fab_mmio_read/
   <not supported>      dfl_fme0/fab_port_mmio_write,portid=0x0/

       1.001750904 seconds time elapsed

The driver also provides a "cpumask" sysfs attribute, which contains only one
CPU id used to access these perf events. Counting on multiple CPU is not allowed
since they are system-wide counters on FPGA device.

The current driver does not support sampling. So "perf record" is unsupported.

Interrupt 지원

일부 FME와 AFU Private Feature는 interrupt를 생성할 수 있다. 사용자는 `DFL_FPGA_*_GET_IRQ_NUM` ioctl로 해당 feature의 interrupt 지원 여부와 개수를 확인한다.

Driver는 eventfd 기반 interrupt handling을 구현한다. 사용자는 `DFL_FPGA_*_SET_IRQ` ioctl로 eventfd를 driver에 설정하고 `poll()` 또는 `select()`로 notification을 기다린다. 현재 DFL에서 interrupt를 지원하는 sub-feature는 Port error, FME global error, AFU interrupt 세 가지다.

Interrupt support
=================
Some FME and AFU private features are able to generate interrupts. As mentioned
above, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how
many interrupts are supported for this private feature. Drivers also implement
an eventfd based interrupt handling mechanism for users to get notified when
interrupt happens. Users could set eventfds to driver via
ioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for
notification.
In Current DFL, 3 sub features (Port error, FME global error and AFU interrupt)
support interrupts.

새 FIU 지원 추가

DFL framework 아래에 새 functional block인 FIU를 추가하면 기존 FME와 Port/AFU driver와 같은 방식의 새 platform device driver가 필요하다.

새 FIU type을 발견하고 관련 platform device를 생성해야 하므로 DFL framework enumeration code도 수정해야 한다. 새 FIU는 기존 Private Feature 추가와 달리 상위 feature device type 자체를 확장하기 때문이다.

Add new FIUs support
====================
It's possible that developers made some new function blocks (FIUs) under this
DFL framework, then new platform device driver needs to be developed for the
new feature dev (FIU) following the same way as existing feature dev drivers
(e.g. FME and Port/AFU platform device driver). Besides that, it requires
modification on DFL framework enumeration code too, for new FIU type detection
and related platform devices creation.

새 Private Feature 지원 추가

기존 FME 또는 Port에 새 Private Feature를 추가할 때는 DFL framework enumeration code를 수정할 필요가 없다. 각 Private Feature는 자동 parse되고 MMIO resource는 framework가 만든 FIU platform device 아래에서 찾을 수 있다.

Developer는 일치하는 feature ID를 가진 sub-feature driver만 제공하면 된다. `drivers/fpga/dfl-fme-pr.c`의 FME Partial Reconfiguration Sub Feature driver를 참고할 수 있다. 기존 feature ID table과 신규 ID 신청 안내는 `https://github.com/OPAE/dfl-feature-id`에 있다.

Add new private features support
================================
In some cases, we may need to add some new private features to existing FIUs
(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
framework, as each private feature will be parsed automatically and related
mmio resources can be found under FIU platform device created by DFL framework.
Developer only needs to provide a sub feature driver with matched feature id.
FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
could be a reference.

Please refer to below link to existing feature id table and guide for new feature
ids application.
https://github.com/OPAE/dfl-feature-id

PCI device에서 DFL 위치 찾기

원래 방식은 첫 DFL이 BAR 0의 offset 0에서 시작한다고 가정했다. 첫 node가 FME이면 추가 DFL 위치는 FME header register의 Port 정보로 지정한다. 더 유연한 방식으로 PCIe vendor-specific capability structure가 device의 모든 DFL 위치를 지정할 수 있다.

Intel은 이 용도로 VSEC ID `0x43`을 예약했다. Vendor-specific data는 DFL 개수를 담은 4-byte register 하나와 각 DFL의 위치를 담은 4-byte `Offset/BIR` register로 구성된다. `Offset/BIR`의 bit `2:0`은 BAR indicator이고 bit `31:3`은 8-byte aligned offset이며 하위 세 bit는 0이다.

한 BAR에 DFL 여러 개를 지정하는 방식도 검토했지만 가치가 없다고 판단했다. BAR 하나당 DFL 하나로 제한하면 구현이 단순해지고 추가 error checking이 가능하다.

Location of DFLs on a PCI Device
================================
The original method for finding a DFL on a PCI device assumed the start of the
first DFL to offset 0 of bar 0.  If the first node of the DFL is an FME,
then further DFLs in the port(s) are specified in FME header registers.
Alternatively, a PCIe vendor specific capability structure can be used to
specify the location of all the DFLs on the device, providing flexibility
for the type of starting node in the DFL.  Intel has reserved the
VSEC ID of 0x43 for this purpose.  The vendor specific
data begins with a 4 byte vendor specific register for the number of DFLs followed 4 byte
Offset/BIR vendor specific registers for each DFL. Bits 2:0 of Offset/BIR register
indicates the BAR, and bits 31:3 form the 8 byte aligned offset where bits 2:0 are
zero.
::

        +----------------------------+
        |31     Number of DFLS      0|
        +----------------------------+
        |31     Offset     3|2 BIR  0|
        +----------------------------+
                      . . .
        +----------------------------+
        |31     Offset     3|2 BIR  0|
        +----------------------------+

Being able to specify more than one DFL per BAR has been considered, but it
was determined the use case did not provide value.  Specifying a single DFL
per BAR simplifies the implementation and allows for extra error checking.

DFL device용 userspace driver

FPGA는 새 hardware component로 reprogram할 수 있으므로 새 bitstream이 새 Private Feature와 DFL device를 system에 제시할 수 있다. Diagnostic test, kernel driver의 사용자 공간 prototype, 표준 kernel subsystem에 맞지 않는 전용 hardware에는 userspace driver가 필요할 수 있다.

이 용도는 사용자 공간의 직접 MMIO access와 interrupt handling을 요구한다. `uio_dfl` module은 이를 위한 UIO device interface를 노출한다. 현재 driver는 hardware IRQ가 없는 Ether Group sub-feature만 지원하므로 interrupt handling은 아직 추가되지 않았다.

`UIO_DFL`을 선택하면 `uio_dfl` module driver를 enable한다. 새 DFL feature를 UIO direct access로 지원하려면 feature ID를 driver의 `id_table`에 추가해야 한다.

Userspace driver support for DFL devices
========================================
The purpose of an FPGA is to be reprogrammed with newly developed hardware
components. New hardware can instantiate a new private feature in the DFL, and
then present a DFL device in the system. In some cases users may need a
userspace driver for the DFL device:

* Users may need to run some diagnostic test for their hardware.
* Users may prototype the kernel driver in user space.
* Some hardware is designed for specific purposes and does not fit into one of
  the standard kernel subsystems.

This requires direct access to MMIO space and interrupt handling from
userspace. The uio_dfl module exposes the UIO device interfaces for this
purpose.

Currently the uio_dfl driver only supports the Ether Group sub feature, which
has no irq in hardware. So the interrupt handling is not added in this driver.

UIO_DFL should be selected to enable the uio_dfl module driver. To support a
new DFL feature via UIO direct access, its feature id should be added to the
driver's id_table.

Reconfiguration interface의 향후 과제

현재 FME driver는 partial reconfiguration을 위해 `DFL_FPGA_FME_PORT_PR` ioctl 하나를 사용자 공간에 제공한다.

향후 통합 reconfiguration user interface가 추가되면 FME driver는 전용 ioctl interface에서 그 공통 interface로 전환해야 한다는 것이 문서의 open discussion이다.

Open discussion
===============
FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
to user now. In the future, if unified user interfaces for reconfiguration are
added, FME driver should switch to them from ioctl interface.