Network Function Representor

SmartNIC 내부 switch를 Linux stack으로 제어

이 문서는 SmartNIC 내부 switching을 제어하는 representor netdevice의 의미와 사용법을 설명합니다. Physical multi-port switch의 밀접한 개념인 port representor는 `Documentation/networking/switchdev.rst`를 참고합니다.

2010년대 중반부터 NIC virtualization은 단순 MAC/VLAN switching에 기반한 기존 SR-IOV가 표현할 수 있는 범위를 넘어섰습니다. OpenVSwitch 같은 software-defined network가 각 function의 connectivity를 지정하도록 그 switching을 NIC에 offload하려는 요구가 생겼고, 이런 device를 SmartNIC 또는 DPU라고 부릅니다.

Network function representor는 표준 Linux networking stack을 virtual switch와 IOV device에 연결합니다. Linux가 제어하는 physical switch의 port마다 netdev가 있듯, virtual switch의 virtual port마다 별도 representor netdev를 둡니다.

Boot 직후 offload rule이 하나도 없을 때 VF에서 나온 모든 packet은 representor를 통해 PF networking stack에 나타납니다. 따라서 PF는 언제나 VF와 통신할 수 있고, representor·uplink·다른 netdev 사이에 routing, bridge, TC classifier 같은 표준 Linux forwarding을 설정할 수 있습니다.

Representor는 administrative command에서 function을 나타내는 control-plane object이면서 virtual pipe 한쪽 끝인 data-plane object입니다. 일반 netdevice처럼 설정할 수 있고 link state처럼 representee가 representor 설정을 따르는 속성도 있지만, 일부 속성은 별도 API로 representee를 구성합니다.

.. SPDX-License-Identifier: GPL-2.0
.. _representors:

=============================
Network Function Representors
=============================

This document describes the semantics and usage of representor netdevices, as
used to control internal switching on SmartNICs.  For the closely-related port
representors on physical (multi-port) switches, see
:ref:`Documentation/networking/switchdev.rst <switchdev>`.

Motivation
----------

Since the mid-2010s, network cards have started offering more complex
virtualisation capabilities than the legacy SR-IOV approach (with its simple
MAC/VLAN-based switching model) can support.  This led to a desire to offload
software-defined networks (such as OpenVSwitch) to these NICs to specify the
network connectivity of each function.  The resulting designs are variously
called SmartNICs or DPUs.

Network function representors bring the standard Linux networking stack to
virtual switches and IOV devices.  Just as each physical port of a Linux-
controlled switch has a separate netdev, so does each virtual port of a virtual
switch.
When the system boots, and before any offload is configured, all packets from
the virtual functions appear in the networking stack of the PF via the
representors.  The PF can thus always communicate freely with the virtual
functions.
The PF can configure standard Linux forwarding between representors, the uplink
or any other netdev (routing, bridging, TC classifiers).

Thus, a representor is both a control plane object (representing the function in
administrative commands) and a data plane object (one end of a virtual pipe).
As a virtual link endpoint, the representor can be configured like any other
netdevice; in some cases (e.g. link state) the representee will follow the
representor's configuration, while in others there are separate APIs to
configure the representee.

Switchdev function·representee와 세 역할

`switchdev function`은 device의 virtual switch를 administrative하게 제어하는 PCIe function입니다. 보통 PF지만 NIC 설정에 따라 VF나 SF(subfunction)에 권한을 줄 수도 있습니다. Multi-port NIC는 device 전체에 하나를 두거나 physical port마다 별도 virtual switch와 switchdev function을 둘 수 있습니다.

Nested switching을 지원하면 nested switch마다 switchdev function이 있을 수 있습니다. 각 function은 자신이 직접 관리하는 해당 sub-switch port의 representor만 만들어야 합니다. `representee`는 representor가 대신 나타내는 object이며 VF representor의 representee는 해당 VF입니다.

첫째, representor는 representee가 보는 network connection을 구성합니다. Link up/down과 MTU 등이 이에 해당하며 representor를 administratively UP으로 만들면 representee에 link-up 또는 carrier-on event가 보여야 합니다.

둘째, virtual switch의 offloaded fast-path rule과 일치하지 않는 traffic이 통과하는 slow path입니다. Representor로 transmit한 packet은 representee에 전달되고, representee가 보내지만 switching rule에 맞지 않는 packet은 representor netdevice에서 receive됩니다. 이 virtual pipe는 개념적으로 veth pair와 비슷하며 OVS나 Linux bridge가 representee 사이 또는 외부 network로 packet을 software forwarding할 수 있게 합니다.

셋째, TC filter 같은 switching rule이 representee를 참조할 수 있는 handle입니다. 이 rule은 hardware로 offload할 수 있습니다. Software TC에서 VF representor ingress에 적용한 filter는 hardware에서는 VF가 transmit한 packet에 적용되고, VF representor로 향하는 mirred egress redirect는 hardware에서 VF에 직접 전달하는 동작이 됩니다.

따라서 성능을 제외한 결과는 TC filter의 offload 여부와 무관하게 같아야 합니다.

Definitions
-----------

This document uses the term "switchdev function" to refer to the PCIe function
which has administrative control over the virtual switch on the device.
Typically, this will be a PF, but conceivably a NIC could be configured to grant
these administrative privileges instead to a VF or SF (subfunction).
Depending on NIC design, a multi-port NIC might have a single switchdev function
for the whole device or might have a separate virtual switch, and hence
switchdev function, for each physical network port.
If the NIC supports nested switching, there might be separate switchdev
functions for each nested switch, in which case each switchdev function should
only create representors for the ports on the (sub-)switch it directly
administers.

A "representee" is the object that a representor represents.  So for example in
the case of a VF representor, the representee is the corresponding VF.

What does a representor do?
---------------------------

A representor has three main roles.

1. It is used to configure the network connection the representee sees, e.g.
   link up/down, MTU, etc.  For instance, bringing the representor
   administratively UP should cause the representee to see a link up / carrier
   on event.
2. It provides the slow path for traffic which does not hit any offloaded
   fast-path rules in the virtual switch.  Packets transmitted on the
   representor netdevice should be delivered to the representee; packets
   transmitted by the representee which fail to match any switching rule should
   be received on the representor netdevice.  (That is, there is a virtual pipe
   connecting the representor to the representee, similar in concept to a veth
   pair.)
   This allows software switch implementations (such as OpenVSwitch or a Linux
   bridge) to forward packets between representees and the rest of the network.
3. It acts as a handle by which switching rules (such as TC filters) can refer
   to the representee, allowing these rules to be offloaded.

The combination of 2) and 3) means that the behaviour (apart from performance)
should be the same whether a TC filter is offloaded or not.  E.g. a TC rule
on a VF representor applies in software to packets received on that representor
netdevice, while in hardware offload it would apply to packets transmitted by
the representee VF.  Conversely, a mirred egress redirect to a VF representor
corresponds in hardware to delivery directly to the representee VF.

어떤 function과 virtual port에 필요한가

원칙적으로 device internal switch의 virtual port마다 representor가 있어야 합니다. 일부 vendor는 uplink와 physical network port representor를 생략해 uplink netdev 자체를 physical port representor처럼 사용하지만, 이 방식은 port나 uplink가 여러 개인 device로 일반화하기 어렵습니다.

Representor 대상에는 switchdev function 소속 VF, local PCIe controller의 다른 PF와 그 VF, SmartNIC embedded SoC 같은 external PCIe controller의 PF·VF가 포함됩니다. Network access가 virtual switch port를 거치는 경우에만 vDPA virtio-blk PF 같은 다른 personality의 function, 자체 switch port가 있는 SF, PCIe PF/VF가 없는 accelerator나 plugin에도 representor가 필요합니다. Representee에 netdev가 없어도 virtual port가 있으면 representor가 필요할 수 있습니다.

중요한 구분은 representor가 PCIe function이나 end-user netdev가 아니라 virtual port를 나타낸다는 점입니다. Simple device에서는 VF netdevice와 VF representor가 1:1일 수 있지만 복잡한 configuration에서는 그렇지 않습니다. Internal switch를 직간접적으로도 사용하지 않는 PCIe function은 netdev가 있어도 representor를 만들면 안 됩니다.

Footnote의 network block device 예에서는 device 내부 hardware IP stack이 block DMA request를 network packet으로 변환하고 그 packet만 virtual port를 지나면 TC rule로 VLAN이나 VxLAN 정책을 적용할 수 있으므로 representor가 적절합니다. Block device 자체의 설정은 networking entity가 아니므로 devlink 같은 다른 channel을 사용합니다.

반대로 virtio-blk가 DMA request를 변경 없이 다른 PF로 넘기고 그 PF driver가 software에서 IP traffic을 시작·종료한다면 DMA traffic은 virtual switch를 지나지 않습니다. 이 virtio-blk PF에는 representor가 없어야 합니다.

What functions should have a representor?
-----------------------------------------

Essentially, for each virtual port on the device's internal switch, there
should be a representor.
Some vendors have chosen to omit representors for the uplink and the physical
network port, which can simplify usage (the uplink netdev becomes in effect the
physical port's representor) but does not generalise to devices with multiple
ports or uplinks.

Thus, the following should all have representors:

 - VFs belonging to the switchdev function.
 - Other PFs on the local PCIe controller, and any VFs belonging to them.
 - PFs and VFs on external PCIe controllers on the device (e.g. for any embedded
   System-on-Chip within the SmartNIC).
 - PFs and VFs with other personalities, including network block devices (such
   as a vDPA virtio-blk PF backed by remote/distributed storage), if (and only
   if) their network access is implemented through a virtual switch port. [#]_
   Note that such functions can require a representor despite the representee
   not having a netdev.
 - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have
   their own port on the switch (as opposed to using their parent PF's port).
 - Any accelerators or plugins on the device whose interface to the network is
   through a virtual switch port, even if they do not have a corresponding PCIe
   PF or VF.

This allows the entire switching behaviour of the NIC to be controlled through
representor TC rules.

It is a common misunderstanding to conflate virtual ports with PCIe virtual
functions or their netdevs.  While in simple cases there will be a 1:1
correspondence between VF netdevices and VF representors, more advanced device
configurations may not follow this.
A PCIe function which does not have network access through the internal switch
(not even indirectly through the hardware implementation of whatever services
the function provides) should *not* have a representor (even if it has a
netdev).
Such a function has no switch virtual port for the representor to configure or
to be the other end of the virtual pipe.
The representor represents the virtual port, not the PCIe function nor the 'end
user' netdevice.

.. [#] The concept here is that a hardware IP stack in the device performs the
   translation between block DMA requests and network packets, so that only
   network packets pass through the virtual port onto the switch.  The network
   access that the IP stack "sees" would then be configurable through tc rules;
   e.g. its traffic might all be wrapped in a specific VLAN or VxLAN.  However,
   any needed configuration of the block device *qua* block device, not being a
   networking entity, would not be appropriate for the representor and would
   thus use some other channel such as devlink.
   Contrast this with the case of a virtio-blk implementation which forwards the
   DMA requests unchanged to another PF whose driver then initiates and
   terminates IP traffic in software; in that case the DMA traffic would *not*
   run over the virtual switch and the virtio-blk PF should thus *not* have a
   representor.

Representor 생성과 devlink port 식별

Switchdev function에 붙은 driver instance는 switch의 각 virtual port마다 pure-software netdevice를 만듭니다. 이 netdevice는 switchdev function 자체 netdevice 또는 `netdev_priv()`의 driver-private data를 참조해야 합니다. Probe 시 port를 enumerate하거나 runtime port 생성·삭제에 동적으로 반응하거나 두 방식을 함께 사용할 수 있습니다.

Representor operation은 대체로 switchdev function을 통해 실제 동작합니다. 예를 들어 `ndo_start_xmit()`은 switchdev function에 연결된 hardware TX queue로 packet을 보내되 packet metadata나 queue configuration으로 특정 representee에 전달할 packet임을 표시할 수 있습니다.

Representor netdevice는 representee나 switchdev function의 PCIe device를 `net_dev->dev.parent` 또는 `SET_NETDEV_DEV()`로 직접 참조하면 안 됩니다. 대신 netdevice를 등록하기 전에 `SET_NETDEV_DEVLINK_PORT` macro로 devlink port instance를 할당합니다. Kernel은 이 devlink port를 통해 `phys_switch_id`와 `phys_port_name` sysfs node를 제공합니다.

일부 legacy driver가 `ndo_get_port_parent_id()`와 `ndo_get_phys_port_name()`을 직접 구현하지만 이 방식은 deprecated입니다. 자세한 API는 devlink port 문서를 참고합니다.

Userspace는 sysfs 정보를 udev rule 등에서 사용해 알아보기 쉬운 이름이나 alias를 구성합니다. 예를 들어 switchdev function이 `eth4`이고 `phys_port_name`이 `p0pf1vf2`인 representor를 `eth4pf1vf2rep`으로 바꿀 수 있습니다. PCIe function에 대응하지 않는 accelerator·plugin representor에는 아직 정착한 naming convention이 없습니다.

How are representors created?
-----------------------------

The driver instance attached to the switchdev function should, for each virtual
port on the switch, create a pure-software netdevice which has some form of
in-kernel reference to the switchdev function's own netdevice or driver private
data (``netdev_priv()``).
This may be by enumerating ports at probe time, reacting dynamically to the
creation and destruction of ports at run time, or a combination of the two.

The operations of the representor netdevice will generally involve acting
through the switchdev function.  For example, ``ndo_start_xmit()`` might send
the packet through a hardware TX queue attached to the switchdev function, with
either packet metadata or queue configuration marking it for delivery to the
representee.

How are representors identified?
--------------------------------

The representor netdevice should *not* directly refer to a PCIe device (e.g.
through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the
representee or of the switchdev function.
Instead, the driver should use the ``SET_NETDEV_DEVLINK_PORT`` macro to
assign a devlink port instance to the netdevice before registering the
netdevice; the kernel uses the devlink port to provide the ``phys_switch_id``
and ``phys_port_name`` sysfs nodes.
(Some legacy drivers implement ``ndo_get_port_parent_id()`` and
``ndo_get_phys_port_name()`` directly, but this is deprecated.)  See
:ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>` for the
details of this API.

It is expected that userland will use this information (e.g. through udev rules)
to construct an appropriately informative name or alias for the netdevice.  For
instance if the switchdev function is ``eth4`` then a representor with a
``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``.

There are as yet no established conventions for naming representors which do not
correspond to PCIe functions (e.g. accelerators and plugins).

TC rule과 simple redirect·mirror

Representor에 설정한 TC rule은 software TC에서는 그 representor netdevice가 receive한 packet에 적용됩니다. Rule의 delivery 대상이 virtual switch의 다른 port라면 driver는 이를 hardware에 offload해 representee가 transmit한 packet에 적용할 수 있습니다.

마찬가지로 representor를 대상으로 하는 TC `mirred egress` action은 software에서는 representor를 통해 간접적으로 representee에 전달합니다. Hardware offload도 이를 representee로의 직접 delivery로 해석해야 합니다.

예제에서 `PORT_DEV`는 physical port representor이고 `REP_DEV`는 VF representor입니다. 첫 rule은 VF에서 온 모든 IPv4 packet을 physical port로 redirect합니다. 둘째 rule은 physical port에서 받은 IPv4 packet을 `PORT_DEV`에도 남기면서 VF로 mirror합니다.

둘째 rule에는 `skip_sw`가 필요합니다. 이를 빼면 `PORT_DEV`에서 software receive할 때 TC rule이 다시 실행되어 `REP_DEV`로 한 번 더 mirror하므로 VF가 packet 두 사본을 받을 수 있습니다. Port·uplink representor가 분리되지 않은 device에서는 `PORT_DEV` 대신 switchdev function의 uplink netdevice를 사용합니다.

NIC가 지원하면 virtual switch가 VLAN push/pop 같은 packet-modifying action도 수행할 수 있습니다.

How do representors interact with TC rules?
-------------------------------------------

Any TC rule on a representor applies (in software TC) to packets received by
that representor netdevice.  Thus, if the delivery part of the rule corresponds
to another port on the virtual switch, the driver may choose to offload it to
hardware, applying it to packets transmitted by the representee.

Similarly, since a TC mirred egress action targeting the representor would (in
software) send the packet through the representor (and thus indirectly deliver
it to the representee), hardware offload should interpret this as delivery to
the representee.

As a simple example, if ``PORT_DEV`` is the physical port representor and
``REP_DEV`` is a VF representor, the following rules::

    tc filter add dev $REP_DEV parent ffff: protocol ipv4 flower \
        action mirred egress redirect dev $PORT_DEV
    tc filter add dev $PORT_DEV parent ffff: protocol ipv4 flower skip_sw \
        action mirred egress mirror dev $REP_DEV

would mean that all IPv4 packets from the VF are sent out the physical port, and
all IPv4 packets received on the physical port are delivered to the VF in
addition to ``PORT_DEV``.  (Note that without ``skip_sw`` on the second rule,
the VF would get two copies, as the packet reception on ``PORT_DEV`` would
trigger the TC rule again and mirror the packet to ``REP_DEV``.)

On devices without separate port and uplink representors, ``PORT_DEV`` would
instead be the switchdev function's own uplink netdevice.

Of course the rules can (if supported by the NIC) include packet-modifying
actions (e.g. VLAN push/pop), which should be performed by the virtual switch.

VxLAN tunnel offload와 golden rule

Tunnel encapsulation과 decapsulation은 metadata mode tunnel netdevice라는 세 번째 netdevice가 필요해 더 복잡합니다. 예제는 `ip link add vxlan0 type vxlan external`로 만든 VxLAN device를 사용하고, underlay device인 switchdev uplink netdev 또는 port representor에 IP address를 bind해야 합니다.

VF egress rule은 `tunnel_key set`으로 VNI, local·remote IP와 UDP destination port 4789를 metadata에 넣고 `vxlan0`으로 redirect합니다. Driver는 `LOCAL_IP` route lookup이 `PORT_DEV`로 이어진다는 사실과 ARP/neighbour lookup 결과를 사용해 outer Ethernet MAC address까지 결정하고, VF packet을 VxLAN으로 encapsulate해 physical port로 보냅니다.

Reverse rule은 `vxlan0`에서 remote·local encapsulation address, VNI와 port를 match하고 `tunnel_key unset` 뒤 `REP_DEV`로 redirect합니다. Physical port에서 UDP 4789 packet을 받으면 VxLAN으로 parse하고 VSID가 `$VNI`와 일치할 때 decapsulate해 VF로 전달하는 hardware 동작에 대응합니다.

TC offload의 golden rule은 hardware가 slow path와 같은 최종 결과를 보장해야 한다는 것입니다. 즉 packet이 representor netdevice를 통해 software TC를 거친 결과와 동일해야 하며, `skip_hw` rule은 hardware에서 무시하고 `skip_sw` rule은 hardware에서 적용한다는 차이만 있습니다.

Tunnel encapsulation and decapsulation are rather more complicated, as they
involve a third netdevice (a tunnel netdev operating in metadata mode, such as
a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and
require an IP address to be bound to the underlay device (e.g. switchdev
function uplink netdev or port representor).  TC rules such as::

    tc filter add dev $REP_DEV parent ffff: flower \
        action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \
                              dst_port 4789 \
        action mirred egress redirect dev vxlan0
    tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \
        enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \
        action tunnel_key unset action mirred egress redirect dev $REP_DEV

where ``LOCAL_IP`` is an IP address bound to ``PORT_DEV``, and ``REMOTE_IP`` is
another IP address on the same subnet, mean that packets sent by the VF should
be VxLAN encapsulated and sent out the physical port (the driver has to deduce
this by a route lookup of ``LOCAL_IP`` leading to ``PORT_DEV``, and also
perform an ARP/neighbour table lookup to find the MAC addresses to use in the
outer Ethernet frame), while UDP packets received on the physical port with UDP
port 4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``,
decapsulated and forwarded to the VF.

If this all seems complicated, just remember the 'golden rule' of TC offload:
the hardware should ensure the same final results as if the packets were
processed through the slow path, traversed software TC (except ignoring any
``skip_hw`` rules and applying any ``skip_sw`` rules) and were transmitted or
received through the representor netdevices.

Representee link·MTU·MAC 설정

Representee의 link state는 representor를 통해 제어합니다. Representor를 administratively UP 또는 DOWN으로 설정하면 representee의 carrier도 ON 또는 OFF가 되어야 합니다.

Representor MTU를 설정하면 representee도 같은 MTU를 보고해야 합니다. Hardware가 MTU와 MRU를 서로 다른 값으로 설정할 수 있다면 representor MTU는 representee MRU에, representor MRU는 representee MTU에 대응합니다.

현재 representor를 통해 representee의 station permanent MAC address를 설정하는 방법은 없습니다. 대신 legacy SR-IOV의 `ip link set DEVICE vf NUM mac LLADDR` 또는 devlink port function API를 사용합니다.

Configuring the representee's MAC
---------------------------------

The representee's link state is controlled through the representor.  Setting the
representor administratively UP or DOWN should cause carrier ON or OFF at the
representee.

Setting an MTU on the representor should cause that same MTU to be reported to
the representee.
(On hardware that allows configuring separate and distinct MTU and MRU values,
the representor MTU should correspond to the representee's MRU and vice-versa.)

Currently there is no way to use the representor to set the station permanent
MAC address of the representee; other methods available to do this include:

 - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``)
 - devlink port function (see **devlink-port(8)** and
   :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`)