NET_FAILOVER

Master·primary·standby model

`net_failover` driver는 failover master netdev 생성·파괴 API를 제공하고 generic failover infrastructure에 등록되는 primary와 standby slave netdev를 관리합니다. Paravirtual interface는 standby, 같은 MAC의 passthrough/VF device는 primary가 되며 failover master와 standby는 같은 PCI device에 연결됩니다. User는 failover netdev만 사용합니다.

Primary가 존재하고 link가 up이며 running이면 failover master가 기본 송신 경로로 선택합니다. Paravirtual driver는 이를 통해 low-latency direct datapath를 추가할 수 있고, hypervisor는 live migration 때 VF를 unplug하여 자동으로 paravirtual path로 전환할 수 있습니다.

.. SPDX-License-Identifier: GPL-2.0

============
NET_FAILOVER
============

Overview
========

The net_failover driver provides an automated failover mechanism via APIs
to create and destroy a failover master netdev and manages a primary and
standby slave netdevs that get registered via the generic failover
infrastructure.

The failover netdev acts a master device and controls 2 slave devices. The
original paravirtual interface is registered as 'standby' slave netdev and
a passthru/vf device with the same MAC gets registered as 'primary' slave
netdev. Both 'standby' and 'failover' netdevs are associated with the same
'pci' device. The user accesses the network interface via 'failover' netdev.
The 'failover' netdev chooses 'primary' netdev as default for transmits when
it is available with link up and running.

This can be used by paravirtual drivers to enable an alternate low latency
datapath. It also enables hypervisor controlled live migration of a VM with
direct attached VF by failing over to the paravirtual datapath when the VF
is unplugged.

virtio-net STANDBY와 libvirt 설정

`net_failover`는 guest userspace 변경 없이 또는 최소 변경으로 virtio-net VM에 hypervisor-controlled accelerated datapath를 제공합니다. Hypervisor는 virtio-net에 `VIRTIO_NET_F_STANDBY` feature를 켜고 virtio-net과 VF에 같은 MAC address를 할당해야 합니다.

Libvirt 예제의 첫 interface는 항상 연결되는 virtio-net persistent device입니다. `<teaming type='persistent'>`로 표시하고 초기 link를 down으로 두어 정상 통신에서는 failover master가 VF passthrough를 선호하게 합니다. Live migration 때는 끊김 없는 통신을 위해 virtio-net을 up으로 올립니다.

두 번째 hostdev interface는 일시적으로 unplug될 수 있는 VF라 `<teaming type='transient' persistent='ua-backup0'>`를 사용합니다. `persistent` attribute는 virtio-net device의 alias를 가리켜 두 interface를 같은 failover team으로 묶습니다. MAC, PCI address와 XML 구조는 원문 code block 그대로 보존합니다.

virtio-net accelerated datapath: STANDBY mode
=============================================

net_failover enables hypervisor controlled accelerated datapath to virtio-net
enabled VMs in a transparent manner with no/minimal guest userspace changes.

To support this, the hypervisor needs to enable VIRTIO_NET_F_STANDBY
feature on the virtio-net interface and assign the same MAC address to both
virtio-net and VF interfaces.

Here is an example libvirt XML snippet that shows such configuration:
::

  <interface type='network'>
    <mac address='52:54:00:00:12:53'/>
    <source network='enp66s0f0_br'/>
    <target dev='tap01'/>
    <model type='virtio'/>
    <driver name='vhost' queues='4'/>
    <link state='down'/>
    <teaming type='persistent'/>
    <alias name='ua-backup0'/>
  </interface>
  <interface type='hostdev' managed='yes'>
    <mac address='52:54:00:00:12:53'/>
    <source>
      <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
    </source>
    <teaming type='transient' persistent='ua-backup0'/>
  </interface>

In this configuration, the first device definition is for the virtio-net
interface and this acts as the 'persistent' device indicating that this
interface will always be plugged in. This is specified by the 'teaming' tag with
required attribute type having value 'persistent'. The link state for the
virtio-net device is set to 'down' to ensure that the 'failover' netdev prefers
the VF passthrough device for normal communication. The virtio-net device will
be brought UP during live migration to allow uninterrupted communication.

The second device definition is for the VF passthrough interface. Here the
'teaming' tag is provided with type 'transient' indicating that this device may
periodically be unplugged. A second attribute - 'persistent' is provided and
points to the alias name declared for the virtio-net device.

Guest interface와 network daemon 주의점

예제 VM 안에는 세 interface가 생깁니다. `ens10`은 user가 사용하는 failover master, `ens10nsby`는 down 상태의 standby virtio-net slave, `ens11`은 up 상태의 primary VF passthrough slave입니다. 세 device는 같은 MAC을 사용하고 slave는 master `ens10`에 연결됩니다.

systemd-networkd나 ifupdown 같은 일부 userspace network configuration daemon은 `net_failover`를 이해하지 못합니다. 첫 boot에 failover master와 VF가 둘 다 DHCP로 IP를 받으면 같은 IP든 다른 IP든 VM connectivity가 끊길 수 있으므로 IP는 failover device에서만 받아야 합니다.

Debian cloud image의 `cloud-ifupdown-helper` patch 예제는 `/sys/class/net/${INTERFACE}/master` directory가 있으면 slave interface configuration 생성을 즉시 건너뜁니다. 이렇게 slave가 별도 IP를 받는 것을 막습니다.

Booting a VM with the above configuration will result in the following 3
interfaces created in the VM:
::

  4: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff
      inet 192.168.12.53/24 brd 192.168.12.255 scope global dynamic ens10
         valid_lft 42482sec preferred_lft 42482sec
      inet6 fe80::97d8:db2:8c10:b6d6/64 scope link
         valid_lft forever preferred_lft forever
  5: ens10nsby: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master ens10 state DOWN group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff
  7: ens11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens10 state UP group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff

Here, ens10 is the 'failover' master interface, ens10nsby is the slave 'standby'
virtio-net interface, and ens11 is the slave 'primary' VF passthrough interface.

One point to note here is that some user space network configuration daemons
like systemd-networkd, ifupdown, etc, do not understand the 'net_failover'
device; and on the first boot, the VM might end up with both 'failover' device
and VF acquiring IP addresses (either same or different) from the DHCP server.
This will result in lack of connectivity to the VM. So some tweaks might be
needed to these network configuration daemons to make sure that an IP is
received only on the 'failover' device.

Below is the patch snippet used with 'cloud-ifupdown-helper' script found on
Debian cloud images::

  @@ -27,6 +27,8 @@ do_setup() {
       local working="$cfgdir/.$INTERFACE"
       local final="$cfgdir/$INTERFACE"

  +    if [ -d "/sys/class/net/${INTERFACE}/master" ]; then exit 0; fi
  +
       if ifup --no-act "$INTERFACE" > /dev/null 2>&1; then
           # interface is already known to ifupdown, no need to generate cfg
           log "Skipping configuration generation for $INTERFACE"

Source hypervisor의 live migration 절차

Direct-attached SR-IOV VF를 가진 VM도 VF를 unplug할 때 paravirtual datapath로 자동 failover하여 live migration할 수 있습니다. 예제는 VM의 vnet device와 별도 bridge-attached VF 하나가 software bridge `br0`에 연결되었다고 가정합니다. 이 bridge VF는 `vf.xml`로 VM에 passthrough한 VF와는 다릅니다.

Source script는 먼저 `virsh domif-setlink`로 VM의 virtio-net을 up으로 올립니다. 이어 `virsh detach-device --live --config`로 passthrough VF를 제거하고 PF의 해당 VF MAC을 zero MAC으로 바꿉니다.

통신이 VF→`br0`→vnet path로 계속 흐르도록 VF와 TAP master에 MAC FDB entry를 추가한 뒤 `virsh migrate --live --persistent`를 실행합니다. Migration이 끝나면 source의 두 FDB entry를 삭제합니다. Destination hypervisor는 시작 전에 공유 bridge `br0`를 만들고 destination PF의 VF와 알맞은 FDB entry를 준비합니다.

Live Migration of a VM with SR-IOV VF & virtio-net in STANDBY mode
==================================================================

net_failover also enables hypervisor controlled live migration to be supported
with VMs that have direct attached SR-IOV VF devices by automatic failover to
the paravirtual datapath when the VF is unplugged.

Here is a sample script that shows the steps to initiate live migration from
the source hypervisor. Note: It is assumed that the VM is connected to a
software bridge 'br0' which has a single VF attached to it along with the vnet
device to the VM. This is not the VF that was passthrough'd to the VM (seen in
the vf.xml file).
::

  # cat vf.xml
  <interface type='hostdev' managed='yes'>
    <mac address='52:54:00:00:12:53'/>
    <source>
      <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
    </source>
    <teaming type='transient' persistent='ua-backup0'/>
  </interface>

  # Source Hypervisor migrate.sh
  #!/bin/bash

  DOMAIN=vm-01
  PF=ens6np0
  VF=ens6v1             # VF attached to the bridge.
  VF_NUM=1
  TAP_IF=vmtap01        # virtio-net interface in the VM.
  VF_XML=vf.xml

  MAC=52:54:00:00:12:53
  ZERO_MAC=00:00:00:00:00:00

  # Set the virtio-net interface up.
  virsh domif-setlink $DOMAIN $TAP_IF up

  # Remove the VF that was passthrough'd to the VM.
  virsh detach-device --live --config $DOMAIN $VF_XML

  ip link set $PF vf $VF_NUM mac $ZERO_MAC

  # Add FDB entry for traffic to continue going to the VM via
  # the VF -> br0 -> vnet interface path.
  bridge fdb add $MAC dev $VF
  bridge fdb add $MAC dev $TAP_IF master

  # Migrate the VM
  virsh migrate --live --persistent $DOMAIN qemu+ssh://$REMOTE_HOST/system

  # Clean up FDB entries after migration completes.
  bridge fdb del $MAC dev $VF
  bridge fdb del $MAC dev $TAP_IF master

On the destination hypervisor, a shared bridge 'br0' is created before migration
starts, and a VF from the destination PF is added to the bridge. Similarly an
appropriate FDB entry is added.

Destination에서 VF 재연결

Migration 완료 후 destination script는 준비했던 VF와 TAP의 FDB entry를 삭제하고 `virsh attach-device --config --live vm01 vf.xml`로 VF를 VM에 다시 연결합니다. 마지막으로 `virsh domif-setlink vm01 vmtap01 down`으로 standby virtio-net link를 내리면 정상 traffic이 다시 primary VF를 사용합니다.

The following script is executed on the destination hypervisor once migration
completes, and it reattaches the VF to the VM and brings down the virtio-net
interface::

  # reattach-vf.sh
  #!/bin/bash

  bridge fdb del 52:54:00:00:12:53 dev ens36v0
  bridge fdb del 52:54:00:00:12:53 dev vmtap01 master
  virsh attach-device --config --live vm01 vf.xml
  virsh domif-setlink vm01 vmtap01 down